Method, apparatus and system for full-skeleton 3D pose recovery from a monocular camera

Abstract

Recent advances in neural networks have shown significant progress in human pose estimation tasks. Pose estimation can be classified into monocular 2D pose estimation, multi-view 3D pose estimation, and single-view 3D pose estimation. Recently, 3D pose has attracted increasing attention and been applied to AR / VR, gaming, and human-computer interaction applications. However, current academic benchmarks for human 3D pose estimation only consider performance with respect to their relative pose. Route positioning over time, in other words, the "trajectory" of the whole body in 3D space, is not sufficiently well considered. Applications such as motion capture not only require accurate relative body poses, but also the whole body's route position in 3D space. Therefore, this paper describes an efficient monocular full 3D pose recovery model from 2D pose input, which can be applied to the above applications.

Description

[Technical Field]

[0001] The present invention relates to content estimation, and more particularly to 3D pose estimation. [Background technology]

[0002] After the great success of human 2D pose estimation, human 3D pose estimation has attracted much attention to expand its applications in movies, surveillance, human-computer interaction, etc. Many methods have been proposed, including multi-view methods, temporal methods, monocular 3D pose methods for skeletons, and monocular 3D pose methods including 3D meshes. Summary of the Invention [Problem to be solved by the invention]

[0003] Recent advances in neural networks have demonstrated significant progress in human pose estimation tasks. Pose estimation can be classified into monocular 2D pose estimation, multi-view 3D pose estimation, and single-view 3D pose estimation. Recently, 3D pose has increasingly attracted attention and been applied to AR / VR, gaming, and human-computer interaction applications. However, current academic benchmarks for human 3D pose estimation only consider performance with respect to their relative pose. Root positioning over time, in other words, the "trajectory" of the whole body in 3D space, is not sufficiently well considered. Applications such as motion capture not only require accurate relative body poses but also the whole body's root position in 3D space. Therefore, this paper describes an efficient monocular full-3D pose recovery model from 2D pose input, which can be applied to the above applications. This paper describes a network architecture that combines temporal 1D convolutions and long-short-term memory (LSTM) for root position estimation, a method for formulating the output, the design of a loss function, and a comparison with state-of-the-art models to demonstrate the effectiveness of our method for application use. As described herein, we perform 3D pose estimation on 15 and 17 keypoints, but this can be extended to any keypoint definition. [Means for solving the problem]

[0004] In one aspect, a method includes receiving camera information, the camera information including a two-dimensional pose and camera parameters including a focal length; applying feature extraction to the camera information, the feature extraction including a residual determination involving one-dimensional convolution; estimating bone lengths based on the feature extraction; estimating bone unit vectors based on the feature extraction conditioned on the bone lengths; estimating relative poses from the bone lengths and the bone unit vectors, and deriving root positions based on the feature extraction conditioned on the bone lengths and the bone unit vectors. The method further includes receiving one or more frames as input. No bone length is assumed to exceed one meter in length. Long-short-term memory is used to estimate the root positions and stabilize the root positions. The method further includes applying automatic scaling to global position and rotation to simulate dynamic motion. The method further includes randomly changing the camera field of view for each batch sample to estimate arbitrary videos containing different camera parameters. The method further includes performing 2D pose perturbation on the 2D pose input using Gaussian noise and random keypoint drop to simulate noise and occlusion conditions for 2D pose prediction.

[0005] In another aspect, an apparatus includes: a non-transitory memory for storing an application for receiving camera information, the camera information including a two-dimensional pose and camera parameters including a focal length; applying feature extraction to the camera information, the feature extraction including a residual determination including a one-dimensional convolution; estimating bone lengths based on the feature extraction; estimating bone unit vectors based on the feature extraction conditioned on the bone lengths; estimating a relative pose from the bone lengths and the bone unit vectors; and deriving a root position based on the feature extraction conditioned on the bone lengths and the bone unit vectors. The application is configured to receive one or more frames as input. No bone length is assumed to exceed one meter in length. Long- and short-term memory is used to estimate the root position and stabilize the root position. The application is configured to apply automatic scaling to global position and rotation to simulate dynamic motion. The application is configured to randomly change the camera field of view for each batch sample to estimate arbitrary videos with different camera parameters. The application is configured to perform 2D pose perturbations with Gaussian noise and random keypoint dropping for the 2D pose input to simulate noise and occlusion conditions for 2D pose prediction.

[0006] In another aspect, a system includes a camera configured to capture content and a computing device, the computing device configured to: receive camera information from the camera, the camera information including camera parameters including a two-dimensional pose and a focal length; apply feature extraction to the camera information, the feature extraction including a residual determination including a one-dimensional convolution; estimate bone lengths based on the feature extraction; estimate bone unit vectors based on the feature extraction conditioned on the bone lengths; estimate a relative pose from the bone lengths and the bone unit vectors; and derive a root position based on the feature extraction conditioned on the bone lengths and the bone unit vectors. The computing device is further configured to receive one or more frames as input. No bone length is assumed to be greater than one meter in length. Long-short-term memory is used to estimate the root position and stabilize the root position. The computing device is further configured to apply automatic scaling to global position and rotation to simulate dynamic motion. The computing device is further configured to randomly change the camera field of view for each batch sample to estimate any video with different camera parameters. The computing device is further configured to perform 2D pose perturbations with Gaussian noise and random keypoint dropping on the 2D pose input to simulate noise and occlusion conditions for 2D pose prediction. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 illustrates visualization of model output on in-the-wild (e.g., containing uncontrolled variables / environment) video, according to some embodiments. [Figure 2] FIG. 1 illustrates how the models described herein overcome depth ambiguity, according to some embodiments. [Figure 3] 10A-10C illustrate the same 3D pose reprojected onto UV with different FOVs, according to some embodiments. [Figure 4] FIG. 10 illustrates the variation of keypoint definitions according to some embodiments. [Figure 5] FIG. 10 illustrates 3D camera coordinates for designs described herein, according to some embodiments. [Figure 6] FIG. 1 illustrates how values ​​are encoded in normalized space, according to some embodiments. [Figure 7] FIG. 10 illustrates the distribution of root positions in the XZ and ZY planes in camera coordinates according to some embodiments. [Figure 8] FIG. 1 illustrates perturbations to an input and keypoint drops according to some embodiments. [Figure 9] FIG. 1 is a simplified block diagram of a network as described herein, according to some embodiments. [Figure 10] FIG. 1 illustrates a visualization of route position predictions for an object, according to some embodiments. [Figure 11] FIG. 10 shows a table of results from adding the data augmentation scheme described herein to Human3.6M, according to some embodiments. [Figure 12] FIG. 10 illustrates a table comparing LSTM and 1D convolution root location estimation, according to some embodiments. [Figure 13] FIG. 10 illustrates a visualization of Z-axis root position traces in a sample sequence for comparing models using LSTM and 1D convolutions, according to some embodiments. [Figure 14A] FIG. 1 illustrates a backflip video from YouTube® and AlphaPose applied as a 2D pose detector, performed using methods described herein according to some embodiments. [Figure 14B] FIG. 1 illustrates a backflip video from YouTube® and AlphaPose applied as a 2D pose detector, performed using methods described herein according to some embodiments. [Figure 15]FIG. 1 is a block diagram of an exemplary computing device configured to implement a full-skeleton 3D pose reconstruction method, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0008] Recent advances in neural networks have demonstrated significant progress in human pose estimation tasks. Pose estimation can be classified into monocular 2D pose estimation, multi-view 3D pose estimation, and single-view 3D pose estimation. Recently, 3D pose has increasingly attracted attention and been applied to AR / VR, gaming, and human-computer interaction applications. However, current academic benchmarks for human 3D pose estimation only consider performance with respect to their relative pose. Route positioning over time, in other words, the "trajectory" of the whole body in 3D space, is not sufficiently well considered. Applications such as motion capture not only require accurate relative body poses but also the whole-body route position in 3D space. Therefore, this paper describes an efficient monocular full-3D pose recovery model from 2D pose input, which can be applied to the above applications. This paper describes a network architecture that combines temporal 1D convolutions and long-short-term memory (LSTM) for route position estimation, a method for formulating the output, the design of a loss function, and a comparison with state-of-the-art models to demonstrate the effectiveness of our method for application use. As described herein, we perform 3D pose estimation on 15 and 17 keypoints, but this can be extended to any keypoint definition.

[0009] Monocular human 3D pose estimation has emerged as a hot topic in the research community because it can be applied to in-the-wild (e.g., uncontrolled environment) videos available over the Internet as consumer-generated videos. Furthermore, enabling pose estimation in a monocular setting eliminates the need for multiple camera setups and camera alignment to resolve triangulation. While recent research on monocular human 3D pose estimation has shown significant improvement over time, combining global trajectories and relative poses is an extremely challenging problem due to the nature of ambiguity, as multiple 3D poses can be mapped to the same 2D pose. Furthermore, qualitatively evaluating these methods from video results is extremely challenging and sometimes misleading in terms of performance, because depth ambiguity and trajectory drift cannot be fully observed when results are simply overlaid on the input image plane. Second, the methods described in the Background section only estimate relative 3D poses, which are defined as root bones that should be at a fixed (e.g., zero) position, and trajectory recovery in motion has not been thoroughly studied. Finally, the main dataset used in the above human 3D pose estimation evaluation, Human3.6M, lacks real-world settings to cover situations that may occur when applied to in-the-wild videos. The dataset has eight cameras with almost identical camera parameters captured in a laboratory setting within a 3x4 meter area. Therefore, additional 2D pose data is typically used in semi-supervised methods, including adversarial losses.

[0010] To address the above issues, which are crucial for applying monocular human 3D pose estimation for motion capture purposes, we describe a unified human 3D relative pose and trajectory recovery network from 2D pose inputs combined with 1D convolutions for relative pose and LSTMs for trajectories. This model is efficient in terms of parameter size compared to previous state-of-the-art methods, and more stable trajectory recovery is observed using LSTMs over convolutions.

[0011] The model captures multiple frames when available, but is not limited by design to using multiple frames, as is the method in VP3D. VP3D stands for VideoPose3D and is from "3D Human Pose Estimation in Video with Temporal Convolutions and Semi-Supervised Training" (https: / / github.com / facebookresearch / VideoPose3D). The VP3D method achieves its best performance with a 243-frame input, but the model described here also works with a single-frame input. This is important when we want to accommodate an arbitrary number of frames to be input for processing. We place more value on usability rather than on relative pose accuracy that is a few millimeters apart.

[0012] To simultaneously regress the root position and relative pose in a unified network, we use kinematic chain space (KCS) for regularization purposes. Instead of using regularization, we directly estimate bone unit vectors and bone lengths and apply each loss separately to enforce bone length consistency across input frames. All bone lengths are assumed to be in the {0,1} range, and human bones are assumed to be no longer than one meter in length. Furthermore, we apply a tanh-like encoding / decoding scheme to the root position to ensure that the network parameters are within the same dynamic range.

[0013] Because the Human3.6M dataset is small in terms of coverage in 3D space and actions, we apply automatic augmentation to the global position and rotation to simulate dynamic motions such as backflips or cartwheels. We randomly change the camera field of view for each batch of samples to allow estimation of arbitrary videos with different camera parameters, where predictions should be conditioned on given camera parameters. For 2D pose input, we perform 2D pose perturbation using Gaussian noise and random keypoint dropping to simulate noise and occlusion conditions for 2D pose prediction. This allows using only motion capture data, in which case no adversarial modules or losses are introduced, and therefore shorter preparation and training time. The Human3.6M data is merely an exemplary dataset to be used with the methods and systems described herein and is not intended to be limiting in any way. Any 3D human motion capture dataset can be used with the methods and systems described herein.

[0014] FIG. 1 illustrates visualization of model output on in-the-wild (e.g., uncontrolled variables / environment) video according to some embodiments. In FIG. 1, column (a) shows video frames with 2D pose estimation, column (b) shows XY-plane 3D pose, and column (c) shows XZ-plane 3D pose. The red line on the 3D plot (roughly passing through the person's back and head) indicates the global trajectory. The model can output a trajectory that includes a stable z-position in dynamic motion. For a detailed definition of camera coordinates, see FIG. 5.

[0015] The model described here overcomes depth ambiguity, as shown in Figure 2. The top row of Figure 2 shows the camera plane projection of the person 3D pose prediction. The bottom row shows a side view of the reconstruction. Even if the person is only moving parallel to the camera, the whole body, especially in the depth direction, is poorly estimated.

[0016] Monocular human 3D pose estimation methods can be broadly divided into two categories: mesh-based methods and 2D lifting methods.

[0017] Mesh-based methods Mesh-based methods use a conventional model, such as a person mesh, to recover not only the pose but also the skin and fit it to the image plane. Specifically, mesh-based methods perform reasonably well when the results are overlaid on the image plane, but when viewed from a different perspective, such as the side view in Figure 2, unstable trajectory traces are visible. This stems from the highly ambiguous nature of the problem, which is a drawback of monocular methods. Even if the problem space is made smaller by using a conventional model of a person, the problem is still not well solved.

[0018] 2D lifting method Another category is monocular human 3D skeleton pose, where the input to the model is a 2D pose predicted by a well-established human 2D pose detector. To stabilize along the temporal dimension, some implementations have used LSTM sequence-to-sequence techniques. However, these involve encoding all frames to a fixed length. VP3D exploits temporal information by performing 1D convolutions across the temporal dimension. VP3D also splits the network into two, in this case, the relative pose and trajectory estimation networks, which are trained separately and jointly. However, the networks for relative pose and trajectory use 16M parameters each, while the full pose estimation network uses 32M parameters. It also uses a 243-frame input to obtain the best performance, but due to the limited camera configuration of Human3.6M, it does not work well on videos with different camera parameters than the training data.

[0019] Kinematic Chain Space Using the kinematic chain space (KCS), pose can be decomposed into bone vectors and their lengths. Instead of estimating relative poses in Cartesian coordinates, we followed the idea of ​​using KCS. The model described here differs in how KCS is utilized in optimization. We used KCS to map relative poses to KCS and train the model in a semi-supervised manner using an adversarial loss as a regularization term. Unlike the above, the method described here directly regresses bone vectors and bone lengths that exist in the normalized space.

[0020] This paper describes how to perform input and output definitions, datasets and augmentations, network design and loss formulation.

[0021] input Following a similar scheme to the 2D pose lifting method described herein, in this case, 2D pose can be estimated from any 2D pose detector. For example, AlphaPose can be used. As shown in Figure 4, the 2D pose detector outputs a variety of keypoints, e.g., 17 to 25, and Human3.6M uses 17 keypoints (17 are movable out of 32 definitions). To run the model described herein on any 2D pose detector, the 15 most intersecting keypoints are defined, and these can use Human3.6M data (or other data) for evaluation. As input, UV-normalized 2D coordinates are used, where u ∈ {0,1}. Also, 2D pose detectors often fail to detect certain keypoints due to occlusion. For those, the value is set to zero. The camera focal length is also used as input. Monocular human 3D pose estimation methods use Human3.6M and Human-Eva, but neither of these datasets has a wide variety of camera settings, and attempts have been made to apply models to operate on in-the-wild videos and images by applying semi-supervised training using 2D annotations. While it is possible to estimate camera parameters and calculate reprojection errors, camera parameters are still implicitly modeled through a pose generation network. Instead, the network described herein is modeled conditional on 2D pose input and camera focal length. Focal length is a very important cue for supporting arbitrary cameras. As shown in Figure 3, different camera fields of view (FOVs) can significantly change the 2D pose appearance even with the same relative pose and root position in 3D space. As described herein, we assume that a perspective camera containing the principal point is at the center of the image and that lens distortion is not considered.

[0022] Figure 3 shows the same 3D pose reprojected onto UV with different FOVs: (a) 60° FOV, (b) 90° FOV, and (c) 120° FOV. The camera parameters can be different for each of the clips captured in the wild.

[0023] Figure 4 shows the changes in keypoint definitions. Image (a) is MSCOCO with 17 points. Image (b) is OpenPose with 18 points. Image (c) is OpenPose with 25 points. Image (d) is Human3.6 with 17 points (17 of 32 are movable). Image (e) is the method described herein with a 15-point definition. The lines are the standard skeleton pairs in each definition.

[0024] Output and Kinematic Chain Space The network output is defined as a combination of the body's root position and relative pose. The root position is typically defined at a keypoint on the pelvis. The relative pose is defined as the 3D position of other bones relative to the root position. Image (e) of Figure 4 shows the definition of 15 keypoints, where 0 is the pelvis and is used as the root position, and the others are estimated as relative positions to the root. Figure 5 shows the definition of 3D space as described herein. Figure 5 shows the 3D camera coordinates of the design described herein, according to some embodiments. Figure 3 shows 2D projections of this pose at different FOVs. The relative pose and root position are estimated in camera coordinates. Furthermore, KCS is used to decompose the relative pose into bone vectors and their lengths. The i-th joint in the kinematic chain is represented by a vector containing the x, y, and z coordinates of this joint's position. The relative orientation of the kinematic chain, P, is defined by connecting j joint vectors as follows: r A matrix can be constructed to represent P r= (p1, p2,..., p j ) (1) The whole body posture P is expressed as follows: P = (p0, p0,..., p r ) (2) where p0 is the root position, and the relative pose is derived by subtracting the root pose. k is defined as the vector between the rth and tth joints as follows: b k = p r -p t = P rj d k , (3) where: TIFF0007775563000002.tif21168 is 1 at position r and -1 at position t. d is the mapping vector for the rth and tth joints, and by concatenating across all joints, the overall mapping matrix D is TIFF0007775563000003.tif1145. Similar to equation (1), the matrix TIFF0007775563000004.tif1042 can be defined as a matrix containing all b bones as follows: B = (b1, b2, ..., b b ); (5) Here, the matrix B is calculated from P by the following formula: B = P r D. (6) Similar to D, B is converted to P as follows: r A matrix that maps to TIFF0007775563000005.tif1041 can be defined. P r = BE. (7) The network can then learn the mapping function. TIFF0007775563000006.tif21157 where the 2D pose u and camera parameters c are the inputs and the output is the root position TIFF0007775563000007.tif7170 Bone length TIFF0007775563000008.tif10170 and its unit vector TIFF0007775563000009.tif21157 The purpose of this is to estimate ||b k ∈ {0,1}, which in some embodiments is assumed to never exceed 1m. Any symbols that contain a ^ symbol are predictions, symbols that do not contain a ^ symbol are ground truths (e.g., labels), and the loss defines how far the prediction is from the truth.

[0025] Root position in normalized space using the tanh form Encode and decode TIFF0007775563000010.tif1031. Then later decode it to the actual value. The encoding formula is: TIFF0007775563000011.tif36157 The decoding is formed as follows: TIFF0007775563000012.tif31157Here, β and ε are constant values. We use TIFF0007775563000013.tif6170. Figure 6 shows how values ​​are encoded in normalized space. Figure 6 shows the encoding and decoding of the root position, which gives greater granularity for distances near the camera and saturates at 20m. The z-axis values ​​are non-negative.

[0026] This normalization is very important because many pose regression models do not adequately consider how to model the output space and parameter space. VP3D proposes to simultaneously estimate the root position and relative pose using two discrete networks, each containing a weighted loss at the root position, where the loss at the farther root position has a smaller weight. The method described here involves forming a granularity in the encoded space and placing the parameter space within it. This is important for the end-to-end training method to propagate gradients and update parameters not only for the root position but also for the bone vectors at the same time.

[0027] Datasets and Extensions Instead of using 2D pose annotation in a semi-supervised manner, which many methods employ to generalize well on in-the-wild videos and images, the model described here is trained solely from motion capture data. In a purely academic context, we use Human3.6M for initial experiments, and for commercial purposes, we use motion capture data provided by Sony Interactive Entertainment, Inc. (SIE). However, motion capture data is often too small to cover real-world scenarios. To solve this problem, we automatically use several augmentations and perturbations in the training data. TIFF0007775563000014.tif250160 TIFF0007775563000015.tif38170

[0028] Algorithm 1 is a simplified pseudocode for the extension. Given an entire dataset P, each batch sample p ⊂ P contains a time frame with length T, which means that p t∈ p, t = (0, 1, ..., T). We randomly choose the FOV and fit pose trajectories within the viewport to ensure that no pose is hidden from the camera view. We also randomly perform somersault motions in sequence p to simulate backflip or cartwheel type motions by analyzing the trajectory variance. Figure 7 shows the distribution of root positions for the original Human3.6M and the distribution after data augmentation as described herein, where the implementation described herein has a wider position distribution, making the dataset more suitable for real-world scenarios.

[0029] Figure 7 shows the distribution of root positions in the XZ and ZY planes in camera coordinates according to some embodiments. Image (a) is the original Human3.6M, and image (b) is the extension described herein.

[0030] Furthermore, in the training phase, we use 2D keypoint dropping and perturbation on the input. During data sampling, we project the 3D pose into 2D using perspective projection. However, 2D pose detectors are noisy and prone to missed detections due to occlusions. Methods such as VP3D train models to be noise-proof by using the results of the 2D detector as a noisy 2D input. Instead, as described herein, we perturb the 2D projected keypoints using Gaussian noise and randomly dropping keypoints to simulate occlusion scenarios. The Gaussian radius is adaptive depending on the body size in UV space. All keypoints marked as "dropped" are set to zero. Figure 8 shows the perturbation and keypoint dropping on the input. Figure (a) shows the original clean 2D pose, and Figures (b)-(d) show the noisy 2D poses with random dropping and perturbation applied.

[0031] Network Details As shown in Equation (8), we learn a mapping function given the input 2D pose u and camera parameters c to find the root position TIFF0007775563000016.tif7170 Bone length TIFF0007775563000017.tif10170 and its unit vector TIFF0007775563000018.tif18170 The goal is to output a sigma-based prediction of the sequence. To achieve this, we use 1D convolution and LSTM together to achieve stable prediction of the sequence. Figure 9 shows a simplified block diagram of a network as described herein, according to some embodiments. The reason for using LSTM for root position is that, similar to relative pose estimation in KCS space, 1D convolution with a kernel size of 3 has been experimentally used. However, it has poor stability, especially on the z-axis, which is a common problem in monocular 3D pose estimation. This is assumed to be because, even if a time loss function is applied, 1D convolution cannot guarantee to estimate the output at time t given the previous time t-1. However, LSTM can pass features from the previous time to the current feature, thereby stabilizing the overall root position estimation.

[0032] For an input u containing 512 and 1024 feature maps, there are two feature extraction blocks containing four stacked residual connections, each containing a 1D convolution with a kernel size of 1. Here, the 1D convolution with a kernel size of 1 involves treating all time frames in a discrete manner, thus mapping the feature space at each time frame. Next, the output of each block is cascaded with a 1D convolution with a kernel size of 3, with edge padding applied to the convolution. A 1D convolution with a kernel size of 3 is used to aggregate adjacent frames. The edge padding applied to all convolutions with a kernel size of 3 ensures that the number of output frames is equal to the number of input frames, unlike VP3D, which outputs only one frame out of N frames (243 frames for one frame output is the best model for VP3D). The cascade order is designed to first predict the bone length, then, conditional on it, predict the bone unit vector, and finally predict the root position. Again, a convolution with a kernel size of 1 is used to map each output to a feature space, which is then concatenated with features extracted at earlier stages to estimate subsequent predictions. This stems from how people intuitively estimate subject distance by first estimating overall subject size using the subject's surrounding context. We found that better accuracy was obtained by separating the initial feature extraction block for bone length using bone unit vectors and root positions. The LSTM block has two repeated layers containing 128 hidden units and is unidirectional. In some embodiments, all activation functions use Parametric ReLU.

[0033] loss formulation Here we describe the loss formulation. First, we usually apply an L2 loss to each of the outputs, as follows: TIFF0007775563000019.tif34157Here, B is the bone length ||B|| and its unit vector TIFF0007775563000020.tif16157 Furthermore, the relative orientation P r A term is added for x, which can be derived by equation (7) and involves adding larger weights to bone lengths and vectors. The p0 term is for the root position in both the coded and decoded spaces, which includes a smooth L1 loss with x2 amplitude on the z-axis. The reason for applying smooth L1 for the root position is that the loss on the decoded space is large and may affect other loss ranges by having large errors. We do not apply the loss only to the coded space, but perform the loss on both the coded and decoded spaces. Furthermore, we add a time term for bone B and root position p0, as follows: TIFF0007775563000021.tif26157Here, the first term on Δ||B|| is zero due to bone lengths not changing over time. This ensures bone lengths are consistent across time frames. For the root position, we use not only the delta of adjacent frames, but also up to the third adjacent order and up to the second temporal derivative. We use temporal differencing to regularize the relative motion between frames, so we can converge to a small loss even though the root position may still have offset errors. However, this is important for trajectory tracing, especially for motion capture scenarios. We apply a 2D reprojection error as follows: TIFF0007775563000022.tif11157 Note that this u is a clean 2D projection of the ground truth 3D pose, not the 2D pose input after the perturbations above. TIFF0007775563000023.tif7170 is the predicted 3D pose TIFF0007775563000024.tif11157. Finally, the total loss is given by: TIFF0007775563000025.tif11157Here, each loss is added equally.

[0034] Experimental evaluation Dataset and evaluation Human3.6M contains 3.6 million video frames from 11 subjects, seven of whom are annotated with 3D poses. Following the same rules as above, we split the dataset into five subjects for training (S1, S5, S6, S7, and S8) and two subjects for evaluation (S9 and S11). Each subject performed 15 motions, recorded using four cameras synchronized at 50 Hz. We use the mean per-joint position error (MPJPE) (units: millimeters), which is the average Euclidean distance between predicted and ground-truth joint positions. However, a slight change was made to the way we aggregated MPJPE: instead of averaging across all actions, we treated all actions at once. For root position, we evaluated the mean position error (MPE), which is also the average Euclidean distance across the entire evaluation data. We evaluated Human3.6M using the 15 and 17 keypoint definitions, including the extensions described herein. We only apply perturbations to add noise and keypoint drops to the training data, and to use them with camera and position augmentation for the evaluation set. Figure 4 shows the keypoint differences.

[0035] 9 shows a simplified block diagram of the model described herein, according to some embodiments. As described herein, a variant of the ablation model that does not include KCS combines blocks xB1 and xB2 together to directly estimate the relative pose in Euclidean space, and there is no LSTM model that replaces xP with a 1D convolution from the LSTM.

[0036] In step 900, camera parameters (e.g., focal length, including x and y in 2D space) are fed into the network so that the network can output the state of the camera. The network also receives a 2D pose, which can be from any image or video.

[0037] In step 902, for each frame, feature extraction as described herein is applied. Feature extraction can be implemented in any manner. Feature extraction includes residual determination, including 1D convolution. Additionally, some embodiments implement padded 1D convolution after concatenation. In step 904, bone lengths are estimated as described herein. In step 906, bone unit vectors can be estimated based on feature extraction conditioned on bone lengths. In step 908, a relative pose is estimated from the bone lengths and bone unit vectors, and a root position is derived based on feature extraction conditioned on bone lengths and bone unit vectors. In some embodiments, camera parameters are utilized in estimating the root pose. LSTM can be utilized to help estimate and stabilize the root position.

[0038] In some embodiments, fewer or additional steps may be implemented. In some embodiments, the order of the steps is changed.

[0039] Network Variations For root poses for ablation studies, we conducted experiments on models with and without KCS and with and without LSTM. The model without KCS directly regresses relative poses in Euclidean space using a 1D convolution block with a kernel size of 3, followed by a final 1D convolution with a kernel size of 1, so that the output dimension is the number of keypoints × 3. Similarly, the model without LSTM uses a 1D convolution to regress the root pose. All models are trained under the same training procedure. For comparison with other methods, we compare the method described here with the current state-of-the-art method, VP3D.

[0040] training For the optimizer, we train using Adam with weight decay set to zero for 100 epochs, including a learning rate warmup of 1e for the first epoch. -3 We apply an exponential decay to the learning rate, starting from , with a factor of 0.5 every 10 epochs. We use a batch size of 192 with 121 frame inputs, randomly skipping frames from 1 (no skips) to 5 of the 50 Hz sampled frames of Human3.6M during batch sampling. This is to make the model robust to frame rate variations in real (in the wild) videos. We retrained VP3D with the same strategy as the models described herein, except that VP3D only accepts 243 frames of input; therefore, we used 243 frames of input for VP3D instead of 121 frames. Neither using decay for batch normalization nor using Amsgrad with a decay of 0.95 as suggested in VP3D performed worse than the training procedure above for all models.

[0041] 10 illustrates a visualization of route position predictions for a subject, according to some embodiments. Axis Z has a larger error compared to the other axes, and also has a larger error for people who are far away.

[0042] Evaluation and Ablation Studies Figure 11 shows a table of results for adding the data augmentation scheme described herein to Human3.6M, where the camera FOV varies and has a wider distribution in root position. Since neither of the alternative methods provides root position estimation, a rigorous comparison of relative pose MPJPE exists. Also, VP3D estimates one frame using 243 frames, while the model described herein is trained using 121 frames. While the model described herein can take any frame size, to make a comparison under equal conditions, we perform evaluation on the 243-frame input and evaluate on the intermediate frame (the 121st frame). Two variants are presented: one with KCS applied and one with direct relative pose estimation. The model described herein with KCS performs better than MPJPE, which contains far fewer parameters. This indicates that it may not be possible to implicitly infer camera parameter differences without cues. Also, by examining the variants, the KCS method shows a significant advantage over direct estimation of relative pose. It is also interesting to note that the root positioning block is also comparable in both methods. MPE performance showed a difference. By examining the training curves and validation errors, the current assumption is that there will still be variability in route location performance.

[0043] The root position error, MPE, still appears to have a large error of about 20 cm. This indicates that there are still challenges in resolving uncertain depth from a single eye, especially from 2D pose input alone. Figure 10 shows the overall projection error for the 15-keypoint pose model. X and Y show a very good fit to the subject, but Z shows errors as the subject becomes more distant, as well as some large errors at close distances. The large errors at close distances are primarily due to subjects being too close to the camera and therefore not being able to see the entire body (e.g., only a portion of the body is visible), which occurs in real-world scenarios. The experiments show that there is a lot of room for improving MPE, but we observe the entire trajectory trace, which is important for motion capture scenarios.

[0044] The model without LSTM exhibits comparable or better MPE than the LSTM model. To compare the motion traces, we performed a separate evaluation, using all output frames instead of taking one intermediate frame as input for alignment with VP3D. Using this, we examined the average trajectory error, defined as the second term in Equation (12), and found that the LSTM version exhibited better trajectory performance, as shown in Figure 12. The difference can become more pronounced when attempting to downsize model parameters. Figure 13 shows a backflip sequence applied to a compact version of the model described herein that includes LSTM over 1D convolution for root pose estimation. The 1D convolution estimates a large drift, especially on the Z-axis, which is important for motion recovery. Figure 13 shows a visualization of the Z-axis root position trace in a sample sequence to compare models using LSTM and 1D convolution. 1D convolution tends to have a large trace error, especially in dynamic motion.

[0045] Figures 14A and 14B show a backflip video from YouTube and AlphaPose applied as a 2D pose detector, performed using the method described herein. As shown on the XZ plane reprojection, the motion itself is very dynamic, with many occlusions and errors in the 2D pose detector, yet the overall root position on the Z axis is very stable. Figures 14A and 14B show a visualization of the output of the model described herein on a real (in the wild) video, arranged in two grouped columns, each group representing four frames. From left to right: video frame with 2D pose estimation, XY plane 3D pose, and XZ plane 3D pose. The red line on the 3D plot indicates the global trajectory. The model described herein can output a trajectory with a stable z position in dynamic motion. The sixth frame above has a large error in the 2D pose detection result.

[0046] conclusion The method described herein enables full-skeleton 3D pose recovery from a monocular camera, where the full skeleton includes both root position and relative pose in 3D. The model has been observed to offer significant advantages over the current state-of-the-art in academia, covering various FOVs and dynamic motions such as backflips trained using only motion capture data. Instead of brutally modeling large networks to regress values, basing the model on human perception and using KCS as described herein and forming the model in normalized space results in better performance.

[0047] The method described here only requires 2D pose input normalized in UV space and basic camera parameters as input. It is very difficult to train bone length estimation using a very small distribution and estimate true bone lengths without the support of other cues (e.g., appearance features) such as RGB images. Bone lengths can be derived from the ratio of 2D bone lengths, and we hypothesize that children tend to have longer torsos than arm bones. Based on the surrounding context, it is possible to roughly estimate a person's height. This can be achieved by using a game engine (e.g., Unreal Engine 10). (registered trademark) ) can be used to render images with associated 3D geometry and perform end-to-end estimation of person 3D pose from images. We built an original adversarial module that allows for semi-supervised training using 2D annotations.

[0048] FIG. 15 illustrates a block diagram of an exemplary computing device configured to implement a full-skeleton 3D pose reconstruction method according to some embodiments. The computing device 1500 can be used to acquire, store, compute, process, communicate, and / or display information, such as images and videos. The computing device 1500 can implement any of the aspects of full-skeleton 3D pose reconstruction. In general, a hardware structure suitable for implementing the computing device 1500 includes a network interface 1502, memory 1504, a processor 1506, I / O devices 1508, a bus 1510, and storage 1512. The selection of the processor is not critical as long as a suitable processor with sufficient speed is chosen. The memory 1504 can be any conventional computer memory known in the art. The storage 1512 can include a hard drive, CD-ROM, CDRW, DVD, DVDRW, high-definition disk / drive, ultra-high-definition drive, flash memory card, or any other storage device. The computing device 1500 can include one or more network interfaces 1502. An example of a network interface is an Ethernet (registered trademark)or other types of network cards connected to a LAN. The I / O devices 1508 may include one or more of the following: a keyboard, a mouse, a monitor, a screen, a printer, a modem, a touchscreen, a button interface, and other devices. A full-skeleton 3D pose reconstruction application 1530 used to implement the full-skeleton 3D pose reconstruction method is likely stored in the storage device 1512 and memory 1504 and processed as an application is typically processed. The computing device 1500 may include more or fewer components than those shown in FIG. 15 . In some embodiments, full-skeleton 3D pose reconstruction hardware 1520 is included. While the computing device 1500 of FIG. 15 includes an application 1530 and hardware 1520 for the full-skeleton 3D pose reconstruction method, the full-skeleton 3D pose reconstruction method may be implemented in the computing device as hardware, firmware, software, or any combination thereof. For example, in some embodiments, the full-skeleton 3D pose reconstruction application 1530 is programmed into memory and executed using a processor. In another example, in some embodiments, the full-skeleton 3D pose reconstruction hardware 1520 is programmed hardware logic that includes gates that are specifically designed to implement the full-skeleton 3D pose reconstruction method.

[0049] In some embodiments, the full skeletal 3D pose reconstruction application 1530 includes several applications and / or modules. In some embodiments, a module also includes one or more sub-modules. In some embodiments, fewer or additional modules may be included.

[0050] Examples of suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular / mobile phone, a smart appliance, a game console, a digital camera, a digital camcorder, a camera phone, a smartphone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer / player (e.g., a DVD writer / player, a high-definition disc writer / player, an ultra-high-definition disc writer / player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., a smart watch), a vehicle (e.g., an autonomous vehicle), or any other suitable computing device.

[0051] To utilize the full-skeleton 3D pose reconstruction method described herein, a device such as a digital camera / camcorder is used to capture the content, which can be implemented with user assistance or automatically without user involvement to perform pose estimation.

[0052] In operation, the full-skeleton 3D pose recovery method provides a more accurate and efficient pose estimation implementation. Results show that it produces much better pose estimation compared to traditional implementations.

[0053] Some implementations of full-skeleton 3D pose recovery from a monocular camera 1. A method comprising: receiving camera information, the camera information including a 2D pose and camera parameters including a focal length; applying feature extraction to the camera information, the feature extraction comprising a residual determination comprising a one-dimensional convolution; estimating bone lengths based on the feature extraction; estimating bone unit vectors based on the feature extraction conditioned on the bone lengths; estimating relative poses from the bone lengths and the bone unit vectors, and deriving root positions based on the feature extraction conditioned on the bone lengths and the bone unit vectors; A method comprising:

[0054] 2. The method of claim 1, further comprising the step of receiving one or more frames as input.

[0055] 3. The method of paragraph 1, where no bone length is assumed to exceed one meter in length.

[0056] 4. The method of claim 1, wherein long and short-term memory is used to estimate the route position and stabilize the route position.

[0057] 5. The method of claim 1, further comprising applying automatic extensions to the global position and rotation to simulate dynamic motion.

[0058] 6. The method of claim 1, further comprising the step of randomly varying the camera field of view for each batch sample to estimate any video containing different camera parameters.

[0059] 7. The method of claim 1, further comprising the step of performing 2D pose perturbation on the 2D pose input using Gaussian noise and random keypoint drop to simulate noise and occlusion conditions for 2D pose prediction.

[0060] 8. An apparatus comprising: receiving camera information, the camera information including a two-dimensional pose and camera parameters including a focal length; applying feature extraction to the camera information, the feature extraction comprising a residual determination comprising a one-dimensional convolution; estimating bone lengths based on the feature extraction; estimating bone unit vectors based on the feature extraction conditioned on the bone lengths; estimating a relative pose from the bone lengths and the bone unit vectors, and deriving a root position based on the feature extraction conditioned on the bone lengths and the bone unit vectors; a non-transitory memory for storing an application for performing the a processor coupled to the memory and configured to process the application; An apparatus comprising:

[0061] 9. The apparatus of clause 8, wherein the application is further configured to receive one or more frames as input.

[0062] 10. The apparatus of paragraph 8, wherein no bone length is assumed to exceed one meter in length.

[0063] 11. The device of claim 8, wherein long and short-term memory is used to estimate the route position and stabilize the route position.

[0064] 12. The device of claim 8, wherein the application is further configured to apply automatic extensions to global position and rotation to simulate dynamic motion.

[0065] 13. The apparatus of claim 8, wherein the application is further configured to randomly change the camera field of view for each batch sample to estimate any video containing different camera parameters.

[0066] 14. The apparatus of clause 8, wherein the application is further configured to perform 2D pose perturbation on the 2D pose input using Gaussian noise and random keypoint dropping to simulate noise and occlusion conditions for 2D pose prediction.

[0067] 15. A system comprising: a camera configured to capture content; a computer device; Equipped with The computer device receiving camera information from the camera, the camera information including a two-dimensional pose and camera parameters including a focal length; applying feature extraction to the camera information, the feature extraction comprising a residual determination comprising a one-dimensional convolution; estimating bone lengths based on the feature extraction; estimating bone unit vectors based on the feature extraction conditioned on the bone lengths; estimating a relative pose from the bone lengths and the bone unit vectors, and deriving a root position based on the feature extraction conditioned on the bone lengths and the bone unit vectors; configured to: system.

[0068] 16. The system of clause 15, wherein the computer device is further configured to receive one or more frames as input.

[0069] 17. The system of paragraph 15, wherein no bone length is assumed to exceed one meter in length.

[0070] 18. The system of claim 15, wherein long and short-term memory is used to estimate the route position and stabilize the route position.

[0071] 19. The system of claim 15, wherein the computing device is further configured to apply automatic extensions to global position and rotation to simulate dynamic motion.

[0072] 20. The system of claim 15, wherein the computer device is further configured to randomly vary the camera field of view for each batch sample to estimate any video containing different camera parameters.

[0073] 21. The system of clause 15, wherein the computer device is further configured to perform 2D pose perturbation using Gaussian noise and random keypoint dropping on the 2D pose input to simulate noise and occlusion conditions for 2D pose prediction.

[0074] The present invention has been described with reference to specific embodiments incorporating details to facilitate an understanding of the principles of construction and operation of the invention. Reference herein to specific embodiments and their details is not intended to limit the scope of the claims appended hereto. Those skilled in the art will readily appreciate that various other modifications can be made to the embodiments chosen for illustration without departing from the spirit and scope of the invention as defined by the claims. [Explanation of symbols]

[0075] 900 Provide camera parameters to the network Feature extraction applied every 902 frames 904 Estimate bone length 906 Estimate bone unit vector 1500 Computer Equipment 1502 network interface 1504 memory 1506 processor 1508 I / O Devices 1510 Bus 1512 Storage device 1520 Full Skeleton 3D Posture Reconstruction Hardware 1530 Full-skeleton 3D Posture Recovery Application

Claims

1. A computer-implemented method for pose estimation, comprising: receiving a two-dimensional pose of the body and camera parameters including a focal length; applying feature extraction to the two-dimensional body pose, said feature extraction comprising determining a residual for a one-dimensional convolution; estimating bone lengths based on the feature extraction; estimating bone unit vectors based on the feature extraction, conditional on the bone lengths; estimating relative poses, which are three-dimensional positions of bones relative to root positions, from the bone lengths and the bone unit vectors, and estimating the root positions of the body based on the feature extraction and the camera parameters, using the bone lengths and the bone unit vectors as conditions; A method comprising:

2. 10. The method of claim 1, further comprising the step of receiving one or more frames as input.

3. 2. The method of claim 1, wherein no bone length is assumed to exceed one meter in length.

4. 10. The method of claim 1, wherein the route location is estimated using long and short-term memory.

5. The method of claim 1, further comprising a step of performing perturbation of the two-dimensional body pose using Gaussian noise and randomly reducing keypoints to simulate noise and occlusion conditions in the two-dimensional pose prediction.

6. A posture estimation device, comprising: receiving a two-dimensional body pose and camera parameters including a focal length; applying feature extraction to the two-dimensional body pose, the feature extraction comprising determining a residual for a one-dimensional convolution; estimating bone lengths based on the feature extraction; estimating bone unit vectors based on the feature extraction, conditional on the bone lengths; estimating a relative posture, which is a three-dimensional position of a bone relative to a root position, from the bone lengths and the bone unit vectors, and estimating the root position of the body based on the feature extraction and the camera parameters under the bone lengths and the bone unit vectors; a non-transitory memory for storing an application for performing the a processor coupled to the non-transitory memory and configured to process the application; An apparatus comprising:

7. 7. The apparatus of claim 6, wherein the application is further configured to receive one or more frames as input.

8. 7. The apparatus of claim 6, wherein no bone length is assumed to exceed one meter in length.

9. 7. The apparatus of claim 6, wherein the route location is estimated using long and short-term memory.

10. 7. The apparatus of claim 6, wherein the application is further configured to perform perturbations of the 2D body pose using Gaussian noise and random keypoint reduction to simulate noise and occlusion conditions for 2D pose prediction.

11. A posture estimation system, comprising: A camera and a computer device; Equipped with The computer device receiving a two-dimensional pose of a body and camera parameters including a focal length; applying feature extraction to the two-dimensional body pose, the feature extraction comprising determining a residual for a one-dimensional convolution; estimating bone lengths based on the feature extraction; estimating bone unit vectors based on the feature extraction, conditional on the bone lengths; estimating a relative posture, which is a three-dimensional position of a bone relative to a root position, from the bone lengths and the bone unit vectors, and estimating the root position of the body based on the feature extraction and the camera parameters under the bone lengths and the bone unit vectors; configured to: A system characterized by:

12. 12. The system of claim 11, wherein the computer device is further configured to receive one or more frames as input.

13. 12. The system of claim 11, wherein no bone length is assumed to exceed one meter in length.

14. 12. The system of claim 11, wherein long and short-term memory is used to estimate the route location.

15. 12. The system of claim 11, wherein the computing device is further configured to perform perturbations of the 2D body pose using Gaussian noise and random keypoint reduction to simulate noise and occlusion conditions for 2D pose prediction.

Images