A method for estimating three-dimensional human pose from a single view based on an implicit function
By combining implicit function prediction networks with deep learning, the problems of slow iterative optimization and insufficient information in existing 3D human pose estimation technologies are solved, achieving fast and high-precision 3D human pose estimation while reducing equipment costs and site limitations.
Patent Information
- Application Number
- CN202310996114.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-09
- Publication Date
- 2025-10-24
- Estimated Expiration
- 2043-08-09
AI Technical Summary
Existing technologies for estimating 3D human pose from a single view suffer from problems such as long iterative optimization time, inaccurate reliance on 2D joint detection, and lack of in-depth information mining.
By employing a hidden function prediction network combined with deep learning methods, and through joint encoding, feature extraction, and pose space sampling, we can quickly and accurately estimate the 3D human pose, solve the many-to-one problem of axis-angle vectors, and make full use of joint information.
It achieves fast and high-precision 3D human pose estimation, reduces equipment costs, decreases reliance on complex equipment, and improves training efficiency and robustness.
Smart Images
Figure CN116978066B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of computer vision, in particular, the present application relates to a method for estimating three-dimensional human pose from single-view images based on implicit function. BACKGROUND
[0002] Three-dimensional human pose estimation is an important problem in the field of three-dimensional vision, and has a wide range of applications in motion analysis, animation production, robot control, virtual reality, etc. There are two main ways to express three-dimensional human body, which are three-dimensional skeleton and three-dimensional mesh. Compared with three-dimensional skeleton, three-dimensional mesh can present a complete human body and better express the details of the human body, so it has attracted the attention of many researchers. Among them, the parametric human model is an important branch of three-dimensional mesh representation, the parametric human model defines a three-dimensional human skeleton and a static human mesh in advance, and generates three-dimensional human models with different poses and shapes by changing the pose parameters and shape parameters under the support of skinning algorithm. This method reduces the prediction of complex human mesh to the prediction of low-dimensional vector, greatly facilitating the research of three-dimensional human pose estimation. At present, the method for estimating human pose from single view based on parametric human model mainly includes two categories:
[0003] The first category is to fit the model parameters using traditional optimization methods. Most of the optimization methods first collect the 2D information in the image using detection algorithms, such as 2D human joint points, human contour, human part segmentation, etc., and then use these information to iteratively optimize the parameters of the parameterized human model. The regularization term in the optimization method mainly consists of two parts: prior term and data term. The prior term is used to constrain the human model pose obtained by optimization to conform to the normal human pose; the role of the data term is to make the human model pose as consistent as possible with the 2D information detected in advance. The SMPLify method proposed by Bogo et al. (Bogo F, Kanazawa A, Lassner C, et al. Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image [C] / / Computer Vision - ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14. Springer International Publishing, 2016: 561-578.) is a typical optimization method, which detects the 2D human joint positions in advance, and then constantly optimizes the parameters of the parameterized model SMPL, so that the projected 3D human joint points are more consistent with the detected joint positions. However, this method takes a long time to iterate and process slowly, and depends on the existing 2D joint detection method, which cannot optimize the actual pose under inaccurate detection.
[0004] Another type of method is to use neural networks to directly regress the parameters of the parameter model. This type of method usually uses two-dimensional information such as joints, contours, etc. as weakly supervised items to train the network. Kanazawa A and Black M J et al. (Kanazawa A, Black M J, Jacobs D W, et al. End-to-end recovery of human shape and pose [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7122-7131.) first use a convolutional neural network to extract features, and then regress the SMPL model (Loper M, Mahmood N, Romero J, et al. SMPL: A skinned multi-person linear model [J]. ACM transactions on graphics (TOG), 2015, 34(6): 1-16.) parameters by minimizing the joint re-projection error. Kolotouros N and Pavlakos G et al. (Kolotouros N, Pavlakos G, Black MJ, et al. Learning to reconstruct 3D human pose and shape via model-fitting in the loop [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2019: 2252-2261.) proposed a method that combines regression and optimization together, puts SMPLify into the training process to get better results. This method directly predicts the results, the effect is general, and it lacks the extraction of features of each joint, and does not deeply mine the information. SUMMARY
[0005] In view of the technical defects existing in the above existing methods, in order to estimate a three-dimensional human pose consistent with the picture from a single picture, the present application provides a method for estimating a three-dimensional human pose from a single view picture based on an implicit function.
[0006] In order to achieve the above-mentioned application purposes, the technical solutions adopted by the method of the present application are as follows:
[0007] A method for estimating a three-dimensional human pose from a single view based on an implicit function, comprising the following steps:
[0008] S1, obtaining a single-view picture and corresponding human body data;
[0009] S2, encoding human body skeleton joints in a parameterized model format according to semantic positions to obtain joint encoding; obtaining two-dimensional human body joint positions in a parameterized model joint format as training data;
[0010] S3, taking a single-view human body picture as input, using a feature extraction network to extract picture features and predict positions of each joint in a two-dimensional picture; extracting features of each joint to obtain joint features;
[0011] S4, in a three-dimensional space, different axis angle vectors can represent the same rotation, in order to eliminate this one-to-many characteristic, defining a sphere with a center as the origin O and a radius of π as a three-dimensional rotation pose space; sampling a group of sampling poses P in the pose space according to a certain resolution;
[0012] S5, constructing an implicit function prediction network, combining the joint encoding in step S1, the joint features in step S3, and each pose in the sampling pose group P in step S4, as input of the implicit function prediction network, to obtain a distance group D of each sampling pose from the actual pose, wherein the rotation vector in the sampling pose group P and the distance scalar in the distance group D are one-to-one corresponding;
[0013] S6, processing the distance group D and comparing with the sampling pose group P to obtain a predicted rotation amount of the joint;
[0014] S7, training the implicit function prediction network; in the prediction stage, inputting a single-view human body picture, extracting features of each joint, combining the joint features and the sampling pose with the joint encoding to input the implicit function prediction network, and calculating according to the processes in steps S5 and S6 to predict the rotation amount of each joint, thereby predicting a three-dimensional human body pose.
[0015] Further, in step S1, a single-view human body picture is directly obtained from a data set, and the human body data includes three-dimensional joint positions, two-dimensional joint positions, and pose parameters, shape parameters, and global translation of a parameterized model.
[0016] Further, in step S2, the human body skeleton joints in a parameterized model format are encoded according to semantic positions, each encoding corresponding to a human body joint position; camera parameters are calculated according to the two-dimensional joint positions and the three-dimensional joint positions; the pose parameters, shape parameters, and global translation of the parameterized model are input into the parameterized model to obtain the parameterized model and its three-dimensional joint positions, the three-dimensional joint positions of the human body model are projected to an image plane using the camera parameters to obtain corresponding two-dimensional joint positions as training data of the implicit function prediction network.
[0017] Further, in the step S3, the method for obtaining each joint feature is:
[0018] Taking a single-view human picture as input, first, the global feature of the picture is extracted using a feature extraction network, and after the global feature of the picture is further processed by a heat map network, the heat map of each joint is obtained, which represents the probability of each joint appearing in each pixel of the picture; for each joint, the heat map and the global feature map are multiplied element by element, and the sum along the width and height directions is obtained to obtain the feature vector of the joint, which is part of the input of the implicit function prediction network.
[0019] Further, in the step S4, the rotation amount of each joint is represented in the form of an axis angle vector, and a point in the three-dimensional pose rotation space represents an axis angle vector, and all possible rotations are contained in the pose rotation space.
[0020] Further, in the step S7, the three-dimensional human pose is represented by the rotation amount of the joint on the human skeleton, and the definition of the human joint is consistent with the parameterized model used, and has the same semantics and connection mode.
[0021] According to the position coding of the semantic position of the human joint, the semantic position of the joint in the human body is represented in a digital form, and then the global feature and the local feature of the picture are accurately and efficiently extracted by using the network. In order to avoid the ambiguity of the axis angle vector in the three-dimensional space, the pose space is defined and sampled in the space; after the sampled pose and the joint position coding and the features are input into the implicit function prediction network, the pose distance field can be obtained; by further processing the distance field, the rotation amount of the human joint can be obtained, and the human pose can be predicted.
[0022] Compared with the prior art, the present application has the following advantages:
[0023] (1) The implicit function method is combined with the deep learning method, which can quickly and accurately estimate the three-dimensional human pose;
[0024] (2) According to the position coding of the semantic position of the human joint, the local feature of each joint is extracted, and the information contained in each joint position is fully mined;
[0025] (3) The pose space is defined, the one-to-many problem of the implicit axis angle vector is solved, and the training efficiency and robustness are improved;
[0026] (4) Compared with the traditional motion capture scheme, the present application estimates the human pose from a single picture, does not need complex equipment, has lower cost, is more convenient and fast, and is less limited by the use of the site. BRIEF DESCRIPTION OF DRAWINGS
[0027] Figure 1 is a flow chart of the method of the present application.
[0028] Figure 2 is a flow chart of the running stage in the embodiment of the present application.
[0029] Figure 3 is a flow chart of obtaining the local features of the joints in the embodiment of the present application.
[0030] Figure 4 is a flow chart of the human pose obtained by the implicit function prediction network prediction and post-processing in the embodiment of the present application.
[0031] Figure 5 is a result chart in the embodiment of the present application. DETAILED DESCRIPTION
[0032] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.
[0033] As shown in Figure 1 , the method for estimating three-dimensional human pose from a single view based on implicit function in the embodiment includes the following steps:
[0034] (1) Download Human3.6M, COCO and MPII data sets. From the data set, directly obtain the single-view human picture, the spatial position of the three-dimensional joint of the human body and the position of the two-dimensional joint on the picture, and the pose, morphological parameters and global translation of the parameterized model SMPL.
[0035] (2) The definition of the two-dimensional joint J 2d and the three-dimensional joint J 3d provided by the data set is inconsistent with the definition of the parameterized model SMPL skeleton. In order to obtain the two-dimensional human joint position in the format of the parameter model joint, three steps are needed, which are calculating the camera parameters, obtaining the three-dimensional joint in the format of the parameter model, and using the camera parameters to perform re-projection and calculate the new joint position. In the first step, the camera parameters include s and t, s is a scalar representing the scale parameter of projection, and t is a three-dimensional scalar representing the position of the projection center point in the world coordinate system (here it is assumed that the camera projection mode is weak perspective projection). The projection formula can be expressed as follows: where p 3d represents a three-dimensional point, p2d denote the two-dimensional projection points. The two parameters can be solved by referring to Procrutes algorithm. After inputting the model parameters and translation provided by the dataset into the parametric model SMPL, the human mesh and three-dimensional human joint J 3d_smpl in the parametric model format can be obtained. 2d_smpl
[0036] In this step, the position of the key points of the SMPL model also needs to be encoded. Here, one-hot encoding is used to assign different codes to different joints. Because the SMPL model has 24 joints in total, the encoding vector of the present embodiment is 24-dimensional. For example, the 0th joint represents the pelvis of the human body, and the code is [1, 0, 0…, 0]; the 2nd joint is the left hip, and the code is [0, 0, 1, 0,…0,]. The joint encoding vectorizes the semantic position of the joint, which will be part of the input of the implicit function prediction network, providing the network with the semantic position information of the joint.
[0037] (3) The purpose of this step is to obtain the two-dimensional local feature of each joint. As shown in Figure 3 , after inputting the monocular image, first use HrNet as the feature extraction network (Sun K, Xiao B, Liu D, et al. Deep high-resolution representation learning for human pose estimation [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019: 5693-5703.) to process the image and obtain the global feature map of the whole image. Then input the global feature map into the heat map network module to obtain the joint heat map (Heat map) of the image. From the heat map, the probability of each joint appearing in each pixel of the image can be obtained. In training, the true value heat map is generated using the two-dimensional joint obtained by the second step of re-projection, and the predicted heat map is supervised using L1 loss.
[0038] For joint i, the element-wise multiplication of the heat map Heatmap i about joint i and the global feature map is performed, and then the sum along the width and height directions of the new feature map is performed, so that the local feature vector of joint i can be obtained (which is essentially a weighted average of the channels of each pixel of the global feature map according to the heat map). For joint i, its local feature can be obtained by the following formula:
[0039] local_feature i = Sum_along_width_and_heeightt(Heatmap i ⊙ global_feature)
[0040] Where the symbol represents element-wise multiplication.
[0041] (4) Pose space and pose space sampling: The rotation of each joint is represented in the form of axis-angle. The axis-angle vector has a repetitive distribution in three-dimensional space, that is, two axis-angle vectors may represent the same rotation. In order to remove this repetition, the embodiment provides that the center of the circle is the origin O and the sphere with a radius of π is the axis-angle space. A point in the three-dimensional pose rotation space represents an axis-angle vector, and all possible rotations are contained in the pose rotation space. In this space, each point represents a unique rotation, representing a specific rotation. For each joint, the pose space is sampled in the form of a three-dimensional grid with a resolution of 50, obtaining n sampling points (n = 50 3 ), respectively p1, p2, … p n , generating a set of sampled poses P, each sampling point being a three-dimensional vector.
[0042] (5) The implicit function prediction network mainly adopts the structure of the MLP network used by PIFu (Saito S, Huang Z, Natsume R, et al. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2019: 2304-2314.), but changes the definition of input and output for the human pose estimation task. As shown in Figure 4 , for each joint, its local feature vector and position encoding are respectively combined with each sampling point to generate a new vector as the input of the implicit function prediction network. For joint i and pose p j , this vector is [local_feature i , one_hot i , p j ], and the output of the network is the distance between the true pose p gt and p j . It should be noted that for each input vector, the last output of the MLP is an "indirect" value dj The predicted true value attitude distance and the sampling attitude, rather than directly output the predicted attitude, which is the embodiment of the hidden function of the method. The loss function of the network is the L2 loss between the predicted distance d j and the actual distance d gt .
[0043] (6) The prediction of the hidden function prediction network for each input is a scalar representing the distance, representing the distance between the sampled attitude of the input joint node and the actual attitude of the joint node in the picture. After inputting the attitudes in the sampling attitude group P into the hidden function prediction network, a group of predicted distances D can be obtained. Because the sampling attitude is sampled according to the three-dimensional grid, the sampling points are arranged in the form of a dot matrix in space. The predicted distance group D is also arranged in this form, and then three-dimensional convolution is used to smooth the predicted distance. The sorted results are sorted to obtain the lowest four distances, and then the corresponding four attitude vectors are found in reverse, and the average of the four attitude vectors is obtained. The predicted vector can be obtained. The purpose of smoothing is to use the prediction information of the surrounding points to reduce the noise of the prediction result. The average calculation of the predicted vector is a comprehensive consideration of several relatively low results.
[0044] (7) Train the above hidden function prediction network; prediction stage: input monocular human body picture, first extract the global feature of the picture and predict the position of each joint node, then use the joint node position to obtain the local feature of the joint node. Then, the local feature, joint node position code and sampling attitude are combined to input the hidden function prediction network, and the human body joint node rotation amount is obtained after post-processing the result, which represents the three-dimensional human body attitude in the picture. The definition of human body joint node is consistent with the parameterized model used, and has the same semantics and connection mode. Figure 5 Some prediction results of the method are shown.
Claims
1. A method for estimating 3D human pose from a single view based on an implicit function, characterized in that, The method comprises the following steps: S1, obtaining a single-view picture and corresponding human body data; S2, encoding human body skeleton joints in a parameterized model format according to semantic positions to obtain joint encoding; obtaining two-dimensional human body joint positions in a parameterized model joint format as training data; S3, taking the single-view human body picture as input, using a feature extraction network to extract picture features and predict the positions of each human body joint in a two-dimensional picture; extracting features of each joint to obtain joint features; S4, in a three-dimensional space, different axis angle vectors can represent the same rotation, in order to eliminate this one-to-many characteristic, defining a sphere with a center at the origin O and a radius of π as a three-dimensional rotation pose space; sampling a group of sampling poses P in the pose space according to a certain resolution; S5, constructing a hidden function prediction network, combining the joint encoding in step S1, the joint features in step S3 and each pose in the sampling pose group P in step S4 as inputs of the hidden function prediction network, to obtain a distance group D of each sampling pose from the actual pose, wherein the rotation vectors in the sampling pose group P and the distance scalars in the distance group D are one-to-one corresponding; S6, processing the distance group D and comparing it with the sampling pose group P to obtain the predicted rotation amount of the joint; S7, training the hidden function prediction network; in the prediction stage, inputting a single-view human body picture, extracting features of each joint, combining the joint features and the sampling pose and the joint encoding as inputs of the hidden function prediction network, and calculating according to the processes in steps S5 and S6 to predict the rotation amount of each joint, thereby predicting a three-dimensional human body pose.
2. The method for estimating 3D human pose from monocular image based on implicit function according to claim 1, characterized in that, In step S1, the single-view human body picture is directly obtained from a data set, and the human body data comprises three-dimensional joint positions, two-dimensional joint positions, pose parameters, shape parameters and global translation of the parameterized model.
3. The method for estimating 3D human pose from monocular image based on implicit function according to claim 2, characterized in that, In step S2, the human body skeleton joints in the parameterized model format are encoded according to semantic positions, each encoding corresponding to a human body joint position; the camera parameters are calculated according to the two-dimensional joint positions and the three-dimensional joint positions; The pose parameters, shape parameters and global translation of the parameterized model are input into the parameterized model to obtain the parameterized model and its three-dimensional joint positions, the three-dimensional joint positions of the human body model are projected to the image plane using the camera parameters to obtain the corresponding two-dimensional joint positions, which are used as training data of the hidden function prediction network.
4. The method of estimating 3D human pose from monocular images based on implicit function according to claim 1, wherein, In step S3, the method for obtaining the features of each joint is as follows: Taking the single-view human body picture as input, first using a feature extraction network to extract global features of the picture, then using a heat map network to further process the picture global features to obtain a heat map of each joint, representing the probability of each joint appearing in each pixel of the picture; for each joint, performing element multiplication operation on its heat map and global feature map, and summing along the width and height directions to obtain a feature vector of the joint, which is used as part of the input of the hidden function prediction network.
5. The method of estimating 3D human pose from monocular view based on implicit function according to claim 1, characterized in that, In the step S4, the rotation amount of each joint is represented in the form of an axis angle vector, and a point in the three-dimensional pose rotation space represents an axis angle vector, and all possible rotations are contained in the pose rotation space.
6. The method of estimating 3D human pose from monocular images based on implicit function according to claim 1, wherein, In the step S7, the three-dimensional human pose is represented by the rotation amount of the joint on the human skeleton, the definition of the human joint is consistent with the used parameterized model, and has the same semantics and connection mode.