Method for estimating three-dimensional human pose based on scene constraints

Abstract

The application discloses a three-dimensional human body posture estimation method based on scene constraints, which comprises the following steps: using the constructed three-dimensional scene features, learning improved human body root joint position estimation and scene and human body surface contact points through a sparse three-dimensional convolutional neural network; using the predicted human body root joint position and the scene contact points, extracting scene features through a self-attention mechanism and fusing the scene features into a pre-trained monocular three-dimensional human body posture estimation network through a cross-attention mechanism, so as to realize three-dimensional human body posture estimation based on scene constraints. The application solves the depth and occlusion singularity of monocular human body posture estimation, and the problem of slow operation speed caused by the fact that the traditional human body posture estimation under scene constraints must use an optimization algorithm, and improves the speed and accuracy of human body posture estimation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and particularly to a three-dimensional human pose estimation method based on scene constraints. BACKGROUND

[0002] Three-dimensional human pose estimation is a basic technology in the field of virtual reality and human-computer interaction to provide digital content and immersive interactive applications. For application cases that provide pre-scanned three-dimensional geometric information of the scene environment, three-dimensional human pose estimation has higher requirements: the position of the estimation result in the three-dimensional scene needs to be accurate, and the pose and the surface contact of the scene are harmonious. Current three-dimensional human pose estimation schemes considering scene constraints all rely on optimizing the parameterized human model to minimize the predefined loss function terms, which generally include human key point re-projection error, human scene penetration distance and human scene contact point distance. However, in actual use, the design of the loss function and the hyperparameters based on iterative optimization method can easily affect the convergence result of the model, and the convergence time is relatively long and not efficient enough. SUMMARY

[0003] The present application aims at the deficiencies of the prior art, and proposes a three-dimensional human pose estimation method based on scene constraints, which estimates the human pose in a known scene using a single forward propagation of a neural network.

[0004] The present application is achieved by the following technical scheme: a three-dimensional human pose estimation method based on scene constraints, comprising the following steps:

[0005] S1, obtaining human root joint position and scene contact points according to input image and three-dimensional scene point cloud, comprising:

[0006] S1.1, constructing three-dimensional scene features:

[0007] Using the input image to obtain the initial three-dimensional position of the human root joint in the scene coordinate system, selecting the scene points near the initial three-dimensional position and voxelizing, using sparse representation to record the voxel points, and calculating the bias vector pointing to the initial three-dimensional position as feature one for each voxel point;

[0008] Using a two-dimensional convolutional neural network C1 to obtain a two-dimensional feature map of the input image, and using the camera pose to project the voxel center to the two-dimensional feature map to obtain the feature by bilinear interpolation as feature two;

[0009] Using feature one and feature two to construct three-dimensional scene features;

[0010] S1.2, using sparse three-dimensional convolutional neural network C2 to learn and train according to the constructed three-dimensional scene features:

[0011] The learning goal of defining the three-dimensional scene feature is an improved bias vector and its confidence, and the improved human body root joint three-dimensional position is obtained by weighting;

[0012] The learning goal of defining the three-dimensional scene feature is the classification of each scene voxel in contact with the human body, which is restored to the original resolution of the three-dimensional scene point cloud in contact with the human body through the voxelization method.

[0013] S2, based on the human body root joint position and the scene contact point, the human body posture estimation is carried out, including:

[0014] S2.1, using the human body root joint three-dimensional position obtained by S1, the three-dimensional scene point cloud is centralized to eliminate the influence of global position, and the centralized three-dimensional scene point cloud is obtained;

[0015] S2.2, using a convolutional neural network C3 to extract picture features corresponding to the vertices of the template human body mesh from the input picture, and adding the picture features to the human body mesh, denoted as U; using average pooling method to convert the picture features into features corresponding to the centralized three-dimensional scene point cloud, and adding the converted features to the centralized three-dimensional scene point cloud, denoted as V;

[0016] S2.3, using a neural network T1 with multiple layers of self-attention mechanism to process the feature V, and predicting the centralized three-dimensional scene point cloud position coordinates, and recording the intermediate layer features as W;

[0017] S2.4, using another neural network T2 with multiple layers of self-attention mechanism to process the feature U, and using cross-attention method to transfer the feature W to the intermediate layer of the neural network T2, and predicting the human body network containing posture and body shape.

[0018] Further, the convolutional neural network C3 and the neural network T2 with multiple layers of self-attention mechanism constitute a pre-trained monocular three-dimensional human body posture estimation network.

[0019] Further, in S1.1, the initial three-dimensional position of the human body root joint in the scene coordinate system is specifically:

[0020] The static three-dimensional scene point cloud is reconstructed in advance, the subject is in the reconstructed scene, and the picture is taken to obtain the camera pose of the shooting picture in the reconstructed scene;

[0021] The three-dimensional position of the human body root joint in the camera coordinate system is estimated using a monocular human body root joint position estimation method; according to the solved camera pose, the three-dimensional position is transformed into the scene coordinate system to obtain the initial three-dimensional position of the human body root joint in the scene coordinate system.

[0022] Further, in S1.1, the initial three-dimensional position is obtained by estimating the two-dimensional human root joint position and its depth, and the estimation process shares the same two-dimensional convolutional neural network with the extraction of the two-dimensional feature map.

[0023] Further, in S1.1, the initial three-dimensional position is taken as a center point, and two points in front and back of the center point along a ray from the camera optical center to the center point are sampled, three-dimensional scene point cloud sets within a preset radius range are reserved with the three points as the spherical centers, the scene point clouds are voxelized with a certain voxel size, the voxel points are recorded using sparse representation, and a scene sparse voxel set is constructed.

[0024] Further, in S1.2, the classification of each scene voxel in contact with the human body includes seven categories of contact with seven regions of the human body surface and one non-contact category; the voxels predicted as the non-contact category are removed, and the three-dimensional scene point cloud of the original resolution is restored by de-voxelization, and the classification of each point in the three-dimensional scene point cloud is consistent with the classification of the corresponding voxel point.

[0025] The beneficial effects of the present application are:

[0026] 1. The comprehensive use of pictures and three-dimensional scene information can greatly improve the estimation accuracy and speed of the global position and contact points of the human body through sparse three-dimensional convolution learning.

[0027] 2. The predicted three-dimensional scene contact provides a very natural constraint feature learning for the neural network mapping learning of the human body posture through the cross-attention, and solves the depth and occlusion singularity of monocular human body posture estimation.

[0028] 3. The completely single forward neural network propagation solves the problem of slow running speed caused by the use of optimization algorithm in the traditional scene-constrained human body posture estimation, and greatly improves the speed and accuracy of the posture estimation. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0030] Figure 1 The flow chart of the three-dimensional human body posture estimation method based on scene constraint provided by the embodiment of the present application;

[0031] Figure 2This is a schematic diagram of a three-dimensional human pose estimation method based on a single image and scene constraints provided in an embodiment of the present invention.

[0032] Figure 3 This is a schematic diagram of two features of the structure on a voxel provided in an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of seven types of contact points defined in a human body triangular grid, provided for an embodiment of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0035] like Figure 1 , 2 As shown, this embodiment of the invention provides a 3D human pose estimation method based on scene constraints. It uses an input image and a 3D scene to estimate the 3D positions of the human root joints and the scene contact points. It extracts scene features using a self-attention mechanism and fuses these features using a cross-attention mechanism to perform 3D human pose estimation. The specific implementation steps are as follows:

[0036] S1 requires the pre-reconstruction of a static 3D scene point cloud, which can be achieved using any existing 3D reconstruction method. The subject must be within this reconstructed scene.

[0037] S1.1: Take an image of the subject, run an image localization algorithm to obtain the camera pose of the captured image in the pre-reconstructed scene. Use a monocular human root joint position estimation method to estimate the 3D position of the human root joints in the camera coordinate system. Based on the solved camera pose, this 3D position Transform to the scene coordinate system to obtain the initial 3D position of the human root joint in the scene coordinate system. Two-dimensional feature maps of captured images are extracted using a two-dimensional convolutional neural network (CNN).

[0038] Specifically, camera pose can be solved using any existing localization method, including but not limited to pure vision localization and multi-sensor localization methods.

[0039] In particular, the monocular human root joint position estimation method can employ any existing monocular depth estimation method. In this embodiment, by estimating the two-dimensional human root joint position and its depth, the initial three-dimensional position is obtained, and the estimation process shares the same two-dimensional convolutional neural network CNN with the extraction of the two-dimensional feature map.

[0040] S1.2, according to the initial three-dimensional position r as the center point, along the ray from the camera optical center c to the point sampling r world point distance d sample two points, respectively taking the three points as the spherical center, keeping the radius r near within the range of the three-dimensional scene point cloud set P, voxelizing these scene point clouds with a certain voxel size s vox , recording these voxel points using sparse representation, and constructing a scene sparse voxel set V. Wherein r near ≥ d sample , in this embodiment, d sample is 0.75m, r near is 1.25m, and s vox is 5cm.

[0041] According to the initial three-dimensional position r world , the two-dimensional feature map, the camera pose, and the scene sparse voxel set V obtained in S1.1, a three-dimensional scene feature F is constructed, as shown in Figure 3 . In particular, the feature f i of each voxel v i in the set V includes two parts: the first feature is offset i = r world -o i , indicating the offset vector from the voxel center o i to the initial three-dimensional position r world ; the second feature is the feature g i obtained by bilinear interpolation of the voxel center o i projected onto the two-dimensional feature map using the camera pose; the first feature and the second feature are integrated, i.e. f i = {offset i , g i}.

[0042] S1.3, according to the three-dimensional scene feature F constructed on the sparse voxel in S1.2, learning and training are performed using a sparse three-dimensional convolutional neural network CNN.

[0043] The learning goal of the three-dimensional scene feature F is to improve the offset vector and its confidence conf iThe three-dimensional position of the human body root joint is calculated by weighting

[0044] The learning goal of the three-dimensional scene feature F is to classify the contact between each scene voxel and the human body including 7 categories of contact with 7 regions of the human body (such as shown in Figure 4 ) and 1 non-contact category; the voxels predicted as the non-contact category are removed, and the three-dimensional scene point cloud of the original resolution is restored by de-voxelization where J is the number of point clouds, P * The classification of each point is consistent with the classification of the corresponding voxel point, denoted as

[0045] The sparse three-dimensional convolutional neural network CNN in this embodiment can use SPVCNN in the literature [Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse point-voxel convolution. In ECCV, 2020.].

[0046] S2, requires a monocular three-dimensional human pose estimation network Net HMR pre-trained on a large dataset to be prepared in advance HMR First, a convolutional neural network CNN is used to extract image features corresponding to the vertices of the template human mesh from the image The corresponding features are added to the vertex coordinates of the template human mesh Finally, a multi-layer self-attention neural network Transformers is used to predict the human mesh containing the pose and body shape with H' as the input where N is the number of vertices of the human mesh, and D is the feature dimension, N is 431 and D is 1024 in this embodiment.

[0047] S2.1, the three-dimensional position of the human body root joint obtained by S1.3 The three-dimensional scene point cloud P * obtained by S1.3 is centralized to eliminate the influence of the global position, that is, the centralized three-dimensional scene point cloud

[0048] S2.2, using the pre-trained monocular three-dimensional human pose estimation network NetHMR Convolutional Neural Network CNN extracts picture features Using the correspondence between the point cloud and the human surface area contact obtained in S1.3 j , the picture features H are converted into features corresponding to the scene point cloud in an average pooling manner Wherein Wherein equal(contact i , contact j ) is to determine whether the contact category of the human mesh vertex i and the point j in P * is consistent, if consistent, take 1, otherwise take 0, N j is the total number of consistent; the converted feature L is added to the centralized three-dimensional scene point cloud Position coordinates, neural network Net using multi-layer self-attention mechanism extracts features, and finally predicts a set of point clouds with the same position as the centralized three-dimensional scene point cloud Scene . Supervised by L1 loss function.

[0049] The features extracted by the above multi-layer self-attention mechanism neural network Net Scene are transmitted to the monocular three-dimensional human pose estimation network Net HMR through cross-attention method, and finally predict the human mesh containing pose and body shape. Supervised by L1 loss function.

[0050] Specifically, as Figure 2 shown on the right. In this embodiment, the monocular three-dimensional human pose estimation network Net HMR adopts METRO in the literature [Lin, Kevin, Lijuan Wang, and Zicheng Liu. “End-to-end human pose and mesh reconstruction with transformers.” CVPR. 2021.]; The neural network Net Scene is designed using the same multi-layer self-attention neural network Transformers as METRO, and is randomly initialized; The cross-attention method uses standard dot attention, Query uses the features of Net HMR , and Key and Value use the features of Net Scene .

[0051] The above merely describes preferred embodiments of the present application, and the present application is not limited to the above. Any person skilled in the art, without departing from the technical scope of the present application, can make many possible changes and modifications to the technical solutions of the present application, or modify equivalent embodiments with equivalent changes. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application, without departing from the technical scope of the present application, still falls within the protection scope of the technical solutions of the present application.

Claims

1. A three-dimensional human pose estimation method based on scene constraints, characterized in that, The method includes the following steps: S1, based on the input image and 3D scene point cloud, obtains the positions of the human root joints and scene contact points, including: S1.1, Constructing 3D Scene Features: The initial 3D position of the human root joint in the scene coordinate system is obtained using the input image. Scene points near the initial 3D position are selected and voxelized. Voxel points are recorded using sparse representation. For each voxel point, an offset vector pointing to the initial 3D position is calculated as feature one. A two-dimensional feature map of the input image is obtained by using a two-dimensional convolutional neural network C1. The feature obtained by bilinear interpolation by projecting the voxel center onto the two-dimensional feature map using the camera pose is used as feature 2. Construct 3D scene features using Feature 1 and Feature 2; S1.2, Based on the constructed 3D scene features, a sparse 3D convolutional neural network C2 is used for learning and training: The learning objective for the 3D scene features is defined as the improved bias vector and its confidence, and the improved 3D position of the human root joint is obtained by weighting. The second learning objective for defining the features of a 3D scene is to classify the contact between each scene voxel and the human body, and to restore the original resolution 3D scene point cloud and human body contact classification by devoxification. S2, based on the human root joint positions and scene contact points, performs human pose estimation, including: S2.1: Using the three-dimensional position of the human root joint obtained in S1, the three-dimensional scene point cloud is centered to eliminate the influence of the global position, resulting in a centered three-dimensional scene point cloud. S2.2, use convolutional neural network C3 to extract image features that correspond one-to-one with the vertices of the template human body mesh from the input image, and add the image features one-to-one to the human body mesh, denoted as U; use average pooling to transform the image features into features that correspond one-to-one with the centered 3D scene point cloud, and add the transformed features one-to-one to the centered 3D scene point cloud, denoted as V. S2.3, use a multi-layer self-attention neural network T1 to process feature V, predict the centered 3D scene point cloud position coordinates, and denote the intermediate layer features as W; S2.4 uses another multi-layer self-attention mechanism neural network T2 to process feature U. In the intermediate layer, the cross-attention method is used to pass feature W to the intermediate layer of neural network T2, and the predicted human body network including pose and body shape is obtained.

2. The three-dimensional human pose estimation method based on scene constraints according to claim 1, characterized in that, The convolutional neural network C3 and the multi-layer self-attention mechanism neural network T2 constitute a pre-trained monocular 3D human pose estimation network.

3. The three-dimensional human pose estimation method based on scene constraints according to claim 1, characterized in that, In S1.1, obtaining the initial three-dimensional position of the human root joint in the scene coordinate system specifically involves: A static 3D scene point cloud is pre-reconstructed, the subject is placed in the reconstructed scene, an image is taken of the subject, and the camera pose of the captured image in the reconstructed scene is obtained; The three-dimensional position of the human root joint in the camera coordinate system is estimated using a monocular human root joint position estimation method. Based on the solved camera pose, this 3D position is transformed into the scene coordinate system to obtain the initial 3D position of the human root joint in the scene coordinate system.

4. The three-dimensional human pose estimation method based on scene constraints according to claim 1, characterized in that, In S1.1, the initial three-dimensional position is obtained by estimating the position and depth of the two-dimensional human root joints, and this estimation process shares the same two-dimensional convolutional neural network as the extraction of the two-dimensional feature map.

5. The three-dimensional human pose estimation method based on scene constraints according to claim 1, characterized in that, In S1.1, the initial 3D position is taken as the center point. At the same time, along the ray from the camera optical center to the point, two points at a certain distance before and after the point are sampled. With these three points as the center of the sphere, a set of 3D scene point clouds within a preset radius is retained. These scene point clouds are voxelized with a certain voxel size, and these voxel points are recorded using sparse representation to construct a sparse voxel set of the scene.

6. The three-dimensional human pose estimation method based on scene constraints according to claim 1, characterized in that, In S1.2, the classification of each scene voxel in contact with the human body includes 7 categories of contact with 7 areas of the human body surface and 1 non-contact category; voxels predicted as non-contact category are removed, and the original resolution 3D scene point cloud is restored by voxelization, and the classification of each point in the 3D scene point cloud is consistent with the classification of the corresponding voxel point.

Images