A human pose estimation and mesh recovery method based on skinned multi-person linear model

Abstract

The present invention discloses a human posture estimation and mesh recovery method based on a skinned multi-person linear model. First, a three-dimensional human body model is obtained through a human body mesh recovery algorithm framework. Then, a dense human posture conversion algorithm is used to divide the three-dimensional human body model derived from the human body mesh recovery algorithm, providing a complete spatial coverage of a person while retaining sufficient information about the partial arrangement. Then, a dual mapping is used to obtain a three-dimensional human body model. IUV Projection map, reduce the dimension of the 3D human body model, and use the 3D human skin multi-person linear model data set for diversified simulation. Finally, the posture estimation and mesh recovery are completed by cross-distribution alignment 2D fusion. The present invention improves the data processing capability of complex environments and obtains a more realistic and accurate human body model.

Description

Technical Field

[0001] The present invention relates to a human body model visualization technology in the field of clothing, and in particular to a human body posture estimation and cross-distributed grid recovery method based on a Skinned Multi-Person Linear (SMPL) model. Background Art

[0002] 3D human mesh recovery is widely used in virtual reality, security systems, gaming industry, virtual fitting, etc. Technological progress has broadened the development of the 3D human modeling industry and has led to the emergence of large-scale 3D scanning systems, which rely on laser scanning and depth cameras to obtain 3D human mesh data. However, during the scanning process, it is affected by factors such as the machine's own parameters, the surrounding environment, and complex preprocessing, making it difficult to generate accurate measurement data with high repeatability and low uncertainty.

[0003] With the emergence of the Skinned Multi-Person Linear (SMPL) model, human body modeling is performed by inputting and changing body shape parameters (human height, weight, head-to-body ratio, etc.) and posture parameters (relative angles of joints, human motion posture), simulating the bulges and depressions of muscles during limb movement, and accurately depicting the morphology of muscle stretching and contraction movements. However, the existing Skinned Multi-Person Linear (SMPL) model uses RGB images as input, and the errors in the projection points and the complex structure of the core positions of the human skeleton joints often lead to different degrees of distortion in the reconstructed three-dimensional model. In addition, most of the existing three-dimensional human body modeling methods based on the SMPL model use RGB images as input to solve the three-dimensional features of the human body from two-dimensional data, which increases the difficulty of solving the algorithm on the one hand, and is prone to overfitting and local optimal solutions on the other hand. Summary of the invention

[0004] In view of the problems existing in the existing 3D human body modeling technology, such as missing human body core position parameters, complex measurement system, and inability to adapt to complex and changeable working environments, the present invention proposes a human body posture estimation and mesh recovery method based on a skinned multi-person linear model.

[0005] The present invention first restores the two-dimensional nodes and contours of the human body from the image information collected and preprocessed by the RGB camera, inputs the RGBD image into the human mesh recovery (HMR) algorithm to obtain the posture parameters, body shape parameters, three-dimensional joints and mesh vertices, and then establishes the connection between the three-dimensional human body model of different body parts and the mesh surface by dense human posture conversion (Densepose), and regresses each pixel point to the mesh vertex through double mapping projection. Through the diversified simulation of the data set, the three-dimensional joint points are finally projected to the two-dimensional joint points to predict the final skinned multi-person linear model.

[0006] The present invention comprises the following steps:

[0007] S1. Obtaining a three-dimensional human body model:

[0008] The RGB camera is added with a depth measurement device to collect the color image of the input picture and the corresponding depth image and perform data preprocessing to obtain an RGBD image, restore the two-dimensional joint points and human body contour, and input the RGBD image into the human mesh recovery (HMR) algorithm framework to obtain the pose parameters θ, body shape parameters β and three-dimensional joint j of the skinned multi-person linear model that obeys the normal distribution. 3D The RGB encoder and feature encoder in the HMR algorithm framework can fuse features and extract human contours, thereby deriving a three-dimensional human model from a two-dimensional human image, avoiding the influence of excess background and redundant information on modeling.

[0009] S2. Use the dense pose algorithm to divide the three-dimensional human body model derived from the human body mesh recovery algorithm:

[0010] The correspondence between the 3D human model and the mesh surface is established through dense human pose conversion (Densepose), and the body parts I are defined, where each body part has a corresponding normal map and displacement map (UV) coordinates. Each coordinate regresses the mesh vertex according to the predefined dual mapping projection of the 3D surface space and IUV space. On this basis, a region-based method is used, that is, the image is divided by identifying different regions by utilizing the uniformity of the same area in the image.

[0011] S3. Dimensionality reduction of 3D human body model:

[0012] The mesh vertices are obtained by double mapping projection to generate an IUV map (I represents the specific part of the human body, UV represents the normal and displacement map of the body part, that is, which point on the map is pasted to the corresponding position). The IUV map is projected into the two-dimensional plane as a template to obtain a continuous two-dimensional mapping IUV. The SMPLify-x dataset of three-dimensional human skin multi-person linear model is used for diversified simulation. The camera rotation r∈R 3×3 , dynamic camera translation t∈R 3 As an external parameter, the fixed focal length f∈R 2 As the internal parameter perspective projection, the three-dimensional joint j 3D Projection to 2D joint j 2D .

[0013] S4, cross-distribution alignment 2D fusion pose estimation and mesh recovery:

[0014] Through cross-distribution alignment (using the encoder in the VAE neural network to align the potential distribution of views, and at the same time, in order to reduce the information lost by distribution alignment, cross-alignment is introduced in the decoder reconstruction process to balance the multi-view data) fusion module R refuseThe iterative loop input regressor corrects the predicted posture parameters, body shape parameters, mesh vertices, three-dimensional joints, two-dimensional joints and other parameter errors during the training process (including trainable alignment of the regression mesh output) and infers the skinned multi-person linear model to complete human posture estimation and mesh recovery.

[0015] Compared with the prior art, the beneficial effects of the present invention are:

[0016] (1) The present invention extracts human body size information based on the input image, modifies the geometric structure of the human body model on this basis, and restores the body joints to the three-dimensional model. The root mean square error at the human body joints is about 0.001m.

[0017] (2) The present invention uses an RGB encoder and a feature encoder to perform feature fusion to extract the human body contour, thereby avoiding the influence of excess background and redundant information on modeling and improving the data processing capability for complex environments.

[0018] (3) The present invention adds an adjustment process for the human body shape, introduces skinned multi-person linear model constraints based on actual measurements of the human body structure, and calculates parameter losses. The average error of mesh vertices is close to 0.06m, resulting in a more realistic human body model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Provides the overall framework of the human body mesh recovery algorithm;

[0020] Figure 2 Flowchart for dense human posture conversion;

[0021] Figure 3 Align the grid for cross distribution recovery framework. DETAILED DESCRIPTION

[0022] The present invention will be further described below in conjunction with the accompanying drawings and examples.

[0023] The present invention collects data through an image device, which includes a color camera and a depth camera, and can simultaneously obtain RGB color data and Depth depth data. The dense human pose conversion (Densepose) algorithm is used to establish the correspondence between the two-dimensional image and the mesh surface. The existing skinned multi-person linear model data set SMPLify-x is used for diversified simulation to reduce the dimension of the three-dimensional model to obtain a two-dimensional Gaussian heat map. The final skinned multi-person linear model pose estimation and mesh recovery are completed through cross-distribution alignment. The human pose estimation and mesh recovery method based on the skinned multi-person linear model has the following specific steps:

[0024] Step 1: Add a depth measurement device to the RGB camera to collect the color image and the corresponding depth image of the input image, and perform data preprocessing to obtain the RGBD image. The input image is output through the RGB encoder and the depth encoder. The image features are processed by regression analysis to obtain the camera scaling, rotation, and translation parameters s, r, t, as well as the skinned multi-person linear model posture parameters θ, body shape parameters β and three-dimensional joint positions j 3D The camera parameters and the 3D joint positions of the model are expressed by 2D =sΠ(rj 3D )+t, predict the 2D joint position j by parallel projection 2D The discriminator learns the rotation angle limit of each human joint through training. Where E represents the error between the predicted joint and the original joint, j 2d Representing the two-dimensional joint position of the RGBD image, the model parameters predicted after iteration are input into the discriminator to determine whether the posture of the three-dimensional human body model is reasonable. The specific steps are as follows Figure 1 shown.

[0025] Step 2: Define 24 body parts I (including head, torso, upper and lower arms, upper and lower legs, hands and feet, where the head, torso and upper and lower limbs are divided into upper, lower, front and back to ensure that the body parts and the grid are in the same plane), where each body part has a corresponding texture map (UV) coordinate. The IUV map is denoted as [I,U,V]∈R 3×(P+1)+H×W , where P is the 24 parts of the body, H and W are the height and width of the IUV map, and the index channel I∈{0,1} (P+1)×H×W , where 0 represents the image background and 1 represents the body part index. Each coordinate regresses the mesh vertex according to a predefined dual mapping projection between the 3D surface space and the IUV space. On this basis, a region-based approach is used, that is, the image is partitioned by exploiting the homogeneity of the same area in the image to identify different regions as an intermediate step before mapping to 3D shape and pose. This segmentation provides a complete spatial coverage of a person, rather than the commonly used sparse set of keypoints, while also retaining enough information about the part arrangement to allow efficient lifting to three-dimensional space.

[0026] Step 3: Use the 3D human skinned multi-person linear model dataset SMPLify-x for diversified simulation. Use a dynamically sampled camera t∈R 3 As an external parameter, the fixed focal length f∈R 2 As the perspective projection of internal parameters. 3D , through perspective projection 2D =fΠ(rj 3D +t) Project the three-dimensional joint to the two-dimensional joint j 2D,like Figure 2 shown.

[0027] Step 4: Encoder ResNet-18 extracts 2D joint input and output features Input the iterative regressor and predict the 3D human model Θ = {β, θ, γ} using the posture parameters, body shape parameters, and camera parameters in step 1. Use a multi-layer perceptron (MLP) to reduce the channels while keeping the spatial dimension unchanged to map multiple input data sets to a single output data set. Through the fusion module R refuse Correct the prediction parameter errors during training. Θ final Get predicted vertex v and predicted 3D joints and predicting 2D joints Use the variance uncertainty prediction loss function to evaluate the gap between the predicted value and the true value:

[0028]

[0029] Among them, L2 represents the mean square error, σ v , σ SMPL Represents the mesh vertices, 2D joints, 3D joints, and skinned multi-person linear model parameter weights that are adaptively adjusted during the training process. Human pose estimation and mesh recovery are completed based on the skinned multi-person linear model, such as Figure 3 shown.

[0030] It has been verified that the present invention extracts human body size information based on the input image and restores the body joints to the three-dimensional model. The root mean square error at the human joints is about 0.001m. Combined with the actual measurement of the human body structure, the skinned multi-person linear model constraints are introduced and the parameter loss is calculated. The average error of the mesh vertices is close to 0.06m, resulting in a more realistic model.

[0031] The present invention is not limited to the above-mentioned embodiments. All equivalent changes and modifications made within the scope of application of the present invention should fall within the scope of the present invention.

Claims

1. A human body pose estimation and mesh recovery method based on a skinned multi-person linear model, characterized in that: The steps include: S1. Obtaining a three-dimensional human body model: The RGB camera adds a depth measurement device to collect the color image of the input picture and the corresponding depth image, and performs data preprocessing to obtain an RGBD image; The RGBD image is input into the human mesh recovery HMR algorithm framework to obtain the pose parameters θ, body shape parameters β and three-dimensional joint j of the skinned multi-person linear model that obeys the normal distribution. 3D , realize the export of three-dimensional human body model from two-dimensional human body image; S2. Divide the exported 3D human body model: Through dense human posture transformation, the correspondence between the 3D human body model and the mesh surface is established, and the body parts are defined. Each body part has a corresponding normal map and displacement map UV coordinates. Each coordinate regresses the mesh vertex according to the predefined dual mapping projection of the 3D surface space and IUV space; Using region-based methods, the image is divided by identifying different regions using the uniformity of the same region in the image; S3. Dimensionality reduction of 3D human body model: The mesh vertices are obtained through double mapping projection to generate an IUV map, where I represents the specific part of the human body, UV represents the normal and displacement map of the body part, and the IUV map is projected into a two-dimensional plane as a template to obtain a continuous two-dimensional mapping IUV map; Diversified simulation using 3D human skinned multi-person linear model dataset, using camera rotation r∈R 3×3 and dynamic camera translation t∈R 3 As an external parameter, the fixed focal length f∈R 2 As the internal parameter perspective projection, the three-dimensional joint j 3D Projection to 2D joint j 2D ; S4, cross-distribution alignment 2D fusion pose estimation and mesh recovery: Fusion module R through cross-distribution alignment refuse As the input regressor of the iterative loop, it corrects the parameter errors of the posture parameters, body parameters, mesh vertices, 3D joints and 2D joints predicted during the training process, and infers the skinned multi-person linear model to complete human posture estimation and mesh recovery.

2. The method for human posture estimation and mesh recovery based on skinned multi-person linear model according to claim 1, characterized in that: In the HMR algorithm framework described in S1, the RGB encoder and the feature encoder fuse the features, extract the human body contour, and realize the export of a three-dimensional human body model from a two-dimensional human body image.

3. The method for human posture estimation and mesh recovery based on skinned multi-person linear model according to claim 1, characterized in that: S1 also includes: obtaining camera scaling, rotation and translation parameters through regression analysis, which are used to predict the three-dimensional human body model.

4. The method for human posture estimation and mesh recovery based on skinned multi-person linear model according to claim 1, characterized in that: The body parts described in S2 include the head, torso, upper and lower arms, upper and lower legs, hands and feet, wherein the head, torso and upper and lower limbs are divided into upper, lower, front and back to ensure that the body parts are in the same plane as the grid.

5. The method for human posture estimation and mesh recovery based on skinned multi-person linear model according to claim 1, characterized in that: The cross-distribution alignment described in S4 uses the encoder in the VAE neural network to align the view potential distribution, and introduces cross-alignment to balance the multi-view data during the decoder reconstruction process; The fusion module R refuse Includes trainable alignment of regression grid outputs.

Images