A human body mesh reconstruction method based on pose transformation

Abstract

The present invention relates to the field of computer vision technology, and discloses a human body mesh reconstruction method based on posture transformation, which transforms the posture from an initial posture to a target posture through a neural network. The present invention solves the problems of low accuracy in the prior art.

Description

Technical Field

[0001] The invention relates to the technical field of computer vision, and in particular to a human body mesh reconstruction method based on posture transformation. Background Art

[0002] Human mesh recovery (HMR) aims to estimate the 3D human body from 2D images, which is a challenging task due to the inherent ambiguity in converting 2D observations to 3D space. Existing research on recovering human mesh from a single image can be divided into two categories: model-based methods and model-free methods.

[0003] Model-based methods regress the parameters of statistical models, mainly understanding the semantic information of the image. However, the transformation from image to abstract parameter space is highly nonlinear, resulting in a rough alignment of modeling results and image evidence. Some works introduced different intermediate expressions to reduce the complexity of model learning, such as 2D / 3D bone points, IUV mapping, and labeling. Recently, model-free methods directly regress the vertices of the mesh, which has excellent flexibility and achieves better pixel alignment than model-based methods. However, due to the lack of depth information from a single image, it is difficult to recover reasonable results from side views.

[0004] Diffusion models have gained popularity as generative models and have proven to be effective in generating arbitrary high-quality images. These models gradually add Gaussian noise to the original data and subsequently treat the generation process as a denoising task. In addition to image generation, diffusion models have also been applied as new learning frameworks to other tasks such as image classification, segmentation, and 3D vision. Diff-HMR and EgoHMR generate the parameters of SMPL through a diffusion model, while HMDiff directly adds noise to the vertices of the mesh. However, after the introduction of noise, the intermediate mesh loses its reasonable physical distribution, which poses a challenge to achieving pixel alignment or obtaining smooth results. Therefore, 3D modeling of characters from 2D images remains challenging. Summary of the invention

[0005] In order to overcome the deficiencies of the prior art, the present invention provides a human body mesh reconstruction method based on posture transformation, which solves the problems of low accuracy and the like in the prior art.

[0006] The technical solution adopted by the present invention to solve the above problems is:

[0007] A human body mesh reconstruction method based on pose transformation realizes the pose transformation from initial pose to target pose through neural network.

[0008] As a preferred technical solution, it includes a kinematics-based forward process and a posture inverse process: the initial posture information is stored in an SMPL template grid, the target posture information is stored in a label grid, and the label grid is interpolated to the SMPL template grid in the kinematics-based forward process; the neural network is a deep neural network, and the deep neural network learns the posture transformation from the initial posture to the target posture in the posture inverse process, and adjusts the vertex position of the SMPL template grid in the inference process to align the posture after posture transformation with the posture of the input image.

[0009] As a preferred technical solution, the label grid is linearly interpolated to the SMPL template grid through skeletal animation in a kinematics-based forward process.

[0010] As a preferred technical solution, the kinematics-based forward process uses linear interpolation of the effects of SMPL parameters on vertex offsets on the shape.

[0011] As a preferred technical solution, the linear interpolation process of the forward kinematics forward process is expressed as:

[0012]

[0013] in, represents the posture at time t, represents the initial mesh model of SMPL, represents the change in attitude at time t relative to time t-1, W(·) represents the standard linear mixed skinning function, represents the vertex position after applying the parametric blend shape deformation, J t represents the joint position after shape deformation, θ t represents the attitude change rotation matrix, represents the mixing weight.

[0014] As a preferred technical solution, The expression is:

[0015]

[0016] in, represents the label grid model, t represents the current time, T represents the total number of iterations, β0 represents the shape parameter, B S (β0) represents the vertex displacement caused by the shape, θ0 represents the posture parameter, and B P (θ0) represents the vertex displacement caused by the bone change.

[0017] As a preferred technical solution, in the posture inversion process, a deep neural network uses a mesh-to-pose decoder to learn the posture transformation from the SMPL template mesh to the label mesh. The mesh-to-pose decoder introduces the 3D mesh and 2D mesh constraints estimated from the input image, and the mesh-to-pose decoder captures the interaction information between mesh vertices.

[0018] As a preferred technical solution, the posture reversal process is implemented in the following way:

[0019]

[0020] in, Represents the grid model at different times, Indicates the changes in posture at different times. represents the SMPL template grid model, k represents different moments, T represents the total number of iterations, represents the grid model at time k-1, represents the grid model at time k, represents the input 2D mesh model, Represents the posture change from time k-1 to time k.

[0021] As a preferred technical solution, the grid-to-pose decoder includes a single-layer perceptron head and multiple graphic blocks connected in series. The graphic blocks include an attention layer, a graph module layer, a feedforward layer, and a residual connection layer connected in series. The residual connection layer of the last graphic block is connected to the single-layer perceptron head.

[0022] As a preferred technical solution, the loss function of the deep neural network is:

[0023]

[0024] in, represents the vertex error loss, λ v represents the vertex error loss weight, represents the error loss of skeleton key points, λ j represents the error loss weight of the skeleton key points, represents the mesh surface error loss, λ s Represents the mesh surface error loss weight.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] (1) The present invention proposes a solution for modeling a 3D human figure based on posture conversion, and utilizes a diffusion-based learning process to improve the consistency between the modeling result and the image.

[0027] (2) The present invention can achieve fast reconstruction with fewer steps by converting from the initial posture of SMPL to the current posture; combined with the human body physical structure preserved in the intermediate state, more rule constraints are imposed on the network during the learning process, which improves the smoothness of the modeling results;

[0028] (3) The present invention proposes an edge-aware guidance scheme, which can effectively solve the problem of low attention paid to local area modeling by the model and improve the accuracy of character edge modeling;

[0029] (4) Experiments on three public datasets show that the proposed scheme achieves encouraging performance and outperforms the most advanced methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of the architecture of a human body mesh reconstruction method based on posture transformation according to the present invention;

[0031] Figure 2 Decompose schematics for shapes and poses;

[0032] Figure 3 Schematic diagram of the structure of a deep neural network. DETAILED DESCRIPTION

[0033] The present invention will be further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0034] Example 1

[0035] like Figures 1 to 3 As shown in the figure, in order to solve the problem of large gap between the initial pose model and the pose corresponding to the modeled person, the present invention proposes a pose transformation architecture for 3D human mesh recovery. It realizes the transformation from SMPL template mesh to label mesh according to the input 2D image. The architecture adopts diffusion learning and has a kinematics-based forward process and a pose reverse process.

[0036] In order to solve the problem of inaccurate pose modeling, skeletal animation based on forward kinematics is used to capture the human mesh pose trend in the forward process. Based on the 2D mesh, a mesh-to-pose decoder is designed to learn the pose transformation in the reverse process by guiding the 3D mesh gradually towards the target pose.

[0037] In order to solve the problem of poor smoothness of modeling results, the model training of the present invention comprehensively considers three factors (posture change, human body structure and pixel alignment), which correspond to posture loss, 3D reconstruction loss and three-image guided rendering loss, respectively, and jointly supervise the modeling results.

[0038] This paper proposes a novel human body mesh reconstruction scheme for learning pose transformation from a standard SMPL template mesh to a label mesh. It mainly includes a kinematics-based forward process and a pose reverse process. In the training phase, the proposed kinematics-based forward process uses skeletal animation to linearly interpolate the label mesh to the SMPL template mesh, providing a supervisory signal for the deep learning network in the reverse process. The deep learning network is a reverse process that gradually learns from the SMPL template mesh to the label mesh. In the inference process, the vertex positions of the SMPL template mesh are gradually adjusted to align with the pose corresponding to the input image.

[0039] Specifically, in order to preserve the mesh topology, a forward kinematics (FK) based approach is proposed in the forward process, which uses skeletal animation to perform pose transformation and linear interpolation in a low-dimensional shape space. To alleviate the abstraction of modeling pose from the image in the reverse process, a mesh-to-pose (M2P) decoder is designed, which introduces the 3D mesh estimated from the input image and the 2D mesh constraints. The decoder gradually captures the non-local and neighborhood interaction information between mesh vertices. In addition, a ternary-guided rendering loss is provided to address the pixel-level misalignment problem with the input image. As a kind of supervision, this loss enhances the model's attention to poorly recognized areas, thereby promoting better alignment. By combining this strategy with the pose loss and the 3D reconstruction loss, the entire model gradually optimizes the pose transformation to achieve pixel-aligned and smooth results.

[0040] The present invention has the following characteristics:

[0041] 1. The present invention proposes a solution for character 3D modeling based on posture conversion, and utilizes a diffusion learning process to improve the consistency expression between the modeling result and the image;

[0042] 2. The present invention can achieve fast reconstruction with fewer steps by converting the template mesh pose of SMPL to the label mesh pose; combined with the human body physical structure preserved in the intermediate state, more rule constraints are imposed on the network during the learning process, which improves the smoothness of the modeling results;

[0043] 3. The present invention proposes an edge-aware guidance scheme, which can effectively improve the problem of low attention paid to local area modeling by the model and improve the accuracy of character edge modeling.

[0044] like Figures 1 to 3 As shown, as a further optimization of Example 1, based on Example 1, this embodiment also includes the following technical features:

[0045] The goal of this scheme is to reconstruct the 3D model of the human body in the image given a template human body mesh and an input image. The main process includes a multi-step diffusion process, which consists of a kinematics-based forward process and a posture reverse process. The overall architecture is as follows Figure 1 shown.

[0046] 1. Forward kinematics forward process

[0047] The invention defines a 3D mesh from the SMPL human body statistical model, with an initial state of 6890 vertices as the input to the posture decoder of the network. The 2D mesh extracts a 2D heat map through a deep neural network, and then regresses to obtain 431 key points.

[0048] The kinematics-based forward process uses a novel diffusion-based pose transformation method to achieve pose transformation from the template mesh to the label mesh of the input image through skeletal animation interpolation. The forward process provides supervision guidance for model learning in the backward process, which involves smooth transitions from the T pose to the target pose. The change of posture is mainly affected by two factors: posture and shape. Therefore, the posture and properties are interpolated separately. The shape and posture decomposition are as follows Figure 2 shown.

[0049]

[0050] in, represents the posture at time t, represents the initial mesh model of SMPL, represents the pose change relative to t-1, and W(·) represents the standard linear mixed skinning function. represents the vertex position after applying the parametric blend shape deformation, J t represents the joint position after shape deformation, θ t represents the attitude change rotation matrix, represents the mixing weight. The expression is:

[0051]

[0052] in, represents the label grid, t represents the current time, T represents the total number of iterations, β0 represents the shape parameter, B S (β0) represents the vertex displacement caused by the shape, θ0 represents the posture parameter, and B P (θ0) represents the vertex displacement caused by the bone change.

[0053] 2. Posture Reversal Process in 2D

[0054] The deep neural network learns the posture transformation from the T-pose mesh to the target mesh in the posture inversion process. The present invention designs a mesh-to-pose (M2P) decoder to learn the posture transformation between different iterative steps. Although it is convenient to use image features as conditional information to reconstruct a specific input image, this reconstruction process is very difficult because the transformation from image input directly to posture is highly nonlinear. The present invention uses a 2D mesh as the guiding information for updating the 3D mesh. It aims to achieve an alignment result with the input image. Formally, given the SMPL initial model and the corresponding constraint information, the posture inversion process can be implemented in the following way:

[0055]

[0056] in, Represents the grid model at different times, Indicates the changes in posture at different times. represents the SMPL template grid, k represents different moments, T represents the total number of iterations, represents the grid model at time k-1, represents the grid model at time k, represents the input 2D mesh model, Represents the posture change from time k-1 to time k.

[0057] The structure of a deep neural network is as follows Figure 3 As shown in the figure. In order to guide posture learning in the posture inversion process, the 2D grid is used as additional information and directly connected to the 3D grid as input. The input information is encoded, position encoding and time encoding are added, and then input to the grid-to-pose decoder; the grid-to-pose decoder includes a single-layer perceptron head, multiple sequentially connected graphic blocks, and the graphic blocks include sequentially connected attention layers, graph module layers, feedforward layers, and residual connection layers. The residual connection layer of the last graphic block is connected to the single-layer perceptron head.

[0058] 3. Loss Function Design

[0059] The loss function of a deep neural network is:

[0060]

[0061] in, represents the vertex error loss, λ v represents the vertex error loss weight, represents the error loss of skeleton key points, λ j represents the error loss weight of the skeleton key points, represents the mesh surface error loss, λ s Represents the mesh surface error loss weight.

[0062] By adopting the technical solution of the present invention, the following technical effects are achieved:

[0063] To demonstrate the effectiveness of the proposed method, we validated it on the publicly available Human3.6M, 3DPW, and SURREAL datasets.

[0064] The model we proposed uniformly crops the input image to a resolution of 256. Similar to other works, it combines MPIINF-3DHP, UP-3D and COCO datasets as additional augmented data to improve the generalization ability of the model. A total of 80 rounds of training were conducted, and the learning rate was set to 0.0005 and 0.00005, corresponding to the learning of 2D grids and 3D grids respectively. After 30 rounds of learning, the learning rate was reduced by half. From the comparison results in Tables 1-3, it can be found that the method we proposed surpasses all previous methods and obtains the best performance, which proves the effectiveness of the proposed scheme in solving the task of modeling human 3D grids from a single image.

[0065] Table 1 Comparison of experimental results of Human3.6M dataset

[0066]

[0067] Table 2: Comparison of experimental results of 3DPW dataset

[0068]

[0069] Table 3: Comparison of experimental results of SURREAL dataset

[0070]

[0071] As described above, the present invention can be preferably implemented.

[0072] All features disclosed in all embodiments in this specification, or steps in all methods or processes implicitly disclosed, except for mutually exclusive features and / or steps, can be combined and / or expanded or replaced in any manner.

[0073] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. According to the technical essence of the present invention, within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. A human body mesh reconstruction method based on posture transformation, characterized in that: The posture transformation from the initial posture to the target posture is achieved through a neural network; The method includes a kinematics-based forward process and a posture inverse process: the initial posture information is stored in the SMPL template grid, the target posture information is stored in the label grid, and the label grid is interpolated to the SMPL template grid in the kinematics-based forward process; the neural network is a deep neural network, and the deep neural network learns the posture transformation from the initial posture to the target posture in the posture inverse process, and adjusts the vertex position of the SMPL template grid in the inference process to align the posture after posture transformation with the posture of the input image; Linearly interpolate the label mesh to the SMPL template mesh via skeletal animation in a kinematics-based forward pass; The kinematics-based forward process uses linear interpolation of the effects of SMPL parameters on vertex offsets on the shape; The linear interpolation process of the forward kinematics forward process is expressed as: in, represents the posture at time t, represents the initial mesh model of SMPL, represents the change in attitude at time t relative to time t-1, W(·) represents the standard linear mixed skinning function, represents the vertex position after applying the parametric blend shape deformation, J t represents the joint position after shape deformation, θ t represents the posture change rotation matrix, W represents the mixing weight; The expression is: in, represents the label grid model, t represents the current time, T represents the total number of iterations, β0 represents the shape parameter, B S (β0) represents the vertex displacement caused by the shape, θ0 represents the posture parameter, and B P (θ0) represents the vertex displacement caused by the bone change; In the pose inversion process, a deep neural network uses a mesh-to-pose decoder to learn the pose transformation from the SMPL template mesh to the label mesh. The mesh-to-pose decoder introduces the 3D mesh and 2D mesh constraints estimated from the input image. The mesh-to-pose decoder captures the interaction information between mesh vertices. The posture reversal process is achieved in the following way: in, Represents the grid model at different times, Indicates the changes in posture at different times. represents the SMPL template grid model, k represents different moments, T represents the total number of iterations, represents the grid model at time k-1, represents the grid model at time k, represents the input 2D mesh model, Represents the posture change from time k-1 to time k.

2. The method for human body mesh reconstruction based on posture transformation according to claim 1, characterized in that: The grid-to-pose decoder includes a single-layer perceptron head, multiple graphics blocks connected in series, and the graphics blocks include an attention layer, a graph module layer, a feedforward layer, and a residual connection layer connected in series. The residual connection layer of the last graphics block is connected to the single-layer perceptron head.

3. A human body mesh reconstruction method based on posture transformation according to claim 1 or 2, characterized in that: The loss function of a deep neural network is: in, represents the vertex error loss, λ v represents the vertex error loss weight, represents the error loss of skeleton key points, λ j represents the error loss weight of the skeleton key points, represents the mesh surface error loss, λ s Represents the mesh surface error loss weight.

Images