Three-dimensional human mesh reconstruction method and system based on graph skeleton attention

Abstract

The present application relates to a kind of methods and systems for reconstructing three-dimensional human mesh based on two-dimensional human pose using graph skeleton attention.The steps of the method include:1) joint characterization, linearly map the input two-dimensional joint into a high-dimensional feature vector;2) joint feature learning, obtain human topological structure information through Skeletal MHA, while GCN strengthens the local information interaction capability, so that the trained model can learn the global and local features of human body, better adapt to human reconstruction task;3) regression of mesh vertex, then add to the mesh template provided based on human model, obtain three-dimensional human mesh output.The present application introduces the prior information of human structure, combines the global perception ability of Transformer and the local aggregation and transmission function of GCN, so that GSAT network can learn at global and local level, thereby reconstructing more accurate three-dimensional human mesh.

Description

Technical Field

[0001] This invention belongs to the field of 3D human body reconstruction and intelligent human-computer interaction in computer vision, specifically involving a method and system for reconstructing a 3D human body mesh using 2D human pose based on graph skeleton attention. Background Technology

[0002] Human body mesh reconstruction has become a hot research topic in computer vision in recent years. Its goal is to reconstruct a 3D human body mesh model using input images or videos. 3D human body mesh models are the foundation of many current real-world applications, such as virtual try-on, virtual reality, augmented reality, and motion capture.

[0003] However, reconstructing 3D human models using 2D input presents numerous challenges: depth ambiguity, as the mapping from 2D to 3D is an inverse problem, and due to the lack of depth information, a 2D pose may satisfy multiple 3D poses, resulting in multiple solutions; occlusion issues, where multiple people occlude each other or a person occludes themselves in real-world scenarios, leading to a loss of information about occluded parts and complicating 3D human reconstruction; and the use of 2D input exposes 3D human reconstruction to classic image processing challenges such as lighting variations, image blurring, and image truncation. Therefore, finding efficient 3D human mesh reconstruction under these challenges and applying it to real-world scenarios is of significant research importance.

[0004] There are two main approaches to 3D human reconstruction based on deep learning: parametric and non-parametric methods. Parametric methods aim to regress the parameters of an existing human model. Human model parameters are divided into pose parameters and shape parameters. Pose parameters control the posture and movement of the 3D human, while shape parameters control its height, build, and overall body shape. Parametric methods simplify the task of 3D human reconstruction and have gained widespread use. However, learning the parameters of a human mesh model from a 2D input is a highly nonlinear problem, which cannot accurately locate the human's position, resulting in low positioning accuracy. To address this issue, non-parametric methods, in contrast to parametric methods, directly predict the position coordinates of each vertex in the human mesh.

[0005] However, both of these methods mostly use 2D images as input, which has the following drawbacks: existing human pose estimation and reconstruction datasets are collected in carefully arranged indoor environments with clean backgrounds, unobstructed human bodies, and common human poses. This is very different from images taken casually in real-world scenes, leading to inter-domain image gaps. Therefore, models trained using such datasets often have poor generalization performance and cannot be well applied to images in real-world scenes. To alleviate this shortcoming, some image-based methods mix multiple datasets to train the model, but this also brings higher training costs and data annotation requirements. The challenges posed by image problems have prompted researchers to start looking at alternative input methods, such as using 2D human joints as input to study their relationship with 3D human meshes, thereby achieving more effective 3D human mesh reconstruction. Summary of the Invention

[0006] This invention utilizes a Graph Skeletal Attention Transformer (GSAT) network to reconstruct a 3D human mesh model from input 2D human poses in an end-to-end manner. By explicitly modeling the structural information of the human skeleton, this invention enables the network to learn at both local and global scales, effectively improving the accuracy and generalization of pose-based 3D human reconstruction methods.

[0007] The technical solution adopted in this invention is as follows:

[0008] A method for reconstructing a 3D human body mesh using 2D human pose based on graph skeleton attention includes the following steps:

[0009] The input two-dimensional joints are mapped to a high-dimensional space to obtain high-dimensional feature vectors;

[0010] The joint position encoding is added to the feature vector to obtain joint features with position information;

[0011] The joint features with location information are input into the Transformer block with graph skeleton attention enhancement to capture global and local human joint features.

[0012] The joint features output by the Transformer block with graph skeleton attention enhancement are subjected to mesh vertex regression and added to the mesh template provided by the human body model to obtain a 3D human body mesh.

[0013] Furthermore, the graph skeleton attention-enhanced Transformer block utilizes the skeleton multi-head attention mechanism Skeletal MHA and the graph convolutional neural network GCN to capture global and local human joint features; wherein the skeleton multi-head attention mechanism Skeletal MHA obtains human topological structure information, while the graph convolutional neural network GCN enhances the local information interaction capability, integrates the information of the two and connects to the forward propagation network FFN to integrate the feature results.

[0014] Furthermore, the Skeletal MHA mechanism encodes the connection relationships between human joints and the prior information of bone length into an attention bias matrix, which is then added to the attention matrix to model the topological structure of the human body.

[0015] Furthermore, the Graph Convolutional Neural Network (GCN) is a parallel module independent of the skeleton multi-head attention mechanism and the forward propagation network. It models human joints as a graph structure, with each joint focusing only on its adjacent joints, in order to aggregate and transmit local joint information, thereby improving the ability to capture local information.

[0016] Furthermore, the MHA of the original Transformer encoder is modified to Skeletal MHA to model the human body topology; and GCN is added at the parallel section of Skeletal MHA and FFN to improve the ability to capture local information; the substructure composed of Skeletal MHA, FFN and GCN is stacked in 6 layers to form the graph skeleton attention-enhanced Transformer block.

[0017] Furthermore, the human body model is an SMPL human body model, etc.

[0018] Furthermore, the aforementioned mesh vertex regression involves upsampling to mesh increments or directly obtaining mesh vertices.

[0019] A system for reconstructing a 3D human body mesh using 2D human pose based on graph skeleton attention, comprising:

[0020] The skeleton encoding module is used to map the input two-dimensional joints into a high-dimensional space to obtain high-dimensional feature vectors;

[0021] The position encoding module is used to add the position encoding of the joint to the feature vector to obtain joint features with position information;

[0022] The joint feature learning module is used to input joint features with positional information into the Transformer block with graph skeleton attention enhancement to capture global and local human joint features.

[0023] The Mesh Vertex Regression module is used to perform mesh vertex regression on the joint features output by the Transformer block with graph skeleton attention enhancement, and add them to the mesh template provided by the human body model to obtain a 3D human body mesh.

[0024] Technical effects of the present invention:

[0025] This invention realizes a method for reconstructing a 3D human body mesh using 2D human pose input. By introducing prior information about human body structure, and combining the global perception capability of Transformer with the local aggregation and transfer function of GCN, the GSAT network can learn at both the global and local levels, thereby reconstructing a more accurate 3D human body mesh. Attached Figure Description

[0026] Figure 1 This is a framework diagram of the overall model of the present invention;

[0027] Figure 2 This is the specific structure of the Skeletal MHA module of the present invention;

[0028] Figure 3 This is the specific structure of the regressor of the present invention;

[0029] Figure 4 This is the visualization result of the three-dimensional human body mesh reconstruction of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] This invention uses the Transformer network as the basic framework and proposes an attention mechanism that conforms to the topological structure of the human body, taking into account the characteristics of the human skeleton and mesh model. The description of Transformer is based on Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, and Aidan N Gomez. Kaiser,and Illia Polosukhin.2017.Attention is all you need.In Advances in NeuralInformation Processing Systems(NeurIPS).

[0032] 5998–6008.

[0033] like Figure 1 As shown, the steps of the method for reconstructing a 3D human mesh using 2D human pose based on graph skeleton attention according to the present invention are as follows:

[0034] 1) Construction of the 3D human body reconstruction model. The basic framework adopts the literature by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Kaiser and Illia Polosukhin, 2017. "Attention is all you need." In Advances in Neural Information Processing Systems (NeurIPS). 5998–6008., the Transformer method is proposed. Furthermore, a Skeletal Multi-Head Attention (SMA) mechanism is introduced, encoding prior knowledge of the connections between human joints and bone lengths into an attention matrix to model the topological information of the human body. A Graph Convolutional Neural Network (GCN) module is added at the parallel points of the Skeletal MHA and FFN modules to aggregate and transmit local joint information. Therefore, the entire model can learn at both global and local levels, thereby reconstructing a more accurate 3D human mesh.

[0035] 2) Joint feature mapping. Using a linear transformation method, the input two-dimensional joints are mapped to a high-dimensional space to obtain high-dimensional vectors, thereby learning the latent features.

[0036] 3) Add the joint position encoding to the feature vector to obtain joint features with position information.

[0037] 4) Joint Feature Learning. The characterized joint vectors are fed into a 6-layer stacked GSAT block. Human topological structure information is obtained through Skeletal MHA, while GCN enhances the ability to interact with local information. The information from both is integrated and connected to FFN (Forward Propagation Network) to integrate the feature results. This allows the trained model to capture both global and local features, making it better suited for human reconstruction tasks. This invention modifies the original Transformer encoder structure by replacing MHA with Skeletal MHA to model human topology; and adds GCN at the parallel points of Skeletal MHA and FFN to improve the ability to capture local information. This substructure is stacked 6 layers to form the GSAT block.

[0038] 5) Regression of Mesh Vertices. The joint features output from the GSAT block are fed into a linear regressor. First, the joint features are upsampled into mesh increments, which are then added to the mesh template provided by the SMPL human body model, finally obtaining the 3D human body mesh output. This step does not directly regress the absolute coordinates of the mesh vertices, but instead calculates the mesh vertex increments first and then adds them to the mesh template provided by the SMPL human body model. The advantage is that it utilizes the prior information of the human body model, and the incremental form is more stable than absolute coordinates.

[0039] One embodiment of the present invention provides a method for reconstructing a three-dimensional human body mesh based on graph skeleton attention using two-dimensional human pose, the steps of which are as follows: Figure 1 Here is the overall framework diagram of GSAT:

[0040] 1) Represent the two-dimensional human pose data P∈R N×2 The input is fed into the skeleton embedding module and linearly mapped to a feature vector X0∈R in a high-dimensional space. N×D Where N is the number of joints and D is the feature dimension. In the model of this embodiment, D = 512.

[0041] 2) Add the joint position encoding to the feature vector X0∈R N×D In the process, joint features X∈ with positional information are obtained.

[0042] R N×D .

[0043] 3) Joint features X∈R N×D Entering the Graph SkeletalAttention Transformer Block, it is called the GSAT block.

[0044] 4) The GSAT block differs from the original Transformer Encoder by adding a Skeletal Multi-head Attention Mechanism (Skeletal MHA) and a Graph Convolutional Network (GCN).

[0045] 5) The self-attention mechanism in traditional Transformers assigns different attention levels and obtains the output by calculating the similarity between different representations of the sequence input, while possessing global perceptual capabilities. However, unlike sequence data, the human body has a more complex structure. Therefore, Skeletal MHA encodes the prior information about the connectivity between human joints and bone length into an attention bias matrix and adds it to the attention matrix, such as... Figure 2 As shown. Figure 2 Spatial coding in this context refers to the encoding of the connection relationships between human joints, while path coding refers to the encoding of bone length.

[0046] The formula for calculating the attention bias matrix of joint connections is: ABS(i,j)=b dis(i,j) ·dis(i,j), where b dis(i,j) It is a learnable coefficient related to the connection distance dis(i,j) between joint i and joint j.

[0047] The formula for calculating the attention bias matrix for bone length is: Represents all bone features from joint i to joint j. The weighted average, where is a learnable weight parameter, and k is the bone number traversed from joint i to joint j.

[0048] Finally, the attention formula enhanced with human skeletal information is: in This is the original Transformer's self-attention formula: Q = XW Q and K = XW K Let d represent two linear transformations of the input feature X, and let d represent the dimension of the input feature X.

[0049] 6) To improve the ability to capture local information, a GCN module was added at the parallel section of the Skeletal MHA and FFN modules to aggregate and transmit local joint information. The formula is as follows:

[0050] X out =σ(AX) in W), where X out X represents the joint features of the output. in The joint features represent the input, A∈{0,1} N×N It is an adjacency matrix, indicating whether joints are directly connected. It is a learnable weight matrix, σ is a non-linear activation function, and D in D represents the dimension of the input joint features. out This indicates the dimension of the output joint features.

[0051] 7) In summary, as Figure 1 Joint features X∈R N×D The calculation process after the GSAT block is represented as follows:

[0052] X′=SMHA(LN(X in ))+GCN(LN(X in ))+X in

[0053] X out =FFN(LN(X′))+GCN(LN(X′))+X′

[0054] Where SMHA(·) is the Skeletal MHA module, GCN(·) is the graph convolution module, FFN(·) is the forward propagation network, and LN represents the Layer Normalization operation.

[0055] 8) The last linear regressor (e.g.) Figure 3 As shown), the output feature X out ∈R N×D Upsampled to X′ out ∈R N ×V Where V = 6890 is the number of vertices in the human body mesh, and then it is linearly projected into the mesh increment ΔM∈R. V×3 And add it to the mesh template M0∈R provided by the SMPL human body model. V×3 The final 3D human body mesh output is obtained as M = M0 + ΔM∈R. V×3 .

[0056] The databases used in the experiments were Human3.6M, 3DPW, COCO, and MuCo-3DHP datasets. Human3.6M is a large indoor human dataset containing 3.6 million frames of video from four cameras, featuring 11 actors performing 15 types of actions, such as walking, eating, and sitting. 3DPW is an outdoor dataset containing 51,000 frames of video with 3D human poses and mesh labels; this dataset was used only for testing to verify the generalization performance of the method. COCO is a dataset containing 2D human pose labels, which, through data augmentation, also obtained 3D human poses and mesh labels; this dataset was used only for training. MuCo-3DHP contains 200,000 frames of video with numerous real-world scenes; similarly, this dataset was also used only for training.

[0057] The visualization results of the 3D human body mesh reconstruction are as follows: Figure 4 As shown in the diagram, the first column is the original image, the second column is the 2D human pose obtained by the 2D pose detector (i.e., the model input), and the third column is the result of projecting the reconstructed 3D mesh onto the image. It can be seen that the image matches the human pose and body shape very well.

[0058] Compared with the original Transformer, the method of the present invention has higher reconstruction accuracy, and the comparison results are shown in Table 1.

[0059] Table 1

[0060] Method MPJPE↓ Transformer 65.43 Transformer+Skeletal MHA 64.59 Transformer+GCN 64.91 Transformer + Skeletal MHA + GCN 64.14

[0061] Among them, MPJPE (Mean Per Joint Position Error) represents the accuracy error. It can be seen that the method of the present invention has the highest accuracy, with a 2.0% improvement over the original Transformer.

[0062] Based on the same inventive concept, another embodiment of the present invention provides a system for reconstructing a three-dimensional human body mesh using two-dimensional human pose based on graph skeleton attention, comprising:

[0063] The skeleton encoding module is used to map the input two-dimensional joints into a high-dimensional space to obtain high-dimensional feature vectors;

[0064] The position encoding module is used to add the position encoding of the joint to the feature vector to obtain joint features with position information;

[0065] The joint feature learning module is used to input joint features with positional information into the Transformer block with graph skeleton attention enhancement to capture global and local human joint features.

[0066] The Mesh Vertex Regression module is used to input the joint features output by the Transformer block with graph skeleton attention enhancement into a linear regressor, upsample them as mesh increments, and add them to the mesh template provided by the human model to obtain a 3D human mesh.

[0067] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smartphone, etc.) including a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes instructions for performing the steps of the method of the present invention.

[0068] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk), wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, it implements the various steps of the method of the present invention.

[0069] The above embodiments of the present invention are illustrated using the SMPL human body model as an example. Other embodiments can be replaced with other human body models (such as SMPL-X, GHUM).

[0070] In the above embodiments of the present invention, the mesh vertex regression module uses a linear regressor to upsample the mesh vertex increments, wherein the linear regressor can be replaced by other calculation methods (such as convolution), and the mesh vertex increments can be replaced by directly obtaining the mesh vertices.

[0071] The above examples are merely illustrative of the present invention. Although preferred embodiments and accompanying drawings have been disclosed for illustrative purposes, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the preferred embodiments and accompanying drawings.

Claims

1. A method for reconstructing a three-dimensional human mesh from a two-dimensional human pose based on graph skeleton attention, the method comprising: The method comprises the following steps: mapping the input two-dimensional joints into a high-dimensional space to obtain a high-dimensional feature vector; adding joint position encoding to the feature vector to obtain joint features with position information; inputting the joint features with position information into a graph skeleton attention enhanced Transformer block to capture global and local human joint features; performing grid vertex regression on the joint features output by the graph skeleton attention enhanced Transformer block and adding them to a grid template provided based on a human body model to obtain a three-dimensional human body grid; The graph skeleton attention enhanced Transformer block uses a skeletal multi-head attention mechanism (Skeletal MHA) and a graph convolutional neural network (GCN) to capture global and local human joint features. The skeletal multi-head attention mechanism (Skeletal MHA) obtains human topological structure information, while the graph convolutional neural network (GCN) enhances local information interaction capabilities, integrates the information of both and connects to a forward propagation network (FFN) to integrate feature results. The skeletal multi-head attention mechanism (Skeletal MHA) encodes the connection relationship between human joints and the prior information of bone length as an attention bias matrix and adds it to the attention matrix to model the topological structure of the human body. The calculation formula of the attention bias matrix of the connection relationship between the human joints is: ABS(i, j) = b dis(i,j) ·dis(i, j), where b dis(i,j) is a learnable coefficient related to the connection distance dis(i, j) between joint i and joint j; the calculation formula of the attention bias matrix of the bone length is: represents the weighted average of all bone features from joint i to joint j, where w ek is a learnable weight parameter; the attention formula reinforced by human skeleton information is: where is the self-attention formula of the original Transformer.

2. The method of claim 1, wherein, The graph convolutional neural network (GCN) is a parallel module independent of the skeletal multi-head attention mechanism and the forward propagation network, models human joints as a graph structure, and each joint only focuses on adjacent joints to aggregate and transmit local joint information, thereby improving the capture ability of local information.

3. The method of claim 1, wherein, The MHA in the original Transformer encoder structure is modified to a Skeletal MHA to model the human topological structure, and a GCN is added in parallel with the Skeletal MHA and FFN to improve the capture ability of local information. The substructure composed of the Skeletal MHA, FFN and GCN is stacked for 6 layers to form the graph skeleton attention enhanced Transformer block.

4. The method of claim 1, wherein, The grid vertex regression is upsampling to grid increments or directly obtaining grid vertices.

5. A system for reconstructing a three-dimensional human mesh from a two-dimensional human pose using graph skeleton attention based on the method of any one of claims 1-4, wherein, The method comprises the following steps: a skeleton encoding module for mapping the input two-dimensional joints into a high-dimensional space to obtain a high-dimensional feature vector; a position encoding module for adding joint position encoding to the feature vector to obtain joint features with position information; a joint feature learning module for inputting the joint features with position information into a graph skeleton attention enhanced Transformer block to capture global and local human joint features; a grid vertex regression module for performing grid vertex regression on the joint features output by the graph skeleton attention enhanced Transformer block and adding them to a grid template provided based on a human body model to obtain a three-dimensional human body grid.

6. An electronic device, comprising: A computer program is stored in a memory and executed by a processor, and the computer program comprises instructions for executing the method of any one of claims 1-4. A computer program is stored in a memory and executed by a processor, and the computer program comprises instructions for executing the method of any one of claims 1-4.

7. A computer readable storage medium characterized in that, The computer readable storage medium stores a computer program, and the computer program, when executed by a computer, implements the method in any one of claims 1-4.

Images