A human mesh reconstruction method based on hypergraph attention

Abstract

The application discloses a human body grid reconstruction method based on a hypergraph attention, and proposes a human body grid hierarchical representation based on a hypergraph to form a human body grid representation model containing component semantics, which provides a structural basis for human body grid reconstruction; a Body2Parts feature transfer module is constructed to realize the convergence of component features and the fusion of image information, and information interaction and fusion are performed at a component level, so that high-quality human body reconstruction at a component level can be supported; a Part2Vertices feature transfer module is proposed to realize the transfer of component features and vertex features, and a hypergraph attention is used to refine the vertex-level features, so that the features are transmitted in a component, and the fine human body reconstruction at a grid point level is supported. Based on the hierarchical reconstruction method of the hierarchical human body grid representation model, the application realizes a high-performance trade-off between the three-dimensional human body grid reconstruction accuracy and the calculation cost.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and 3D reconstruction technology, and is used in industries such as virtual reality and human-computer interaction. Specifically, it is a 3D human body mesh reconstruction method based on hypergraph attention. Background Technology

[0002] Reconstructing 3D human body mesh models from 2D images plays a crucial role in human-computer interaction, virtual reality, and computer-aided design. With the development of deep learning technology and the emergence of large databases, many mainstream methods for 3D human body mesh reconstruction have emerged, mainly divided into parametric-based and non-parametric-based methods. Parametric-based methods regress the low-dimensional parameters of the SMPL model to control the generation of the human body mesh. This relies entirely on the model's learning and utilization of pixel information, often leading to inaccurate reconstruction results. To enhance model performance, many methods attempt to introduce human body part segmentation maps or 2D or 3D human poses as intermediate representations to assist in model reconstruction. Non-parametric methods, on the other hand, use mesh vertices as the regression target, avoiding the ill-posedness of low-dimensional parameter-controlled mesh generation. Instead, they utilize the SMPL human body template mesh and model the relationships between vertices containing human body information using graph convolution or Transformer structures, ultimately regressing the 3D positions of the mesh vertices. However, while graph convolution-based methods are efficient at modeling relationships between adjacent vertices, they can only achieve information exchange by increasing the number of network layers for relationships between distant vertices, resulting in low modeling efficiency. Transformer-based methods treat all grid vertices as adjacent nodes and model the relationships between all vertices through a self-attention mechanism. This approach significantly improves the accuracy of reconstruction, but it increases the number of model parameters exponentially, slows down inference speed, and incurs extremely high costs.

[0003] To achieve a high-performance trade-off between accuracy and computational efficiency in human body mesh reconstruction, this invention proposes a hypergraph-based hierarchical representation model of human body mesh, starting from the perspective of rationally modeling the relationships between vertices and utilizing prior information about human body structure. This model enables the model to model the relationships between vertices at the global, local, and individual levels. Simultaneously, based on this representation model, a hypergraph attention-based human body mesh reconstruction method is proposed. This method includes a Body2Parts feature transfer module to model the relationships between parts and to fuse part features and image information. It also includes a Part2Vertices feature transfer module, whose components include a hypergraph-based adaptive feature transfer component to convert part features to vertex features, and a hypergraph attention-based vertex feature refinement component to update vertex features within parts. The hierarchical reconstruction method based on the hierarchical representation model proposed in this invention progressively models the relationships between vertices, ultimately achieving a high-performance trade-off between reconstruction accuracy and computational cost in human body mesh reconstruction. Summary of the Invention

[0004] This invention proposes a 3D human body mesh reconstruction method based on hypergraph attention. It establishes a hierarchical representation of the human body mesh based on hypergraphs to form a human body mesh representation model containing component semantics. This novel representation model provides a structural foundation for human body mesh reconstruction. The Body2Parts feature transfer module is constructed to achieve feature convergence between components and fusion with image information, enabling information interaction and fusion at the component level, supporting high-quality human body reconstruction at the component level. Furthermore, the Part2Vertices feature transfer module is proposed to achieve the transfer of component and vertex features, and hypergraph attention is used to refine vertex-level features. Using vertices as representation units, feature transfer is performed within components, further supporting refined human body reconstruction at the mesh point level. Based on the hierarchical human body mesh representation model, this invention achieves a high-performance trade-off between accuracy and computational cost in 3D human body mesh reconstruction.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] This invention proposes a method for reconstructing a 3D human body mesh from a single image. First, the SMPL standard human body template is decoupled into parts to construct a hierarchical human body mesh representation model. A feature extractor processes the input image to obtain feature maps and generates pixel-level, camera-level, and part-level feature embeddings. A feature encoding module based on hybrid attention is designed to converge features between parts and fuse them with image information. Vertices are hidden within parts to learn coarse-grained global information. Then, a hypergraph-based adaptive feature transfer module is used to transfer part features to vertex features. Next, a hypergraph attention-based feature refinement module updates the vertex features within each part. Finally, a graph regression head and an upsampling layer are used to regress and finely upsample the mesh. The entire invention consists of three stages: construction of a hypergraph-based hierarchical human body representation, feature transfer in Body2Parts, and feature transfer in Part2Vertices.

[0007] The specific implementation process includes the following steps.

[0008] Step 1: The construction stage of the human body layered representation based on the hypergraph is carried out by using the human body joints in the standard state to divide the mesh vertices of the SMPL template into blocks, forming vertex sets corresponding to different parts. The parts serve as the hyperedges of the human body hypergraph, and the vertex set within the parts is the vertex inside the hyperedge. Finally, a mesh layered representation model rich in human body semantics is constructed with body, parts, and vertex levels.

[0009] Step 2: In the feature transfer stage of Body2Parts, a CNN feature extractor is used to extract features from the input image, and pixel-level, camera-level, and part-level feature embeddings are generated based on the pixel feature map and the decoupled human body parts. Then, hybrid attention is used to first aggregate the semantic information and image information between parts, and then fuse the semantic information and pixel information of parts. With the help of the human body mesh hierarchical representation model designed in this invention, the vertices are hidden in the parts, and coarse-grained part-level features are perceived synchronously.

[0010] Step 3: In the feature transfer stage of Part 2 Vertices, a hypergraph-based adaptive feature transfer module is used to transfer part features to vertex features, and a hypergraph attention-based feature refinement module is used to update the vertex features within the part. Finally, a 3D human body mesh is obtained through a graph regression head and an upsampling layer (implemented through a linear layer). Based on the hierarchical representation model of the human body mesh, vertices asynchronously perceive fine-grained vertex-level features within the part.

[0011] Compared with existing technologies, this invention constructs a novel hierarchical representation model for human body meshes. Other methods typically use SMPL templates as graph structures, which do not contain semantic information, and the information transfer between vertices often relies on the topological relationships of triangular meshes. The hierarchical representation model constructed in this invention includes semantic relationships of human body parts, providing the possibility of representing vertices at the global, local, and individual levels. Based on the constructed hierarchical representation, this invention designs a feature transfer module called Body2Parts, which uses a hybrid attention module to aggregate and fuse semantic and pixel information of parts at the part level. This invention also designs a feature transfer module called Part2Vertices, which uses a constructed adaptive hypergraph correlation matrix to achieve the transfer between features at different levels and further updates vertex features within each part. Common human body reconstruction methods rely entirely on SMPL models that do not contain human semantic relationships, which has limitations in modeling vertex relationships containing human information. This invention innovatively constructs a hierarchical representation model and reconstruction method, capturing human information at different scales and progressively completing human body reconstruction. This design effectively enhances the ability to model vertex relationships, thereby achieving a high-performance trade-off between the accuracy and computational cost of human body mesh reconstruction. Attached Figure Description

[0012] Figure 1 This is the overall network framework of the present invention.

[0013] Figure 2 This is a schematic diagram of the internal structure of hybrid attention.

[0014] Figure 3 This is a schematic diagram illustrating the qualitative reconstruction results of the present invention.

[0015] Figure 4 This is a comparison chart of the comparative experiments of this invention on the Human3.6M dataset.

[0016] Figure 5 This is a comparison chart of the comparative experiments of this invention on the 3DPW dataset. Detailed Implementation

[0017] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] The technical solution adopted in this invention is a method for reconstructing a three-dimensional human body mesh from a single image. Specifically, a feature extractor composed of two-dimensional convolution and a regression network composed of graph convolution or Transformer are combined to form a classic three-dimensional human body reconstruction network. This invention reconstructs the human body mesh based on the feature extractor-regression network architecture. The designed model is divided into three stages: the construction stage of the hierarchical representation of the human body mesh, the feature transfer stage of Body2Parts, and the feature transfer stage of Part2Vertices.

[0019] The construction stage of the hierarchical representation of human body mesh: In 3D human body mesh reconstruction, the SMPL human body mesh model is usually used. However, it does not contain semantic information about human body structure. When modeling vertex relationships containing human information, it relies solely on vertex connectivity within the graph topology, progressively increasing the receptive field of vertices. This results in low efficiency in modeling relationships between distant vertices. To address the problem of low efficiency in vertex relationship modeling, this invention designs a hierarchical representation model of human body mesh based on a hypergraph. This model provides the possibility for flexible expression and information transmission of vertices at global, local, and individual levels, thus providing a more effective representational foundation for human body mesh reconstruction methods.

[0020] The general human body mesh model SMPL template mesh generates 6890 mesh vertices controlled by shape and pose parameters. In actual reconstruction, it is usually downsampled to 431 mesh vertices. coarse ={v1,v2,…,v n Here, n takes the value 431. Additionally, a regressor can be used to obtain the 3D joint point Jnt from the mesh. 3D ={Jnt1,Jnt2,…,Jnt m Here, m takes the value 14, representing the number of joints. The body vertex set V is defined with the joints as cluster centers. coarse It can be decoupled into different sets of components, that is, each vertex of the human body mesh is assigned to the nearest cluster center based on its distance from each cluster center.

[0021]

[0022] dist(·) represents Euclidean distance, where The final component set is P = {p1, p2, ..., p...} m This implicitly implies the ownership relationship between vertices and components. Based on the component set and vertex set obtained from the block division, this invention constructs a human hypergraph G = (V coarse ,P).

[0023] Based on the constructed human hypergraph, this invention designs a hierarchical representation model of the human body mesh, which includes three representations of the human body mesh, enabling each vertex to transfer information from global, local, and individual perspectives, defined as:

[0024]

[0025] (2a) represents the general structure of the human body mesh, where each vertex is connected to its neighboring points based on topological relationships, and each vertex independently perceives information about its neighboring nodes; (2b) represents body parts, p j (2c) represents each part of the human body, with each vertex hidden in the component and synchronously sensing global information of the human body; (2c) represents the set of component vertices, representing the relationship between mesh vertices and body components, with each vertex depending on the hyperedge structure and adjacent to other vertices within the component, and asynchronously sensing local information within the body component.

[0026] Body2Parts Feature Transfer Stage: Based on the hierarchical representation model of human body mesh, this invention designs a feature encoding module based on hybrid attention to learn the semantic relationships at the part level and the information fusion between parts and pixel information.

[0027] After inputting a color image, the image feature map is extracted through a CNN block, and its shape is transformed (1×1 convolution and flattening operations) to obtain the pixel-level feature embedding f. pixel ∈R hw×D Where h and w are the height and width of the feature map, respectively, and D is the dimension of the feature; in order to project the 3D skeleton onto the 2D plane to learn the camera projection parameters, a camera feature embedding f is generated. cam ∈R 1×D Furthermore, a set of component-level feature embeddings f associated with the decoupled body components is defined. part ∈R m×D This is used to represent the semantic information of each component.

[0028] To fully learn the semantic information between components and achieve its fusion with pixel information, this invention designs a feature encoder based on hybrid attention, the specific structure of which is as follows: Figure 2As shown. To maintain the positional invariance of pixel-level information and the position of component-level embeddings on the human body structure, this invention adds positional encoding to both, then uses self-attention (SA) to learn camera parameters from the pixel information. Simultaneously, self-attention is used to learn the global correlations between different components. Finally, cross-attention (CA) is used to fuse component information and image information. The process is defined as follows:

[0029] f' cam ,f' pixel ,f' part =CA(SA1([f cam ||f pixel ⊕pos1]),SA2(f part ⊕pos2))(3) where ⊕ represents element-level addition, [,||,] represents concatenation, and pos1 and pos2 represent positional encoding. This invention addresses f' cam ,f' pixel ,f' part After dimensionality reduction, hybrid attention is used again to enhance and aggregate the data, resulting in F. cam ∈R 1×d ,F pixel ∈R hw×d ,F part ∈R m ×d , respectively, correspond to the learned weak perspective projection camera parameters, enhanced pixel features, and fused component features, where d represents the reduced feature dimension.

[0030] Part 2 Vertices Feature Transfer Stage: In order to convert coarse-grained component-level features into fine-grained vertex-level features, and at the same time learn the relationships between vertices within a component, this invention designs an adaptive feature transfer module based on hypergraph and a vertex feature refinement module based on hypergraph attention.

[0031] The correlation matrix of a hypergraph serves as the hub connecting hyperedge and node information, and plays a crucial role in the feature transfer process between components and vertices. The correlation matrix of the human hypergraph constructed in this invention is H. T = [h1;h2;…;h m ],H T ∈R n×m Related to body parts, and to part p j The relevant matrix block is represented as h j ∈R n×1 If vertex v i ∈V coarse ,but on the contrary Simultaneously, in order to learn the differences between vertices within a component, this invention introduces a learnable weight matrix M∈R to the hypergraph association matrix. n×m This is to achieve shallow differences between vertices within a component, namely:

[0032] F v =σ(M⊙H T )F part (4)

[0033] Where F v σ represents the vertex feature, σ is a non-linear function used by softmax to regularize vertex weights, and ⊙ represents element-wise multiplication.

[0034] The initially defined hypergraph weight association matrix can achieve the transfer of component-level features and vertex-level features. However, vertices within a component do not necessarily contribute equally. To further learn the relationships between vertices within a component, this paper introduces hypergraph attention to update the hypergraph association matrix H. Given a vertex v... i Its characteristics are and the hyperedge p associated with that vertex j Its characteristics are Attention score h ij The calculation is as follows:

[0035]

[0036]

[0037] Where Γ is a learnable weight matrix used to update node features, sim(·) is a function used to calculate the similarity between parts and vertices, and a T This is a weight vector used to output a scalar value of similarity. Using the learned association matrix that includes vertex heterogeneity, hypergraph convolution is used to update the features of hyperedges and vertices:

[0038]

[0039] Where D v ∈R n×n The degree matrix represents the number of hyperedges to which each vertex belongs; D e ∈R m×m W represents the degree matrix of a hyperedge, where the diagonal elements represent the number of vertices in each hyperedge; e ∈R m×m This represents the weight matrix of the hyperedge. and These represent the vertex features of the l-th and l+1-th layers, respectively.

[0040] To further learn the vertex connection relationships at component connections, this invention designs a graph regression head that uses a general graph structure and graph convolution to model these relationships, and uses linear layers and predefined upsampling matrices to respectively process V. coarse Perform two upsampling operations to obtain the intermediate grid V. inter and refined grid V fine This serves as the final output of the model in this invention.

[0041] Loss functions: There are five loss functions for networks, namely grid point loss L... V 3D joint loss Two-dimensional joint loss Surface normal vector loss L normal Edge loss L edge First, there are the different levels of mesh refinement used to constrain the output of the reconstructed model and the corresponding ground truth V. * Grid point loss L V :

[0042]

[0043] Where n, n2, and n3 correspond to the number of vertices in meshes of different refinement levels, and ||||1 is used to calculate the L1 norm distance. Then, the L1 distance is used to constrain the refinement mesh V. fine The three-dimensional joint points J calculated by the predefined linear regressor 3D ∈R m×3 And its true value, and the two-dimensional joint J calculated from the three-dimensional joints and camera projection parameters. 2D ∈R m×2 And its truth value:

[0044]

[0045]

[0046] Then, the surface normal vector loss L normal To enhance the smoothness of the mesh surface, edge loss L edge This improves the smoothness of densely populated vertex areas (such as hands and feet). The formula is defined as:

[0047]

[0048]

[0049] Where f represents the mesh surface, n f v is the unit normal vector of f. i and v j These are the coordinates of vertices i and j, respectively. The total loss function is:

[0050]

[0051] Where λ1, λ2, λ3, λ4, and λ5 are the weights of each loss function. In this invention, λ1 and λ3 are set to 100.0, λ2 is set to 1000.0, λ4 is set to 5.0, and λ5 is set to 0.1.

[0052] Experimental Section

[0053] Experimental datasets: This invention was trained on a hybrid dataset of 3D datasets Human3.6M, UP-3D, and MuCo-3DHP, and 2D datasets COCO and MPII. The model was tested on Human3.6M and 3DPW datasets, and was used to fine-tune the model when testing on 3DPW.

[0054] Human3.6M is a 3D human pose estimation dataset containing 17 general indoor scenes from multiple perspectives. The typical setup uses objects S1, S5, S6, S7, and S8 as training objects and S9 and S11 as testing objects. This invention uses the P2 protocol to demonstrate the model's results. The UP-3D dataset contains 7k images for training, encompassing a wide variety of activities. It obtains ground truth SMPL parameters by fitting 2D keypoints to SMPL. MuCo-3DHP is a synthetic and augmented dataset based on MPI-INF-3DHP, containing 200K images, each typically containing 1 to 4 objects, covering many real-world scenes of human occlusion and interaction. COCO and MPII are commonly used 2D human pose estimation datasets, containing rich motion poses and ground truth 2D human keypoints, but without 3D annotations. The images in the 3DPW dataset were captured in complex scenes, featuring 7 actors performing a variety of activities, and provide accurate ground truth mesh values.

[0055] Evaluation Metrics: This invention employs three evaluation metrics to assess the results, all in millimeters (mm). MPJPE evaluates the average Euclidean distance between the predicted 3D joints and their ground truth values. PA-MPJPE is calculated after aligning the 3D joints using Procrustes analysis. MPVPE evaluates the Euclidean distance between the predicted mesh vertices and their ground truth values. For all three metrics, a smaller value indicates a better reconstruction.

[0056] Experimental Setup: This invention was built using the PyTorch framework and trained on an RTXA600048G. In all experiments, the input images were cropped and resized to 224×224 based on the human bounding boxes. During the training phase, we used ResNet50 or HRNet-W64 pre-trained on ImageNet as the CNN module. We chose the AdamW optimizer to optimize the objective function and set the initial learning rate to 1e-4.

[0057] The model's performance was compared with state-of-the-art single-image-based 3D human reconstruction methods. Table 1 shows the quantitative comparison results on the 3DPW and Human3.6M datasets. The present invention provides a schematic diagram of the model's qualitative reconstruction results, as shown below. Figure 3 As shown. Simultaneously, the qualitative reconstruction results of this invention on Human3.6M and 3DPW are compared with those of advanced methods, as shown... Figure 4 , Figure 5 As shown.

[0058] Ablation Experiments: This invention uses ResNet50 as the CNN block and conducts ablation experiments on the Human3.6M dataset to test the contributions of different network components, as shown in Table 2. Based on the model, the effects of removing the Body2Parts and Part2Vertices modules, removing the hypergraph attention module and using adaptive feature transfer, and removing hypergraph attention and using fixed feature transfer are tested. Body2Parts and Part2Vertices are denoted by B2P & P2V, and hypergraph attention is denoted by HGAT. Table 3 shows a comparison of our method with state-of-the-art methods using graph convolution and Transformer in terms of parameter count and reconstruction results.

[0059] Table 13 compares the accuracy of different methods for human mesh reconstruction based on single-view images using the DPW and Human3.6M datasets (bold text indicates the best results, and ↓ indicates that lower values ​​result in better reconstruction).

[0060]

[0061] Table 2 Ablation experiments of this invention on the Human 3.6M dataset

[0062]

[0063] Table 3 compares the parameter quantities and reconstruction results of this invention with some advanced methods on Human 3.6M.

[0064]

[0065]

Claims

1. A method for human body mesh reconstruction based on hypergraph attention, characterized in that, First, the SMPL standard human body template is decoupled into components to construct a hierarchical human body mesh representation model. A feature extractor processes the input image to obtain feature maps and generates pixel-level, camera-level, and component-level feature embeddings. A feature encoding module based on hybrid attention is designed to converge features between components and fuse them with image information. Vertices are hidden within components to learn coarse-grained global information. Then, a hypergraph-based adaptive feature transfer module is used to transfer component features to vertex features. Next, a hypergraph attention-based feature refinement module updates the vertex features within each component. Finally, a graph regression head and an upsampling layer are used to regress and finely upsample the mesh. The specific steps include: Step 1: Using the human body joints in the standard state, divide the mesh vertices of the SMPL template into blocks to form vertex sets corresponding to different parts. The parts serve as the hyperedges of the human body hypergraph, and the vertex sets within the parts serve as the vertices within the hyperedges, thus constructing a mesh hierarchical representation model rich in human semantics at the body, parts, and vertex levels. Step 2: Use a CNN feature extractor to extract features from the input image to obtain pixel-level feature embeddings. and camera projection feature embedding Where h and w are the height and width of the feature map, respectively, and D is the dimension of the feature; define a set of part-level feature embeddings related to the decoupled body parts. To represent the semantic information of each component; positional encoding is added to pixel-level feature embedding and camera projection feature embedding respectively, and a feature encoder based on self-attention and cross-attention is constructed. The processing of the feature encoder includes: first based on the formula: Calculations are performed, in which, This represents element-wise addition. Indicates splicing, and The position is encoded, and then hybrid attention is used again on the dimensionality-reduced... Enhancement and aggregation are performed to obtain the learned weak perspective projection camera parameters. Enhanced pixel features Features of the fused components , where d represents the reduced feature dimension; Step 3: Construct the human hypergraph association matrix Related to body parts, related to parts The relevant matrix block is represented as If the vertex ,but ,on the contrary Introduce a learnable weight matrix to the hypergraph incidence matrix. The vertex features are calculated using the following formula: ,in Representing vertex features, It is a non-linear function used by softmax to regularize vertex weights. This represents element-wise multiplication; the hypergraph association matrix is ​​updated through hypergraph attention; using the learned hypergraph association matrix that includes vertex heterogeneity, hypergraph convolution is used to update the features of hyperedges and vertices; a graph regression head is constructed to further learn the vertex connectivity at component connections, specifically: graph structures and graph convolutions are used to model and learn the vertex connectivity at component connections, and linear layers and predefined upsampling matrices are used to respectively... Perform two upsampling operations to obtain the grid. and fine mesh fine mesh This is the final result.

2. The human body mesh reconstruction method based on hypergraph attention according to claim 1, characterized in that, In the construction phase of the human body hierarchical representation based on hypergraph, the mesh vertices of the SMPL template are divided into blocks using human body joints in the standard state, forming vertex sets corresponding to different parts. The parts serve as the hyperedges of the human body hypergraph, and the vertex set within the parts is the vertex within the hyperedge. Finally, a mesh hierarchical representation model rich in human body semantics is constructed with body, parts, and vertex levels.

3. The human body mesh reconstruction method based on hypergraph attention according to claim 2, characterized in that, The input image is used to extract features using a CNN feature extractor. Pixel-level, camera-level, and part-level feature embeddings are generated based on pixel feature maps and decoupled human body parts. Then, hybrid attention is used to aggregate semantic and image information between parts separately, and then the semantic and pixel information of the parts are fused. With the help of a hierarchical human body mesh representation model, vertices are hidden in human body parts, and coarse-grained part-level features are perceived simultaneously.

4. The human body mesh reconstruction method based on hypergraph attention according to claim 3, characterized in that, The adaptive feature transfer module based on hypergraph is used to transfer part features to vertex features, and the feature refinement module based on hypergraph attention is used to update the fixed-point features within the part. Finally, a 3D human body mesh is obtained through a graph regression head and an upsampling layer. Based on the human body mesh hierarchical representation model, vertices asynchronously perceive fine-grained vertex-level features within the component.

Images