Three-dimensional mesh reconstruction method and system based on single image under deep learning framework

By using graph convolution module and Transformer encoding module to process local and global information in a deep learning framework, the problem of non-Euclidean structure data processing in single-image three-dimensional grid reconstruction is solved, and the reconstruction accuracy is significantly improved.

CN114140601BActive Publication Date: 2024-11-12HANGZHOU NORMAL UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111522113.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-13
Publication Date
2024-11-12
Estimated Expiration
2041-12-13

AI Technical Summary

Technical Problem

In the prior art, in three-dimensional grid reconstruction based on single-image images, it is difficult to effectively process grid data of non-Euclidean structures, resulting in insufficient reconstruction accuracy.

Method used

The graph convolution module is used to process local grid features, and the Transformer encoding module is cascaded to supplement global information to build a deep learning framework to improve the accuracy of three-dimensional grid reconstruction.

Benefits of technology

Effective processing of non-Euclidean grid data improves the accuracy and accuracy of three-dimensional grid reconstruction in single image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114140601B_ABST
    Figure CN114140601B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and system for 3D mesh reconstruction based on a single image under a deep learning framework. The present invention uses a VGG16 network as an image feature extraction network to obtain three scale feature maps. Based on the obtained features and an initial elliptical mesh model, a mesh deformation network is applied to complete the complete reconstruction task. The mesh deformation network includes a graph convolution module to fuse local feature information, and a Transformer encoding module to supplement global feature information. The method of the present invention effectively improves the accuracy of 3D mesh reconstruction of a single image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and image three-dimensional mesh reconstruction, and specifically relates to a three-dimensional mesh reconstruction method and system based on a single image under a deep learning framework. Background Art

[0002] 3D modeling is an important computer-aided design technology, playing a fundamental role in industries such as construction engineering, industrial design, and film and television animation. Nowadays, with the acceleration of industrialization in various fields, the requirements for 3D modeling efficiency are constantly increasing. Compared with traditional manual modeling, 3D reconstruction based on a single image has the characteristics of high reconstruction efficiency and low cost. Therefore, accurate 3D model reconstruction based on a single image has attracted more and more attention from researchers. At the same time, deep learning has been continuously developing in recent years and has achieved great success in the field of computer vision, making the research work combining deep learning with single image 3D reconstruction a hot topic.

[0003] The three-dimensional model structure representation of image reconstruction includes voxels, point clouds, meshes and other types. The early research on image-based three-dimensional reconstruction was mainly based on voxels and point clouds. However, the geometric expression ability of the voxel-based reconstruction model is limited, and there are inherent quantization errors. The point cloud-based reconstruction model lacks a complete surface display. The mesh is an irregular model structure representation, composed of vertices, edges, and faces. It is lightweight and has rich shape details. Reconstruction based on mesh representation can further provide local neighborhood relationships of the model, realize the continuity representation of the object surface, and perform more accurate error calculations based on vertex coordinates. Therefore, model reconstruction research based on mesh representation has gradually become a hot topic.

[0004] However, due to the ordered order of vertices in the grid structure, the number of adjacent vertices for each point in the topological surface is not consistent, and convolutional neural networks cannot directly process such non-Euclidean structured data, so grid-based reconstruction is full of challenges. Summary of the invention

[0005] The purpose of the present invention is to propose a three-dimensional mesh reconstruction method and system based on a single image under a deep learning framework. The innovation of the invention lies in that on the basis of using a graph convolution module to process local mesh features, a cascaded Transfomer encoding module is used to supplement global information, thereby ultimately improving the accuracy of three-dimensional mesh reconstruction from a single image under a deep learning framework.

[0006] A three-dimensional image reconstruction method based on deep learning is as follows:

[0007] Step 1: Obtain a two-dimensional image and corresponding three-dimensional model data;

[0008] Step 2: preprocessing the above two-dimensional image and three-dimensional model data;

[0009] Step 3: Neural network construction and training;

[0010] 3-1. Build a complete reconstruction network

[0011] The complete reconstruction network includes an image feature extraction network and a mesh deformation network; the image feature extraction network is used to input a single two-dimensional image to obtain three scale feature maps; the mesh deformation network is used to input an initial elliptical mesh model and the three scale feature maps output by the image feature extraction network, perform mesh deformation on the initial elliptical mesh model and complete reconstruction;

[0012] 3-2. Model training

[0013] The image data preprocessed in step 2 is input into the complete reconstruction network in step 3-1 for training; during the training process, the loss value of the reconstruction model output by the complete reconstruction network and the target model is calculated by the fusion loss function, and then the loss value is back-propagated, and the network iteration parameters continuously reduce the loss value to complete the training;

[0014] Step 4: According to the network model built and trained in step 3, a single two-dimensional image to be reconstructed is input, and finally a reconstructed three-dimensional model is output.

[0015] Another object of the present invention is to provide a deep learning-based three-dimensional image reconstruction system, the system comprising:

[0016] A data acquisition module, used to acquire a two-dimensional image and a corresponding three-dimensional model;

[0017] A data preprocessing module, used to adjust the format and size of the two-dimensional image and the corresponding three-dimensional model;

[0018] The complete reconstruction network module is used to train the complete reconstruction network using the data processed by the data preprocessing module, and then use the complete reconstruction network to reconstruct the corresponding three-dimensional model of the two-dimensional image to be reconstructed.

[0019] The beneficial effects of the present invention are as follows:

[0020] 1. The present invention is aimed at the task of 3D mesh reconstruction of a single image. It introduces a graph convolution module to effectively process non-Euclidean mesh model data and completes 3D mesh reconstruction based on local mesh information.

[0021] 2. The invention is aimed at the task of 3D mesh reconstruction from a single image. It supplements global information and improves reconstruction accuracy by cascading a graph convolution module and a Transformer encoding module. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a model structure diagram of the complete reconstruction network in the present invention.

[0023] Figure 2 It is a model structure diagram of the image feature extraction network in the present invention.

[0024] Figure 3 It is a model structure diagram of the grid deformation network in the present invention. DETAILED DESCRIPTION

[0025] The present invention is further analyzed below in conjunction with specific embodiments.

[0026] A three-dimensional image reconstruction method based on deep learning, including the following contents:

[0027] Step 1: Get the 2D image and the corresponding 3D model data. The data source is the Shapene public dataset.

[0028] Step 2: Preprocess the two-dimensional image and three-dimensional model data. The preprocessing method includes two-dimensional image scaling, cropping, and normalization, and three-dimensional model Poisson disk algorithm pre-sampling.

[0029] Step 3: Build and train the neural network.

[0030] 3-1. Build a complete reconstruction network

[0031] like Figure 1 The complete reconstruction network includes an image feature extraction network and a mesh deformation network. The image feature extraction network is used to input a single two-dimensional image to obtain three scale feature maps; the mesh deformation network is used to input the initial elliptical mesh model and the three scale feature maps output by the image feature extraction network, and perform mesh deformation on the initial elliptical mesh model to complete the reconstruction, that is, the mesh deformation network performs multiple vertex sampling in the three scale feature maps according to the input initial elliptical mesh model, and then uses multiple graph convolution modules and Transformer encoding modules to deform the initial elliptical mesh according to the sampling results, and uses multiple vertex anti-pooling modules to expand the mesh vertices, and finally outputs the reconstructed three-dimensional model; the present invention proposes that the cascaded graph convolution module and the Transformer encoding module jointly process the mesh model, which can effectively combine local information with global information to complete the reconstruction.

[0032] 3-2. Model training

[0033] The image data preprocessed in step 2 is input into the complete reconstruction network in step 3-1 for training. During the training process, the loss value of the reconstruction model output by the complete reconstruction network and the target model is calculated through the fusion loss function, and then the loss value is back-propagated, and the network iteration parameters continuously reduce the loss value to complete the training.

[0034] Step 4: According to the network model built and trained in step 3, a single two-dimensional image to be reconstructed is input, and finally a reconstructed three-dimensional model is output.

[0035] In step 3-1, if Figure 2 The image feature extraction network structure is composed of 13 series-connected two-dimensional convolution units of the existing VGG16 network. The first two-dimensional convolution unit is used as the input end of the feature extraction network, and receives the preprocessed two-dimensional image data; the output end of the seventh two-dimensional convolution unit outputs a shallow feature map; the output end of the tenth two-dimensional convolution unit outputs a middle feature map; the output end of the thirteenth two-dimensional convolution unit outputs a deep feature map;

[0036] The shallow feature map, the middle feature map and the deep feature map output by the feature extraction network are superimposed to form three scale feature maps as the input of the mesh deformation network;

[0037] like Figure 3 The grid deformation network includes a first deformation unit, a second deformation unit, and a third deformation unit connected in series in sequence;

[0038] The first deformation unit includes a first vertex feature sampling module, a first graph convolution module, a first vertex anti-pooling module, and a first Transfomer encoding module; wherein the first graph convolution module is composed of five first to fifth graph convolution units connected in series; the first input end of the first vertex feature sampling module is used as the first input end of the mesh deformation network, connected to the input initial elliptical mesh model, the second input end is used as the second input end of the mesh deformation network, connected to the three scale feature maps output by the feature extraction network, the first output end is connected to the input end of the first graph convolution module, and the second output end is connected to the input end of the first Transfomer encoding module; the output end data of the first graph convolution module is added to the output end data of the first Transfomer encoding module and then connected to the input end of the first vertex anti-pooling module; the output end of the first vertex anti-pooling module is used as the output end of the first deformation unit, connected to the second input end of the second deformation unit;

[0039] The second deformation unit includes a second vertex feature sampling module, a second graph convolution module, a second vertex anti-pooling module, and a second Transfomer encoding module; wherein the second graph convolution module is composed of five serially connected sixth to tenth graph convolution units; the first input end of the second vertex feature sampling module is used as the third input end of the mesh deformation network, connected to the three scale feature maps output by the feature extraction network, the second input end is connected to the output end of the first deformation unit, the first output end is connected to the input end of the second graph convolution module, and the second output end is connected to the input end of the second Transfomer encoding module; the output end data of the second graph convolution module is added to the output end data of the second Transfomer encoding module and then connected to the input end of the second vertex anti-pooling module; the output end of the second vertex anti-pooling module is used as the output end of the second deformation unit and connected to the second input end of the third deformation unit;

[0040] The third deformation unit includes a third vertex feature sampling module, a third graph convolution module, and a third Transfomer encoding module; wherein the third graph convolution module is composed of five eleventh to fifteenth graph convolution units connected in series; the first input end of the third vertex feature sampling module is used as the fourth input end of the mesh deformation network, connected to the three scale feature maps output by the feature extraction network, the second input end is connected to the output end of the second deformation unit, the first output end is connected to the input end of the third graph convolution module, and the second output end is connected to the input end of the third Transfomer encoding module; the output end data of the third graph convolution module is added to the output end data of the third Transfomer encoding module and then output as the reconstructed three-dimensional model of the mesh deformation network;

[0041] The first Transfomer encoding module includes a first linear layer, a position encoding layer, a multi-head attention mechanism layer, a first normalization layer, a forward propagation layer, a second normalization layer, and a second linear layer connected in series in sequence; the second Transfomer encoding module includes a third linear layer, a position encoding layer, a multi-head attention mechanism layer, a third normalization layer, a forward propagation layer, a fourth normalization layer, and a fourth linear layer connected in series in sequence; the third Transfomer encoding module includes a fifth linear layer, a position encoding layer, a multi-head attention mechanism layer, a fifth normalization layer, a forward propagation layer, a sixth normalization layer, and a sixth linear layer connected in series in sequence;

[0042] In step 3-2, the fusion loss function in model training is shown in formula (6). The fusion loss is the sum of chamfer distance loss, edge length loss, normal vector loss and Laplace loss.

[0043] The chamfer distance loss is shown in formula (1):

[0044]

[0045] Where P and Q are the two point sets to be compared, which in this article represent the vertex sets of the reconstructed 3D model and the real 3D model respectively; p and q represent the unit vertices in the corresponding point sets; |P| represents the number of elements in the point set P; the calculation method is to calculate the squared Euclidean distances from all unit points in the two point sets to the nearest points in the other point set, and then obtain the nearest average distances between the point sets and add them up.

[0046] The edge length loss is shown in formula (2):

[0047]

[0048] Where p is the coordinate of any vertex, and k is the coordinate of any vertex in the adjacent vertex set N(p) of vertex p. This loss limits the occurrence of overly long edges in the mesh model.

[0049] The normal vector loss is shown in formula (3):

[0050]

[0051] Where p represents the vertex of the reconstructed 3D model, q represents the vertex closest to p in the real model, that is, it is regarded as the approximate point of p, and n q represents the normal vector of vertex q. Since p and q are approximate points n q Then it is regarded as the normal vector of the predicted vertex p, k represents the adjacent vertex of p, the vector difference between the two is the required vertex edge vector, and <,> represents the inner product calculation between vectors.

[0052] Laplace loss is shown in formula (4):

[0053]

[0054] Among them l p Represents the Laplace coordinate of the current mesh vertex, l′ p is the Laplace coordinate of the mesh vertex before deformation by each deformation unit. The Laplace coordinate calculation method is shown in formula (5);

[0055]

[0056] Where p is the vertex coordinate, l p is the Laplace coordinate of the vertex, N(p) represents the set of adjacent vertices whose coordinates are vertex p, k represents the coordinates of an adjacent vertex in the set, and the coordinates reflect the relative position of each vertex in the local grid plane.

[0057] The fusion loss includes the above four losses, expressed as follows:

[0058] loss=loss cd +αloss edg +βlossn +γloss lap (6)

[0059] Where α is loss edg The corresponding loss weight, β is loss n The corresponding loss weight, γ is loss lap Corresponding loss weights: In the present invention, α is taken as 0.1, β is taken as 0.3, and γ is taken as 0.0001.

[0060] In order to verify the effectiveness of the present invention, various existing reconstruction methods such as 3D-R2N2, PSGN, Pixel2Mesh, etc. are compared. The experiment selects 13 different categories of 3D models for training and testing based on the Shapenet dataset. The CD (Chamfer Distance) value and F (F-Score) value are used as evaluation indicators. The lower the CD value, the better the reconstruction effect, and the higher the F value, the better the reconstruction effect.

[0061] The experimental results are shown in Table 1 and Table 2. The network model of the present invention achieves excellent reconstruction effect compared with the existing model under two evaluation indicators.

[0062] Table 1 Comparison of CD values ​​of different network reconstruction results

[0063]

[0064] Table 2 Comparison of F values ​​of reconstruction results of different networks

[0065]

[0066]

Claims

1. A method for three-dimensional image reconstruction based on deep learning, characterized in that The method is as follows: Step 1: Obtain a two-dimensional image and corresponding three-dimensional model data; Step 2: preprocessing the above two-dimensional image and three-dimensional model data; Step 3: Neural network construction and training; 3-1. Build a complete reconstruction network The complete reconstruction network includes an image feature extraction network and a mesh deformation network; the image feature extraction network is used to input a single two-dimensional image to obtain three scale feature maps; the mesh deformation network is used to input an initial elliptical mesh model and the three scale feature maps output by the image feature extraction network, perform mesh deformation on the initial elliptical mesh model and complete reconstruction; The image feature extraction network structure includes 13 two-dimensional convolution units of the VGG16 network connected in series; the first two-dimensional convolution unit is used as the input end of the feature extraction network, and receives the preprocessed two-dimensional image data; the output end of the seventh two-dimensional convolution unit outputs a shallow feature map; the output end of the tenth two-dimensional convolution unit outputs a middle feature map; the output end of the thirteenth two-dimensional convolution unit outputs a deep feature map; 3-2. Model training The image data preprocessed in step 2 is input into the complete reconstruction network in step 3-1 for training; during the training process, the loss value of the reconstruction model output by the complete reconstruction network and the target model is calculated by the fusion loss function, and then the loss value is back-propagated, and the network iteration parameters continuously reduce the loss value to complete the training; The fusion loss function is shown in formula (6), and the fusion loss is composed of the sum of chamfer distance loss, edge length loss, normal vector loss and Laplace loss; The chamfer distance loss is shown in formula (1): Where P and Q are the vertex sets of the reconstructed 3D model and the real 3D model; p and q represent the unit vertices in the corresponding point set; |P| represents the number of elements in the point set P; The edge length loss is shown in formula (2): Where k is the coordinate of any vertex in the adjacent vertex set N(p) of vertex p; The normal vector loss is shown in formula (3): where n q represents the normal vector of vertex q, <,> represents the inner product calculation between vectors; Laplace loss is shown in formula (4): Among them l p Represents the Laplace coordinate of the current mesh vertex p, l′ p is the Laplace coordinate of the mesh vertex p before deformation by each deformation unit. The Laplace coordinate calculation method is shown in formula (5); The fusion loss is as follows: loss=loss cd +αloss edg +βloss n +γloss lap (6) Where α is loss edg The corresponding loss weight, β is loss n The corresponding loss weight, γ is loss lap The corresponding loss weight; Step 4: According to the network model built and trained in step 3, a single two-dimensional image to be reconstructed is input, and finally a reconstructed three-dimensional model is output.

2. The method for three-dimensional image reconstruction based on deep learning as claimed in claim 1, characterized in that The preprocessing method in step 2 includes scaling, cropping, and normalizing the two-dimensional image, and the three-dimensional model is pre-sampled using the Poisson disk algorithm.

3. The method for three-dimensional image reconstruction based on deep learning as claimed in claim 1, characterized in that The shallow feature map, the middle feature map and the deep feature map output by the feature extraction network are superimposed to form three scale feature maps as the input of the mesh deformation network.

4. The method for three-dimensional image reconstruction based on deep learning as claimed in claim 1, characterized in that The grid deformation network includes a first deformation unit, a second deformation unit, and a third deformation unit connected in series in sequence; The first deformation unit includes a first vertex feature sampling module, a first graph convolution module, a first vertex anti-pooling module, and a first Transformer encoding module; the first input end of the first vertex feature sampling module is used as the first input end of the mesh deformation network, connected to the input initial elliptical mesh model, the second input end is used as the second input end of the mesh deformation network, connected to the three scale feature maps output by the feature extraction network, the first output end is connected to the input end of the first graph convolution module, and the second output end is connected to the input end of the first Transformer encoding module; the output end data of the first graph convolution module is added to the output end data of the first Transformer encoding module and then connected to the input end of the first vertex anti-pooling module; the output end of the first vertex anti-pooling module is used as the output end of the first deformation unit, and connected to the second input end of the second deformation unit; The second deformation unit includes a second vertex feature sampling module, a second graph convolution module, a second vertex anti-pooling module, and a second Transfomer encoding module; the first input end of the second vertex feature sampling module is used as the third input end of the mesh deformation network, connected to the three scale feature maps output by the feature extraction network, the second input end is connected to the output end of the first deformation unit, the first output end is connected to the input end of the second graph convolution module, and the second output end is connected to the input end of the second Transfomer encoding module; the output end data of the second graph convolution module is added to the output end data of the second Transfomer encoding module and then connected to the input end of the second vertex anti-pooling module; the output end of the second vertex anti-pooling module is used as the output end of the second deformation unit and connected to the second input end of the third deformation unit; The third deformation unit includes a third vertex feature sampling module, a third graph convolution module, and a third Transfomer encoding module; the first input end of the third vertex feature sampling module serves as the fourth input end of the mesh deformation network, connected to the three scale feature maps output by the feature extraction network, the second input end is connected to the output end of the second deformation unit, the first output end is connected to the input end of the third graph convolution module, and the second output end is connected to the input end of the third Transfomer encoding module; the output end data of the third graph convolution module is added to the output end data of the third Transfomer encoding module and output as the reconstructed three-dimensional model of the mesh deformation network.

5. The method for three-dimensional image reconstruction based on deep learning as claimed in claim 4, characterized in that The first graph convolution module includes five first to fifth graph convolution units connected in series; the second graph convolution module includes five sixth to tenth graph convolution units connected in series; and the third graph convolution module includes five eleventh to fifteenth graph convolution units connected in series.

6. The method for three-dimensional image reconstruction based on deep learning as claimed in claim 4, characterized in that The first Transfomer encoding module includes a first linear layer, a position encoding layer, a multi-head attention mechanism layer, a first normalization layer, a forward propagation layer, a second normalization layer, and a second linear layer connected in series in sequence; the second Transfomer encoding module includes a third linear layer, a position encoding layer, a multi-head attention mechanism layer, a third normalization layer, a forward propagation layer, a fourth normalization layer, and a fourth linear layer connected in series in sequence; the third Transfomer encoding module includes a fifth linear layer, a position encoding layer, a multi-head attention mechanism layer, a fifth normalization layer, a forward propagation layer, a sixth normalization layer, and a sixth linear layer connected in series in sequence.

7. A deep learning-based three-dimensional image reconstruction system for implementing the method according to any one of claims 1 to 6, characterized in that include: A data acquisition module, used to acquire a two-dimensional image and a corresponding three-dimensional model; A data preprocessing module, used to adjust the format and size of the two-dimensional image and the corresponding three-dimensional model; The complete reconstruction network module is used to train the complete reconstruction network using the data processed by the data preprocessing module, and then use the complete reconstruction network to reconstruct the corresponding three-dimensional model of the two-dimensional image to be reconstructed.