A 3D Mesh Repair Method Based on Deep Learning

Abstract

The present invention discloses a three-dimensional mesh repair method based on deep learning, comprising the following steps: preprocessing the three-dimensional mesh data to be repaired; manually selecting a hole area by the user or automatically finding the hole area by a program, and extracting the patch features of all patches within the area; predicting the repair result according to the patch features; correcting the prediction result, updating the input three-dimensional mesh data, and obtaining the repair result. The method provided by the present invention can effectively restore the original details of the hole area, so as to repair and obtain more complete three-dimensional mesh data.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a three-dimensional mesh repair method based on deep learning. Background Art

[0002] Nowadays, due to the popularity of low-cost 3D scanning equipment, the application of 3D scanning of real-world objects has spread rapidly. However, the mesh data captured by these 3D scanners usually includes holes or missing areas caused by self-occlusion, dark colors, and surface high light reflectivity. Therefore, how to quickly, accurately, and automatically repair 3D mesh model defects has become one of the important research directions of 3D graphics processing.

[0003] However, there are also some problems and challenges in the research of 3D mesh patching based on deep learning. First of all, the current 3D mesh patching technology based on deep learning sometimes has some low accuracy or unstable patching results in practical applications. Therefore, how to improve the accuracy and stability of the algorithm will be one of the research focuses in the future. In addition, the training data set of the 3D mesh patching technology based on deep learning needs to contain information of various models. The quality and diversity of the data set directly affect the performance and generalization ability of the patching algorithm. Moreover, for some application scenarios with high real-time requirements, it is sometimes necessary to generate models quickly. Therefore, how to build a large-scale, high-quality, and diverse data set, and how to improve the real-time performance of the algorithm, is also a problem that needs to be solved urgently.

[0004] Patent document CN109615702A discloses a mesh hole filling method, an image processing device and a device with a storage function. The mesh hole filling method includes: obtaining a three-dimensional mesh model; detecting whether there are holes in the three-dimensional mesh model; if there are holes, filling the holes; smoothing the newly added vertices in the three-dimensional mesh model after the hole filling operation; wherein the smoothing is based on smooth movement of Laplace and Gaussian curvature.

[0005] Patent document CN113593011A discloses a hole repair method, an electronic device and a storage medium, the hole repair method comprising: obtaining the center coordinates of a marking point in a three-dimensional mesh model and the radius of the marking point; determining a geometric body according to the center coordinates of the marking point and the radius of the marking point, wherein the center coordinates of the geometric body are the same as the center coordinates of the marking point; obtaining the coordinates of each vertex of a facet within the range of the geometric body; determining a set of boundary vertices of a hole to be repaired according to the coordinates of each vertex of the facet; in the three-dimensional mesh model, determining a boundary of the hole to be repaired according to the set of boundary vertices, and performing hole repair on the boundary of the hole to be repaired.

[0006] Patents all use traditional methods for hole repair, taking the local geometric properties of the three-dimensional grid as prior knowledge, and then using interpolation methods to fill the holes. These methods can repair simple small holes well, but when faced with larger and more complex holes, these methods will have obvious problems of detail loss. Summary of the Invention

[0007] The object of the present invention is to provide a three-dimensional grid repair method based on deep learning, which can effectively restore the original details of the hole area, thereby repairing to obtain more complete three-dimensional grid data, and the complete three-dimensional grid data can reconstruct a more refined 3D geometric model.

[0008] To achieve the object of the present invention, a three-dimensional grid repair method based on deep learning is provided, including the following steps:

[0009] Step 1: Preprocess the three-dimensional grid data to be repaired to construct a first data set that conforms to the two-dimensional manifold network topology structure. The first data set includes three-dimensional grid data, as well as the central coordinates and normalization parameters of the vertices in the grid. The preprocessing includes defect deletion, network repair, and coordinate normalization;

[0010] Step 2: Detect the hole area of the grid in the first data set, and perform simple filling on the obtained hole area to obtain a corresponding second data set;

[0011] Perform a grid smoothing operation on the second data set to obtain a corresponding third data set;

[0012] Step 3: Perform data matching based on the second data set and the third data set to generate detail repair information for the non-hole area, and combine the second data set and the detail repair information to combine the feature information of each patch and all patches within a specified range in the three-dimensional grid of the combination result to construct a corresponding patch set;

[0013] Step 4: Input the patch set into a pre-constructed deep learning network to eliminate noise and generate a corresponding hidden vector, calculate the normal vectors of all patches around the vertices in the hole area of the three-dimensional grid based on the hidden vector, and output them as the prediction result;

[0014] Step 5: Update and repair the second data set based on the normal vectors obtained in Step 4 and the central coordinates and normalization parameters of all vertices in the first data set to obtain the repaired three-dimensional grid data.

[0015] The present invention restores the defective parts of a 3D mesh model automatically by training a deep neural network, realizing the repair and improvement of the 3D mesh model. This neural network can perform the repair work very quickly and can better preserve the original features of the model, such as geometric shapes, surface textures, and other geometric properties.

[0016] Specifically, in step 1, the specific process of mesh repair is as follows:

[0017] Step 1-1: Check the adjacent faces of each edge in the 3D mesh data:

[0018] When an edge is connected to more than 2 faces at the same time, it is considered a non-manifold edge;

[0019] When a non-manifold edge is detected, this edge will be directly removed, and the surrounding faces will be removed or merged;

[0020] Step 1-2: Check the connecting edges of each vertex in the 3D mesh data:

[0021] When a vertex is connected to more than 2 edges at the same time, it is considered a non-popular vertex;

[0022] When a non-manifold vertex is detected, the vertex and its adjacent edges will be removed or the edges will be merged according to the situation.

[0023] Specifically, in step 1, the specific process of coordinate normalization is as follows:

[0024] Step 1-3: Calculate the coordinates of the center point of the mesh: Traverse all the vertices in the mesh to find the average value to obtain the coordinate value of the center point of the mesh, and for each vertex in the 3D mesh data, subtract its coordinate value from the coordinate value of the center point to obtain the corrected coordinate corresponding to each vertex;

[0025] Step 1-4: For each vertex in the 3D mesh data, divide its corrected coordinate value by the scale of the corresponding coordinate axis to obtain the normalized coordinate value corresponding to each vertex.

[0026] Specifically, the expression for calculating the coordinates of the center point of the mesh is as follows:

[0027]

[0028] In the formula, (x, y, z) represents the coordinates of the vertex, and n represents the number of vertices of the 3D mesh.

[0029] Specifically, the expression for normalization is as follows:

[0030]

[0031]

[0032]

[0033] Wherein, x max , x min , y max , y min , z max , z min respectively represent the maximum and minimum values of the vertex coordinates on the x-axis, y-axis, and z-axis; x scale , y scale , z scale represent the proportionality parameters of the corresponding coordinate axes.

[0034] Specifically, the specific process of simple filling of the hole area in step 2 is as follows:

[0035] Step 2-1: Search for all hole areas in the three-dimensional grid data to obtain the boundary of the area to be repaired;

[0036] Step 2-2: Move from the boundary of the area to be repaired towards the inside of the hole area to the vertex, interpolate and calculate the position of the new vertex according to the positions of the surrounding vertices, and connect the newly generated vertex and the known vertices to create a new triangular patch to repair the hole area.

[0037] Specifically, the specific process of grid smoothing operation in step 2 is as follows:

[0038] Step 2-3: Calculate the Laplacian coordinates of each vertex in the hole area, where the Laplacian coordinates represent the offset of a vertex's position relative to its surrounding vertices;

[0039] Step 2-4: Update the position of the corresponding vertex according to the calculated Laplacian coordinates, and the update process is as follows:

[0040] v′ i = v i + αδ i

[0041] where α is a smoothing coefficient used to control the strength of smoothing, v i and v i ′ are the original vertex coordinates and the modified vertex coordinates respectively, and δ i is the Laplacian coordinate of the vertex.

[0042] Specifically, the construction process of the patch set in step 3 is as follows:

[0043] Step 2-5: Calculate the corresponding patch radius for each patch, select all vertices whose distance from the center point of the given patch is less than the patch radius, and calculate all the adjacent faces of the selected vertices as the patch range corresponding to the patch;

[0044] Step 2-6: For the patch corresponding to the patch surface, extract the feature information of all patch surfaces within the corresponding patch range, and establish the connection relationship between the patch surfaces to generate the graph-like features of the patch.

[0045] Specifically, the calculation process of the Laplacian coordinates is as follows:

[0046]

[0047] In the formula, δ i is the Laplacian coordinate of the vertex, v i =(x i , y i , z i ) is the coordinate of the i-th vertex, v j =(x j , y j , z j ) is the coordinate of the j-th adjacent vertex of the i-th vertex, E is the number of adjacent vertices of the i-th vertex, and w ij is the weight parameter.

[0048] Specifically, the calculation process of the patch radius is as follows:

[0049]

[0050] where A is the total area of all patch surfaces within the second-ring neighborhood of the patch surface, and s is the proportionality parameter.

[0051] Specifically, in step 4, a graph convolutional neural network is used to process the patch set to obtain the normal vectors of all patch surfaces around the vertices in the hole area of the three-dimensional grid.

[0052] Specifically, the specific process of the update and repair in step 5 is as follows:

[0053] Step 5-1: Use a bilateral filter to process the normal vectors to filter out some incorrect results in the normal vectors;

[0054] Step 5-2: Based on the filtered normal vectors, use a vertex update method to repair the vertices in the second data set to obtain the repaired second data set;

[0055] Step 5-3: Use the center point coordinates and normalization parameters of all vertices in the first data set to update the repaired second data set to the three-dimensional grid data to be repaired to complete the repair work of the three-dimensional grid data.

[0056] Compared with the prior art, the beneficial effects of the present invention:

[0057] The network is trained without prior knowledge, which supports users to select the hole area by themselves and can quickly complete the repair work. At the same time, the original features of the 3D mesh, such as geometric shape, surface texture and other geometric properties, can be better retained during the repair. Brief Description of the Drawings

[0058] Figure 1 It is a flowchart of a 3D mesh repair method based on deep learning provided in this embodiment. Detailed Embodiment

[0059] Next, the technical solutions of the present invention will be clearly and completely described in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the protection scope of the present invention. In addition, the technical features involved in different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0060] As Figure 1 shown, the 3D mesh repair method based on deep learning includes the following steps:

[0061] Perform some preprocessing operations on the input 3D mesh data so that the subsequent repair algorithm can process it better to obtain the first dataset. This part of the work includes defect deletion, network repair and coordinate normalization.

[0062] In this embodiment, the specific process is as follows:

[0063] In the defect deletion work, we need to remove invalid patches, isolated vertices, etc. These invalid data are usually introduced due to file format conversion or other operations.

[0064] In the mesh repair, we need to ensure that the topological structure of the 3D mesh meets certain conditions. We need to remove non-manifold edges to ensure that the mesh is a two-dimensional manifold mesh. Specifically, check the adjacent faces of each edge. If an edge is connected to more than 2 faces at the same time, then it is considered a non-manifold edge. When a non-manifold edge is detected, this edge will be directly removed, and the surrounding faces will be removed or merged. At the same time, check the connecting edges of each vertex: If a vertex is connected to more than 2 edges at the same time, then it is considered a non-popular vertex; when a non-manifold vertex is detected, the vertex and its adjacent edges will be removed according to the situation, or the edges will be merged.

[0065] In coordinate normalization, we scale the coordinate range of the three-dimensional grid to a suitable interval and translate the center point coordinates of the three-dimensional grid to the origin of coordinates to facilitate subsequent calculations and processing. The specific normalization process is as follows:

[0066] Calculate the center point coordinates of the grid: Traverse all the vertices in the grid to find the average value to obtain the center point coordinate value of the grid. For each vertex in the three-dimensional grid data, subtract its coordinate value from the center point coordinate value to obtain the corrected coordinate corresponding to each vertex, so that the center point of the three-dimensional grid can be translated to the origin of coordinates;

[0067] For each vertex in the three-dimensional grid data, divide its corrected coordinate value by the scale of the corresponding coordinate axis to obtain the normalized coordinate value corresponding to each vertex, so that the coordinate values of the vertices in the three-dimensional grid can be normalized to the range of [-1, 1].

[0068] Convert the first data set into a format that is easier for the deep learning network to process. This step mainly includes simple hole filling, mesh smoothing, and patch generation.

[0069] In this embodiment, the specific process is as follows:

[0070] The simple hole filling work includes detecting the hole area of the grid and performing a simple filling on the hole area to obtain the second data set. The specific operation process is as follows:

[0071] Automatically check all the hole areas in the first data set, or directly manually select the hole areas by the user to obtain the boundary of the area to be repaired;

[0072] Move the vertices along the boundary of the hole area towards the inside of the hole, and interpolate and calculate the positions of the new vertices according to the positions of the surrounding vertices. Connect the newly generated vertices and the known vertices to create new triangular patches to repair the holes. Repeat this operation until the holes are completely filled.

[0073] The mesh smoothing work includes performing a mesh smoothing operation on the filled second data set to smooth the noise or discontinuity of the three-dimensional grid to obtain the corresponding third data set. The specific operation process is as follows:

[0074] Calculate the Laplacian coordinates of each vertex in the hole area. The Laplacian coordinates represent the offset of a vertex's position relative to its surrounding vertices;

[0075]

[0076] In the formula, δ i is the Laplacian coordinate of the vertex, v i =(x i , y i , zi ) is the coordinate of the i-th vertex, v j =(x j , y j , z j ), and (x ij , y

[0077] According to the calculated Laplacian coordinates, update the position of each vertex according to the following formula:

[0078] v' i =v i +αδ i

[0079] where α is a smoothing coefficient used to control the strength of smoothing. The degree of smoothing can be controlled by adjusting the size of the smoothing coefficient.

[0080] The patch generation work includes extracting the feature information of each facet in the 3D mesh and all facets within its specified range, and organizing it into a patch. The specific operation process is as follows:

[0081] For each given facet, calculate the patch radius size using the following formula

[0082]

[0083] where A is the total area of all facets within the 2-ring neighbors of the facet, and s is a proportionality parameter.

[0084] Extract all vertices whose distance from the center point of the given facet is less than r, and calculate all the adjacent faces of these vertices, representing the patch range of the given facet;

[0085] For each patch, extract the feature information of all facets within the patch range, establish the connection relationship between the facets, and generate the graph-like features of the patch.

[0086] Based on the position of the hole region in the second dataset and combined with the third dataset, extract the feature information of each facet and all facets within the specified range in the 3D mesh to construct the corresponding patch set.

[0087] In the second dataset here, the holes have been initially filled, but the filling result is too smooth, losing the detailed information in the 3D mesh. Performing a mesh smoothing operation on the second dataset has smoothed out the detailed information in other areas of the second dataset, resulting in the third dataset. After that, the deep learning network receives both the second dataset and the third dataset simultaneously and attempts to "restore" the third dataset to the second dataset. In this process, the deep learning network compares the areas outside the holes in the two datasets (for the same area, the second dataset has the original details while the details in the third dataset are smoothed out), obtains the repair information, and applies this repair information to the initially filled result of the "missing detail" holes, playing the role of "restoring details".

[0088] Input the patch set into a pre - constructed GCN (Graph Convolutional Neural Network) to output the normal vectors of all the faces around the vertices in the hole area of the 3D mesh. The GCN (Graph Convolutional Neural Network) proposed in this implementation is a convolutional neural network based on graph relationships, including 9 layers of graph convolutional neural network, where the first 7 layers are convolutional layers and the last 2 layers are fully connected layers, and a data - driven method is adopted to train the deep learning network using existing datasets. Compared with the traditional CNN (Convolutional Neural Network), GCN needs to additionally add the connection relationships between the input data (construct the input data as graph data) and has a more powerful feature extraction ability.

[0089] In this process, when the input data passes through multiple convolutional layers, the noise and irrelevant features during input are removed, and it enters a space with an extremely high dimension (referred to as the latent space), becoming a feature vector that only the neural network can understand. Subsequently, the neural network will output these results in the remaining layers.

[0090] Based on the obtained normal vectors, the central coordinates, and the normalization parameters of all vertices in the first dataset, update and repair the second dataset to obtain the repaired 3D mesh data. The specific operation process is as follows:

[0091] Use a bilateral filter to process the prediction result to eliminate some incorrect results in the prediction result.

[0092] Adopt the Vertex Update method to update the vertices of the input model to obtain the repaired second dataset.

[0093] Finally, use the central coordinates and the normalization parameters of all vertices in the first dataset to restore the repaired second dataset to the 3D mesh data to be repaired to obtain the repaired 3D mesh data.

[0094] It should be noted that, in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0095] The above are only specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather to the broadest scope consistent with the principles and novel features claimed herein.

Claims

1. A 3D mesh repair method based on deep learning, characterized in that It includes the following steps: Step 1: Preprocess the 3D mesh data to be repaired to construct a first dataset that conforms to the two-dimensional manifold network topology. The first dataset includes 3D mesh data, as well as the central coordinates and normalization parameters of the vertices in the mesh. The preprocessing includes defect deletion, network repair, and coordinate normalization. The specific process of coordinate normalization is as follows: Step 1-3: Calculate the mesh center point coordinates: Traverse all the vertices in the mesh to find the average value to obtain the mesh center point coordinate value. For each vertex in the 3D mesh data, subtract its coordinate value from the coordinate value of the center point to obtain the corrected coordinate corresponding to each vertex; Step 1-4: For each vertex in the 3D mesh data, divide its corrected coordinate value by the scale of the corresponding coordinate axis to obtain the normalized coordinate value corresponding to each vertex; Step 2: Detect the hole regions in the mesh of the first dataset and perform simple filling on the obtained hole regions to obtain the corresponding second dataset; Perform a mesh smoothing operation on the second dataset to obtain the corresponding third dataset. The specific process of the mesh smoothing operation is as follows: Step 2-3: Calculate the Laplacian coordinates of each vertex in the hole region. The Laplacian coordinates represent the offset of a vertex's position relative to its surrounding vertices; Step 2-4: Update the positions of the corresponding vertices according to the calculated Laplacian coordinates. The update process is as follows: ; where is the smoothing coefficient, which is used to control the strength of smoothing, and are the original vertex coordinates and the modified vertex coordinates respectively, is the Laplacian coordinate of the vertex; Step 3: Perform data matching based on the second dataset and the third dataset to generate detailed patching information for the non-hole area, and combine the second dataset and the detailed patching information to combine the feature information of each patch and all patches within a specified range in the three-dimensional mesh of the combined result to construct a corresponding patch set; Step 4: Input the patch set into a pre-constructed deep learning network to eliminate noise and generate a corresponding latent vector, calculate the normal vectors of all patches around the vertices in the hole area of the three-dimensional mesh based on the latent vector and output them as the prediction result; Step 5: Update and patch the second dataset based on the normal vectors obtained in Step 4, the central coordinates of all vertices in the first dataset, and the normalization parameters to obtain the patched three-dimensional mesh data.

2. The 3D mesh repair method based on deep learning according to claim 1, wherein In Step 1, the specific process of the mesh repair is as follows: Step 1-1: Check the adjacent faces of each edge in the 3D mesh data: When an edge is connected to more than 2 faces at the same time, it is considered a non-manifold edge; When a non-manifold edge is detected, this edge will be directly removed, and the surrounding faces will be removed or merged; Step 1-2: Check the connecting edges of each vertex in the 3D mesh data: When a vertex is connected to more than 2 edges at the same time, it is considered a non-manifold vertex; When a non-manifold vertex is detected, the vertex and its adjacent edges will be removed according to the situation, or edge merging will be performed.

3. The three-dimensional mesh repair method based on deep learning according to claim 1, wherein The specific process of simple filling of the hole regions in Step 2 is as follows: Step 2-1: Search for all the hole regions in the 3D mesh data to obtain the boundary of the region to be repaired; Step 2-2: Move from the boundary of the region to be repaired towards the inside of the hole region to a vertex. Interpolate and calculate the position of the new vertex according to the positions of the surrounding vertices, and connect the newly generated vertex and the known vertices to create new triangular patches to repair the hole region.

4. The three-dimensional mesh repair method based on deep learning according to claim 1, characterized in that, In the process of constructing the patch set in Step 3: Step 2-5: Calculate the corresponding patch radius for each patch. Select all the vertices whose distances from the center point of the given patch are less than the patch radius, and calculate all the adjacent faces of the selected vertices as the patch range corresponding to the patch; Step 2-6: For the patch corresponding to the patch, extract the feature information of all the patches within the corresponding patch range and establish the connection relationship between the patches to generate the graph-like features of the patch.

5. The 3D mesh repair method based on deep learning according to claim 4, wherein, The calculation process of the patch radius is as follows: ; where A is the total area of all patches within the two-ring neighborhood of the patch, and s is the proportionality parameter.

6. The 3D mesh repair method based on deep learning according to claim 1, wherein In Step 4, use a graph convolutional neural network to process the patch set to obtain the normal vectors of all the patches around the vertices in the hole region of the 3D mesh.

7. The three-dimensional mesh repair method based on deep learning according to claim 1, characterized in that In Step 5, the specific process of the update and repair is as follows: Step 5-1: Process the normal vectors using a bilateral filter to filter out some incorrect results in the normal vectors; Step 5-2: Based on the filtered normal vectors, use a vertex update method to repair the vertices in the second dataset to obtain a repaired second dataset; Step 5-3: Utilize the center point coordinates and normalization parameters of all vertices in the first dataset to update the repaired second dataset to the 3D mesh data to be repaired, so as to complete the repair work of the 3D mesh data.

Images