Method and apparatus for reconstructing a three-dimensional object

Abstract

The application provides a three-dimensional object reconstruction method and device. The method comprises the following steps: obtaining a plurality of preliminary three-dimensional reconstruction bodies through a three-dimensional reconstruction network according to point cloud data of a target to be reconstructed; obtaining a real three-dimensional reconstruction body by taking an intersection among the preliminary three-dimensional reconstruction bodies; and optimizing the quality of the real three-dimensional reconstruction body through a two-dimensional reconstruction network to obtain a final three-dimensional reconstruction body. The three-dimensional object reconstruction method and device provided in the application can realize lightweight and efficient three-dimensional object reconstruction by combining a two-dimensional reconstruction network and a three-dimensional reconstruction network, and can obtain a three-dimensional reconstruction body with clear structure, strong target boundary feeling and high texture quality without relying on high-level hardware devices or complex reconstruction technology.

Description

Technical Field

[0001] This application relates to the field of object reconstruction technology, specifically to a three-dimensional object reconstruction method and apparatus. Background Technology

[0002] 3D reconstruction refers to establishing mathematical models of 3D objects suitable for computer representation and processing. It is the foundation for processing, manipulating, and analyzing the properties of objects in a computer environment and a key technology for creating virtual reality that represents the objective world. Currently, image-based 3D reconstruction technologies include those that use high-level hardware to measure distances to objects and calculate distance differences to obtain 3D models, and those that rely on numerous object images and complex reconstruction steps to infer camera parameters and object depth to reconstruct 3D models. However, due to the blurred boundaries and relatively complex structures of buildings, existing technologies often result in problems such as blurred boundaries, poor texture quality at structural joints, and low clarity after reconstructing buildings from 3D maps. Reconstruction of occluded buildings suffers from large-area distortion. Therefore, the reconstruction effect of existing 3D reconstruction technologies needs improvement. Summary of the Invention

[0003] This application provides a method for reconstructing three-dimensional objects to solve the technical problem of poor reconstruction effect in existing three-dimensional reconstruction technologies.

[0004] In a first aspect, embodiments of this application provide a method for reconstructing a three-dimensional object, including:

[0005] Based on the point cloud data of the target to be reconstructed, several preliminary three-dimensional reconstructed bodies are obtained through a three-dimensional reconstruction network;

[0006] By taking the intersection of several preliminary 3D reconstructed bodies, the true 3D reconstructed body is obtained;

[0007] The quality of the real 3D reconstructed object is optimized by a 2D reconstruction network to obtain the final 3D reconstructed object.

[0008] In one embodiment, obtaining several preliminary 3D reconstructed bodies from the point cloud data of the target to be reconstructed through a 3D reconstruction network includes:

[0009] Based on the point cloud data of the target to be reconstructed, a preliminary mesh is obtained by combining a 3D reconstruction network with noise processing.

[0010] Based on the preliminary mesh, a preliminary three-dimensional reconstruction is obtained.

[0011] In one embodiment, obtaining a preliminary mesh based on the point cloud data of the target to be reconstructed, using a 3D reconstruction network combined with noise processing, includes:

[0012] Based on the point cloud data of the target to be reconstructed, three-dimensional voxels are extracted through a three-dimensional reconstruction network;

[0013] Based on the occupancy probability and occupancy binarization threshold of 3D voxels, and combined with noise processing, a preliminary mesh is obtained.

[0014] In one embodiment, obtaining the preliminary three-dimensional reconstructed body based on the preliminary mesh specifically involves:

[0015] Based on the initial mesh, mesh subdivision processing is performed to obtain a preliminary three-dimensional reconstructed body.

[0016] In one embodiment, obtaining a true 3D reconstructed object by taking the intersection of several preliminary 3D reconstructed objects includes:

[0017] Measure the distance between every two of the initial 3D reconstructed volumes;

[0018] Based on the distance between every two of the initial 3D reconstructed volumes, a distance metric loss function is obtained for the target to be reconstructed between different views;

[0019] Based on the distance metric loss function between different views of the target to be reconstructed, a true 3D reconstructed object is obtained.

[0020] In one embodiment, optimizing the quality of the real 3D reconstructed object through a 2D reconstruction network to obtain the final 3D reconstructed object includes:

[0021] Based on the point cloud data of the target to be reconstructed, a sparse UV map is obtained;

[0022] Based on the two-dimensional reconstruction network, a dense UV map is obtained from the sparse UV map, which is used to optimize the mesh of the real three-dimensional reconstructed body to obtain the final three-dimensional reconstructed body.

[0023] In one embodiment, the dense UV map includes a spatial location dense UV map and / or a texture dense UV map.

[0024] Secondly, embodiments of this application provide a three-dimensional object reconstruction apparatus, comprising:

[0025] The module for obtaining preliminary 3D reconstructed bodies is used to: obtain several preliminary 3D reconstructed bodies from the point cloud data of the target to be reconstructed through a 3D reconstruction network;

[0026] The module for obtaining a true 3D reconstructed body is used to: take the intersection of several preliminary 3D reconstructed bodies to obtain a true 3D reconstructed body;

[0027] The final 3D reconstruction module is used to: optimize the quality of the real 3D reconstruction through a 2D reconstruction network to obtain the final 3D reconstruction.

[0028] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the three-dimensional object reconstruction method described in the first aspect.

[0029] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the three-dimensional object reconstruction method described in the first aspect.

[0030] The 3D object reconstruction method and apparatus provided in this application embodiment obtains several preliminary 3D reconstructed bodies without texture processing based on the point cloud data of the target to be reconstructed using a 3D reconstruction network. Then, the intersection of these preliminary 3D reconstructed bodies is obtained to get a structurally clear, realistic 3D reconstructed body. Finally, a 2D reconstruction network is used to optimize the quality of the realistic 3D reconstructed body, improving the target boundary perception and texture quality, resulting in a high-quality final 3D reconstructed body. The 3D object reconstruction method and apparatus provided in this application embodiment combine a 2D reconstruction network and a 3D reconstruction network to achieve lightweight and efficient 3D object reconstruction, obtaining 3D reconstructed bodies with clear structure, strong target boundary perception, and high texture quality, without relying on high-level hardware equipment or complex reconstruction techniques. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a schematic flowchart of the three-dimensional object reconstruction method provided in the embodiments of this application;

[0033] Figure 2 This illustrates the structure of the two-dimensional reconstruction network used in the three-dimensional object reconstruction method provided in this application embodiment;

[0034] Figure 3 A comparison diagram showing the effectiveness of the three-dimensional object reconstruction method provided in this application embodiment and the prior art is shown;

[0035] Figure 4 This is a schematic diagram of the structure of the three-dimensional object reconstruction device provided in the embodiments of this application;

[0036] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] Figure 1 This is a flowchart illustrating a three-dimensional object reconstruction method provided in an embodiment of this application.

[0039] Reference Figure 1 This application provides a method for reconstructing a three-dimensional object, which may include:

[0040] S110. Based on the point cloud data of the target to be reconstructed, several preliminary three-dimensional reconstructed bodies are obtained through a three-dimensional reconstruction network.

[0041] S120. Take the intersection of several preliminary three-dimensional reconstructions to obtain the true three-dimensional reconstruction.

[0042] S130. Optimize the quality of the real three-dimensional reconstructed object through a two-dimensional reconstruction network to obtain the final three-dimensional reconstructed object.

[0043] It should be noted that the execution subject of the three-dimensional object reconstruction method provided by the present invention can be any terminal device, such as a three-dimensional object reconstruction system, etc.

[0044] It should be noted that the two-dimensional reconstruction network and the three-dimensional reconstruction network used in this invention can also be called two-dimensional reconstruction network model and three-dimensional reconstruction network model, respectively, and both can be pre-trained.

[0045] In step S110, the terminal device will obtain several preliminary three-dimensional reconstructions through a three-dimensional reconstruction network based on the point cloud data of the target to be reconstructed.

[0046] It should be noted that point cloud data refers to a collection of vectors in a three-dimensional coordinate system. Scanned data is recorded in the form of points, each containing three-dimensional coordinates, and some may contain color information (RGB) or reflectance intensity information.

[0047] It should be noted that the terminal device can pre-convert the image data of buildings in the 3D map into point cloud data and store it in the database. After the target to be reconstructed is determined, the point cloud data of the image corresponding to the target to be reconstructed can be extracted from the database.

[0048] It should be noted that the 3D reconstruction network used in this invention is a 3D reconstruction network based on an improved MeshCNN, and the 3D reconstruction network can be pre-built by the terminal device.

[0049] Specifically, the terminal device can pre-generate a sample dataset. By acquiring the point cloud data of the target to be reconstructed, it can obtain the spatial information of the target, namely the X, Y, and Z dimension coordinates, which serve as the input to the 3D reconstruction network. For the sample space D = {pic} i}, i = 1, 2, ..., n, for a single sample image pic i The 3D reconstruction network focuses on the values ​​of the three dimensions X, Y, and Z in space, denoted by the array (X, Y, Z). i This represents the spatial information of the i-th sample point.

[0050] Simultaneously, color information from samples can be collected and used as input to the two-dimensional reconstruction network, denoted by the array (R, G, B). i For the sample image pic i The color information of the point is used to generate a sample number, which is denoted as id.

[0051] Furthermore, after generating the sample dataset, the terminal device can initialize a 3D reconstruction network to construct a curved mesh of the building based on the input data. This curved mesh contains fine details of the building's geometry and texture. Specifically, based on the given initial point cloud data, the terminal device generates a convex hull as an initial mesh using the 3D reconstruction network, and then deforms this mesh into the target shape. A graph is constructed on the edges of the initial convex hull, and a model network can be run on this graph to generate vertex displacements, thereby updating the 3D object shape. The features on each graph node are assigned as random vectors sampled from a Gaussian distribution, and a mesh CNN is trained to deform the mesh by minimizing the chamfer distance to the input point cloud. The network topology is updated, and the model network is used to refine the mesh again. The terminal device can then reconstruct the 3D object using this initialized 3D reconstruction network.

[0052] The terminal device first obtains several candidate preliminary 3D reconstructions based on the point cloud data of the target to be reconstructed through a 3D reconstruction network, which is helpful for determining the structure of the real 3D reconstruction in the future.

[0053] Furthermore, step S110 may include:

[0054] Based on the point cloud data of the target to be reconstructed, a preliminary mesh is obtained by combining a 3D reconstruction network with noise processing.

[0055] Based on the preliminary mesh, a preliminary three-dimensional reconstruction is obtained.

[0056] It should be noted that for the point cloud data corresponding to an image of the target to be reconstructed, the terminal device can add multiple different noises to it through a 3D reconstruction network to obtain several initial networks. Each initial network corresponds to a preliminary 3D reconstructed object, thus obtaining multiple reconstruction results. j Let j = 1, 2, ..., s, thus obtaining s preliminary 3D reconstructed objects. Furthermore, to obtain clearer 3D reconstructed objects, similar processing can be performed on the point cloud data corresponding to different views of the target to be reconstructed, thereby obtaining more preliminary 3D reconstructed objects, which is beneficial for obtaining higher-precision intersections later.

[0057] Furthermore, the preliminary mesh is obtained based on the point cloud data of the target to be reconstructed, using a 3D reconstruction network combined with noise processing, including:

[0058] Based on the point cloud data of the target to be reconstructed, three-dimensional voxels are extracted through a three-dimensional reconstruction network;

[0059] Based on the occupancy probability and occupancy binarization threshold of 3D voxels, and combined with noise processing, a preliminary mesh is obtained.

[0060] It should be noted that obtaining the preliminary three-dimensional reconstruction based on the preliminary mesh specifically involves performing mesh subdivision processing on the preliminary mesh to obtain the preliminary three-dimensional reconstruction.

[0061] Specifically, the terminal device can first receive the original image of the target to be reconstructed, then identify the 2D stereoscopic object to obtain the point cloud data of the target to be reconstructed, then extract 3D voxels, establish a 3D mesh, and then obtain a preliminary 3D reconstructed body.

[0062] Regarding the extraction of 3D voxels, the terminal device can predict the probability network occupied by a single voxel to give the 3D shape of each detected geometry when inputting a sample array (X,Y,Z). i The cube region occupied by the building is labeled G×G×G, which predicts the complete 3D shape of the region. The occupancy probability of each 3D voxel and the occupancy binarization threshold are input. Each occupied voxel is replaced by a cuboid triangle mesh, merging shared vertices and edges between adjacent occupied voxels.

[0063] After obtaining the initial mesh, the terminal device can perform mesh subdivision processing to obtain a preliminary 3D reconstructed volume. Specifically, after the initial mesh is established, a coarse 3D shape is provided by a 3D voxel mesh. This initial mesh is then processed through mesh subdivision branches, refining its vertex positions with a series of subdivision stages.

[0064] First, generate a feature vector aligned with the image for each grid vertex. Given a feature map, compute a bilinear interpolated image feature at each projected vertex location.

[0065] Secondly, information is propagated along the mesh edges. Given the input vertex features {fi}, the updated features f′ are calculated using equation (1). i .

[0066] f′ i =ReLu(W0f i +∑ j∈N(i) W1f j (1),

[0067] Where N(i) represents the neighbors of the i-th vertex in the initial mesh, and W0 and W1 are the weight matrices learned by the 3D reconstruction network. Each stage of the mesh subdivision branch uses several graph convolutional layers to aggregate information on the local mesh region.

[0068] Finally, the updated vertex positions are calculated, and the mesh geometry is updated. Each stage of the mesh subdivision branch ends with vertex subdivision, generating an intermediate mesh output. When updating vertex positions, as the loss function approaches convergence, the initial 3D volume reconstruction prediction is complete, and the result is output.

[0069] Regarding the loss function of the 3D reconstruction network, when the loss function gradually converges, the model is considered to have reached its optimum, and the fitting stops. In this embodiment, an edge length regularization term and a chamfer loss function are used to improve the MeshCNN network through a linearly cumulative additive method to obtain the 3D reconstruction network. The loss function can be:

[0070] L = aL chamfer +bL edge (2),

[0071] in,

[0072]

[0073]

[0074] Where a and b represent configurable hyperparameters, p represents the vertex of the generator network, p^ represents the point sampled from the surface of the generator network, q represents a point in the input point cloud data, N(p) represents the ring adjacent to the vertex in the generator network, and L chamfer L represents the distance between the initial 3D reconstructed objects. edge This represents the edge length regularization term.

[0075] In this embodiment, unlike previous construction methods, an edge length regularization term is added to the chamfer loss function, which can accelerate the convergence speed of the 3D reconstruction network and greatly improve the accuracy of the initial 3D reconstruction.

[0076] In step S120, the terminal device will take the intersection of several preliminary three-dimensional reconstructions to obtain the real three-dimensional reconstruction.

[0077] By adding different noises to a single image of the target to be reconstructed, and obtaining several preliminary 3D reconstructed bodies based on the 3D reconstruction network, the final reconstruction target of the 3D reconstruction network can be regarded as the intersection of several preliminary 3D reconstructed bodies.

[0078] Specifically, step S120 may include:

[0079] Measure the distance between every two of the initial 3D reconstructed volumes;

[0080] Based on the distance between every two of the initial 3D reconstructed volumes, a distance metric loss function is obtained for the target to be reconstructed between different views;

[0081] Based on the distance metric loss function between different views of the target to be reconstructed, a true 3D reconstructed object is obtained.

[0082] The terminal device can measure the distance L between every two preliminary 3D reconstructed bodies using the following formula (5) based on the 3D reconstruction network. chamfer The distance metric loss function between different views of the target to be reconstructed is obtained by the following formula (6). The image under different noise conditions is fitted and optimized to obtain the intersection and the real three-dimensional reconstruction.

[0083]

[0084]

[0085] Where S i This represents the results under different views of the target to be reconstructed, i.e., S i S represents several preliminary 3D reconstructed volumes obtained by adding noise to the i-th image of the target to be reconstructed. j This represents several preliminary 3D reconstructed bodies of the j-th image of the target to be reconstructed after adding noise.

[0086] By first obtaining several preliminary 3D reconstructed objects based on point cloud data with reconstruction targets using a 3D reconstruction network, and then taking the intersection of these preliminary 3D reconstructed objects, a more realistic 3D reconstructed object with clearer structure and stronger target boundary sense can be obtained, so as to better simulate and improve the 3D reconstruction effect.

[0087] In step S130, the terminal device optimizes the quality of the real three-dimensional reconstruction through a two-dimensional reconstruction network to obtain the final three-dimensional reconstruction.

[0088] It should be noted that the terminal device can pre-construct a 2D reconstruction network using an improved fully convolutional network semantic segmentation model. Through this 2D reconstruction network, the terminal device can refine the initial mesh of the real 3D reconstructed object, thereby effectively improving the structural clarity and texture quality of the real 3D reconstructed object.

[0089] Specifically, when the terminal device constructs a two-dimensional reconstruction network using the improved fully convolutional network semantic segmentation model, it can first create a spatial coordinate mapping and then refine the spatial coordinate mapping to train the two-dimensional reconstruction network.

[0090] To create spatial coordinate mappings, you can first create a UV atlas based on the initial mesh of the real 3D reconstructed object. Then, for each point q in the input point cloud, find its nearest neighbor on the initial mesh. Then query the triangle face ID and centroid coordinates inside the triangle to obtain... The UV coordinates of the texture atlas are then calculated, where the (x, y, z) coordinates of q are randomly selected for traversal. Finally, a sparse 3D XYZ map is created in UV space to record the 3D positions of all points in the input point cloud.

[0091] Regarding the refinement of spatial coordinate mapping, a 2D sparse mapping projection of the real 3D reconstructed object is trained using a fully convolutional network semantic segmentation model. The structure of the 2D reconstruction network built based on the improved fully convolutional network semantic segmentation model is as follows: Figure 2 As shown. Terminal devices can improve the FCN model architecture to have a two-dimensional structure, dividing it into several blocks along the depth dimension. An early exit classifier is implemented at the end of each block, and flexibility is achieved through the possibility of stopping computation at any desired classifier. This allows for a thinner version of the complete network obtained by removing entire convolutional channels to achieve different widths. All subnetworks share their parameters except for batch standard statistics. The spatial vertex positions corresponding to the image of the target to be reconstructed are predicted through the two-dimensional reconstruction network, and the predicted spatial vertex positions are directly updated onto the 3D mesh of the real three-dimensional reconstructed object, completing iterative optimization.

[0092] By optimizing the quality of a real 3D reconstructed object, including its structure and texture, using a 2D reconstruction network, a high-quality final 3D reconstructed object can be obtained.

[0093] Furthermore, step S130 may include:

[0094] Based on the point cloud data of the target to be reconstructed, a sparse UV map is obtained;

[0095] Based on the two-dimensional reconstruction network, a dense UV map is obtained from the sparse UV map, which is used to optimize the mesh of the real three-dimensional reconstructed body to obtain the final three-dimensional reconstructed body.

[0096] It should be noted that the dense UV map includes a spatial location dense UV map and / or a texture dense UV map. The spatial location dense UV map can be used to optimize the spatial vertex position of the mesh of the real 3D reconstructed object, and the texture dense UV map can be used to optimize the texture, color, etc. of the mesh of the real 3D reconstructed object.

[0097] Optimizing the mesh of a realistic 3D reconstructed object using dense UV mapping via a 2D reconstruction network significantly improves its sharpness and texture, and effectively enhances its smoothness. The terminal device encodes the (r, g, b) values ​​of the point cloud data of the target object into a sparse UV map, and then reconstructs a dense UV map supervised by sparse signals from the input point cloud using a 2D reconstruction network. The iterative operation of the combined 2D and 3D reconstruction networks effectively improves the surface quality of the realistic 3D reconstructed object, resulting in a high-quality final 3D reconstructed object.

[0098] The 3D object reconstruction method provided in this application embodiment obtains several preliminary 3D reconstructed bodies without texture processing based on the point cloud data of the target to be reconstructed using a 3D reconstruction network. Then, the intersection of these preliminary 3D reconstructed bodies is obtained to get a structurally clear, realistic 3D reconstructed body. Finally, a 2D reconstruction network is used to optimize the quality of the realistic 3D reconstructed body, improving the target boundary perception and texture quality, resulting in a high-quality final 3D reconstructed body. The 3D object reconstruction method provided in this application embodiment combines a 2D reconstruction network and a 3D reconstruction network to achieve lightweight and efficient 3D object reconstruction, producing 3D reconstructed bodies with clear structure, strong target boundary perception, and high texture quality, without relying on high-level hardware or complex reconstruction techniques.

[0099] Furthermore, the 3D object reconstruction method provided in this application, by constructing a two-layer model and thinning the model, meets the reconstruction conditions for buildings with texture requirements of sharp boundaries and complex structures. For comparison, this application selects Poisson reconstruction as the baseline method for model comparison. The effectiveness of Poisson reconstruction and the 3D object reconstruction method provided in this application is compared for the same 3D map building sample and the public database (free3D). The F1-score is selected as the evaluation standard for the model, and the F1-score can be calculated by the following formula (7).

[0100]

[0101] Where P is the recall and R is the precision. The higher the F1 score, the better the model's performance.

[0102] The comparison results are shown in the figure below. Figure 3 As shown, in the scenario of building reconstruction in a 3D map, the 3D object reconstruction method provided by this application embodiment, which combines a 2D reconstruction network and a 3D reconstruction network, significantly outperforms existing baseline methods in terms of imaging quality. Furthermore, the 3D object reconstruction method provided by this application embodiment, through iterative optimization of the reconstructed body using both 2D and 3D reconstruction networks, effectively solves the problems of blurred edge definition of buildings in this scenario, as well as edge blurring and low texture caused by the complex and sharp structure of the buildings themselves. Compared to the benchmark comparison model, the performance is improved by approximately 5% on the same dataset in a 3D map scenario. The 3D object reconstruction method provided by this application embodiment can obtain 3D reconstructed bodies with clearer structures, stronger target boundary perception, and higher texture quality.

[0103] The 3D object reconstruction method provided in this application provides a complete process from the acquisition and storage of point cloud data of buildings to the establishment of algorithms and downstream applications, and outputs high-quality 3D reconstructed bodies. It is a complete and standard construction process for a 3D map building reconstruction system. Unlike the general approach of 3D reconstruction based on machine learning, this invention constructs a two-layer hybrid network for refined modeling. After obtaining the real 3D reconstructed body, it refines it through a 2D reconstruction network based on an improved fully convolutional semantic segmentation model. At the 2D level, it refines the texture for color and image granularity, and updates the image point coordinates to obtain a high-definition final 3D reconstructed body. It can be widely applied to the rapid prediction and reconstruction of targets on occluded 2D images. The method also has model reusability and generalization, and achieves fast and high-quality reconstruction of objects in a scene.

[0104] The 3D object reconstruction method provided in this application is independent of hardware capabilities and does not require high-quality images of the target object. By combining 2D and 3D reconstruction networks, it can achieve high-definition 3D reconstruction even with average image quality and without hardware limitations. Based on a complete closed-loop 3D reconstruction approach, it provides a holistic architecture process from building generation from a 3D map to detail optimization. Through iterative methods, it optimizes the clarity of the building objects globally, thereby generating high-texture, high-definition 3D buildings applicable to 3D maps. It fully considers the problems of blurred building boundaries and complex building structures in 3D maps, ultimately presenting high-texture, high-definition reconstructed images. The method is lightweight, simple, and advanced.

[0105] The three-dimensional object reconstruction apparatus provided in the embodiments of this application is described below. The three-dimensional object reconstruction apparatus described below and the three-dimensional object reconstruction method described above can be referred to in correspondence.

[0106] Figure 4 This is a schematic diagram of the structure of a three-dimensional object reconstruction device provided in an embodiment of this application.

[0107] Reference Figure 4 This application provides a three-dimensional object reconstruction apparatus, which may include:

[0108] The module 410 for obtaining preliminary 3D reconstructed bodies is used to: obtain several preliminary 3D reconstructed bodies through a 3D reconstruction network based on the point cloud data of the target to be reconstructed.

[0109] The module 420 for obtaining a real 3D reconstructed body is used to: take the intersection of several preliminary 3D reconstructed bodies to obtain a real 3D reconstructed body;

[0110] The final 3D reconstruction module 430 is used to: optimize the quality of the real 3D reconstruction through a 2D reconstruction network to obtain the final 3D reconstruction.

[0111] In one embodiment, the preliminary three-dimensional reconstruction module 410 may include:

[0112] The preliminary mesh is obtained by sub-modules, which are used to: obtain the preliminary mesh based on the point cloud data of the target to be reconstructed through a 3D reconstruction network combined with noise processing;

[0113] The preliminary 3D reconstruction submodule is used to: obtain a preliminary 3D reconstruction based on the preliminary mesh.

[0114] In one embodiment, the initial mesh obtaining sub-modules may include:

[0115] The 3D voxel extraction submodule is used to extract 3D voxels from the point cloud data of the target to be reconstructed through a 3D reconstruction network.

[0116] The noise processing submodule is used to obtain a preliminary mesh based on the occupancy probability and occupancy binarization threshold of 3D voxels, combined with noise processing.

[0117] In one embodiment, the preliminary three-dimensional reconstruction volume obtaining submodule is specifically used for:

[0118] Based on the initial mesh, mesh subdivision processing is performed to obtain a preliminary three-dimensional reconstructed body.

[0119] In one embodiment, the real 3D reconstructed body obtaining module 420 may include:

[0120] The distance measurement submodule is used to measure the distance between every two of the initial 3D reconstructed volumes.

[0121] The distance metric loss function acquisition submodule is used to: obtain the distance metric loss function of the target to be reconstructed between different views based on the distance between every two of the initial 3D reconstructed bodies;

[0122] The real 3D reconstruction submodule is used to: obtain the real 3D reconstruction based on the distance metric loss function between different views of the target to be reconstructed.

[0123] In one embodiment, the final 3D reconstructed body obtaining module 430 may include:

[0124] The sparse UV mapping submodule is used to: obtain a sparse UV mapping based on the point cloud data of the target to be reconstructed;

[0125] The dense UV mapping submodule is used to: obtain a dense UV map based on the sparse UV map using a 2D reconstruction network, and use it to optimize the mesh of the real 3D reconstructed body to obtain the final 3D reconstructed body.

[0126] It should be noted that the dense UV map includes spatial location dense UV map and / or texture dense UV map.

[0127] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call a computer program stored in the memory 830 to execute the steps of a three-dimensional object reconstruction method, such as:

[0128] Based on the point cloud data of the target to be reconstructed, several preliminary three-dimensional reconstructed bodies are obtained through a three-dimensional reconstruction network;

[0129] By taking the intersection of several preliminary 3D reconstructed bodies, the true 3D reconstructed body is obtained;

[0130] The quality of the real 3D reconstructed object is optimized by a 2D reconstruction network to obtain the final 3D reconstructed object.

[0131] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the three-dimensional object reconstruction method provided in the above embodiments, such as including:

[0133] Based on the point cloud data of the target to be reconstructed, several preliminary three-dimensional reconstructed bodies are obtained through a three-dimensional reconstruction network;

[0134] By taking the intersection of several preliminary 3D reconstructed bodies, the true 3D reconstructed body is obtained;

[0135] The quality of the real 3D reconstructed object is optimized by a 2D reconstruction network to obtain the final 3D reconstructed object.

[0136] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing a processor to execute the steps of the three-dimensional object reconstruction method provided in the above embodiments, such as including:

[0137] Based on the point cloud data of the target to be reconstructed, several preliminary three-dimensional reconstructed bodies are obtained through a three-dimensional reconstruction network;

[0138] By taking the intersection of several preliminary 3D reconstructed bodies, the true 3D reconstructed body is obtained;

[0139] The quality of the real 3D reconstructed object is optimized by a 2D reconstruction network to obtain the final 3D reconstructed object.

[0140] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0141] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0142] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for reconstructing a three-dimensional object, characterized in that, include: Based on the point cloud data of the target to be reconstructed, several preliminary three-dimensional reconstructed bodies are obtained through a three-dimensional reconstruction network; By taking the intersection of several preliminary 3D reconstructed bodies, the true 3D reconstructed body is obtained; The quality of the real 3D reconstructed object is optimized by a 2D reconstruction network to obtain the final 3D reconstructed object. The process involves obtaining several preliminary 3D reconstructed volumes from the point cloud data of the target to be reconstructed through a 3D reconstruction network, including: Based on the point cloud data of the target to be reconstructed, three-dimensional voxels are extracted through a three-dimensional reconstruction network; Based on the occupancy probability and occupancy binarization threshold of three-dimensional voxels, and combined with noise processing, a preliminary mesh is obtained; Based on the initial mesh, mesh subdivision processing is performed to obtain a preliminary three-dimensional reconstructed volume; The process of subdividing the initial mesh to obtain a preliminary 3D reconstructed body includes: Generate an image-aligned feature vector for each grid vertex in the initial grid; Information is propagated along the grid edges using graph convolutional layers to update vertex features; The updated vertex positions are calculated based on the updated vertex features, and the geometry of the preliminary 3D reconstructed body is optimized using a loss function. The loss function is: ； in, ； ； in, and This indicates configurable hyperparameters. Represents the vertices of the generated network. This represents the points used for sampling the surface of the generative network. This represents a point in the input point cloud data. This represents a cycle adjacent to a vertex in the generated network. Indicates the distance between the initial 3D reconstructed objects. This represents the edge length regularization term.

2. The three-dimensional object reconstruction method according to claim 1, characterized in that, The step of finding the intersection of several preliminary 3D reconstructed bodies to obtain a true 3D reconstructed body includes: Measure the distance between every two of the initial 3D reconstructed volumes; Based on the distance between every two of the initial 3D reconstructed volumes, a distance metric loss function is obtained for the target to be reconstructed between different views; Based on the distance metric loss function between different views of the target to be reconstructed, a true 3D reconstructed object is obtained.

3. The three-dimensional object reconstruction method according to claim 1, characterized in that, The process of optimizing the quality of the real 3D reconstructed object through a 2D reconstruction network to obtain the final 3D reconstructed object includes: Based on the point cloud data of the target to be reconstructed, a sparse UV map is obtained; Based on the two-dimensional reconstruction network, a dense UV map is obtained from the sparse UV map, which is used to optimize the mesh of the real three-dimensional reconstructed body to obtain the final three-dimensional reconstructed body.

4. The three-dimensional object reconstruction method according to claim 3, characterized in that, The dense UV map includes a spatial location dense UV map and / or a texture dense UV map.

5. A three-dimensional object reconstruction device, characterized in that, include: The module for obtaining preliminary 3D reconstructed bodies is used to: obtain several preliminary 3D reconstructed bodies from the point cloud data of the target to be reconstructed through a 3D reconstruction network; The module for obtaining a true 3D reconstructed body is used to: take the intersection of several preliminary 3D reconstructed bodies to obtain a true 3D reconstructed body; The final 3D reconstruction module is used to: optimize the quality of the real 3D reconstruction through a 2D reconstruction network to obtain the final 3D reconstruction. The module for obtaining the preliminary 3D reconstructed body is further configured to: extract 3D voxels from the point cloud data of the target to be reconstructed using a 3D reconstruction network; obtain a preliminary mesh based on the occupancy probability and occupancy binarization threshold of the 3D voxels, combined with noise processing; and perform mesh subdivision processing based on the preliminary mesh to obtain the preliminary 3D reconstructed body. The module for obtaining the preliminary 3D reconstructed volume is further configured to: generate a feature vector aligned with the image for each grid vertex in the preliminary grid; and update the vertex features by propagating information along the grid edges through a graph convolutional layer. The updated vertex positions are calculated based on the updated vertex features, and the geometry of the preliminary 3D reconstructed body is optimized using a loss function. The loss function is: ； in, ； ； in, and This indicates configurable hyperparameters. Represents the vertices of the generated network. This represents the points used for sampling the surface of the generative network. This represents a point in the input point cloud data. This represents a cycle adjacent to a vertex in the generated network. Indicates the distance between the initial 3D reconstructed objects. This represents the edge length regularization term.

6. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the three-dimensional object reconstruction method according to any one of claims 1 to 4.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional object reconstruction method according to any one of claims 1 to 4.

Images