Three-dimensional reconstruction method and device, equipment, storage medium and product
By decoupling the input of point cloud data into the learning, reconstruction, and refinement networks, the problem of computational complexity in 3D point cloud reconstruction is solved, achieving efficient 3D reconstruction.
Patent Information
- Application Number
- CN202510422231.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-03
- Publication Date
- 2025-12-12
AI Technical Summary
Existing technologies using point clouds for 3D reconstruction are computationally complex, which affects reconstruction efficiency.
The surface point cloud data of the target object is input into the learning network for feature extraction to obtain the latent vector; the latent vector and the overall image of the target object are input into the reconstruction network for data reconstruction to obtain the reconstructed point cloud data; the reconstructed point cloud data is input into the thinning network for thinning to obtain the thinned point cloud data; based on the thinned point cloud data, the 3D reconstruction result of the target object is determined.
It simplifies the 3D reconstruction task, improves reconstruction efficiency, enables efficient structured or parametric restoration, and accurately performs geometric reconstruction.
Smart Images
Figure CN121120980A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to three-dimensional reconstruction methods, apparatus, devices, storage media, and products. Background Technology
[0002] Reconstructing 3D objects from monochrome RGB (Red, Green, Blue) images is fundamental to technologies such as augmented reality, mixed reality, holographic teleportation, and the metaverse. Currently, point cloud-based 3D reconstruction is commonly used. Point clouds are irregular data structures that use a set of points to represent objects. Point clouds are typically represented as 3D matrices, or as raster data encoding three channels, or as depth maps. They are concise and flexible, and can be used to represent complex geometric and topological structures as well as fine geometric details. However, they lack the concept of structure or parameterization and rely on complex end-to-end architectures, making it difficult to regress from images using modern deep networks, thus affecting reconstruction efficiency.
[0003] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention
[0004] The main purpose of this application is to provide a three-dimensional reconstruction method, apparatus, device, storage medium and product, which aims to solve the technical problem that the calculation is complicated and affects the reconstruction efficiency when using point clouds for three-dimensional reconstruction in the prior art.
[0005] To achieve the above objectives, this application provides a three-dimensional reconstruction method, the method comprising:
[0006] The surface point cloud data of the target object is input into the learning network for feature extraction to obtain the latent vector;
[0007] The latent vector and the overall image of the target object are input into a reconstruction network to reconstruct data, thereby obtaining reconstructed point cloud data.
[0008] The reconstructed point cloud data is input into a thinning network for thinning processing to obtain thinned point cloud data;
[0009] Based on the refined point cloud data, the 3D reconstruction result of the target object is determined.
[0010] In one embodiment, the learning network includes a point encoder, a first fully connected unit, a first quantization layer, and a first folding decoder connected in sequence. The step of inputting the surface point cloud data of the target object into the learning network for feature extraction to obtain a latent vector includes:
[0011] The surface point cloud data of the target object is input into the point encoder for global feature extraction to obtain a global feature vector;
[0012] The global feature vector is input into the first fully connected unit for nonlinear transformation to obtain the transformed vector;
[0013] The transformed vector is input into the first quantization layer for quantization to obtain the latent vector;
[0014] The latent vector is input into the first folding decoder for decoding to obtain three-dimensional point cloud data.
[0015] In one embodiment, the reconstruction network includes an image encoder, an average pooling layer, a second fully connected unit, a second quantization layer, and a second folded decoder connected in sequence. The step of inputting the latent vector and the overall image of the target object into the reconstruction network for data reconstruction to obtain reconstructed point cloud data includes:
[0016] The overall image of the target object is input into the image encoder for feature extraction to obtain the image feature vector;
[0017] The image feature vector is input into the average pooling layer for processing to obtain the target feature vector;
[0018] The target feature vector and the latent vector are input into the second fully connected unit. Based on the second fully connected unit, the target feature vector is mapped onto the latent vector to obtain the mapped latent vector.
[0019] The mapped latent vector is input into the second quantization layer for quantization to obtain the target latent vector;
[0020] The target latent vector is input into the second folding decoder for decoding to obtain the reconstructed point cloud data.
[0021] In one embodiment, the step of determining the 3D reconstruction result of the target object based on the refined point cloud data includes:
[0022] Obtain the 3D point cloud data corresponding to the decoded potential vector;
[0023] Based on the potential distance between the 3D point cloud data and the reconstructed point cloud data, the learning reconstruction loss is determined;
[0024] Based on the refined point cloud data and the chamfer distance between the 3D point cloud data and the reconstructed point cloud data, the refinement loss is determined;
[0025] The reconstruction network and the learning network are trained based on the learning reconstruction loss, and the refinement network is trained based on the refinement loss;
[0026] Upon completion of training, the output of the refined network is used as the 3D reconstruction result of the target object.
[0027] In one embodiment, the step of determining the learning reconstruction loss based on the potential distance between the 3D point cloud data and the reconstructed point cloud data includes:
[0028] Calculate the potential distance between the 3D point cloud data and the reconstructed point cloud data;
[0029] Feature extraction is performed on the three-dimensional point cloud data and the reconstructed point cloud data respectively to obtain a first training feature vector and a second training feature vector, and the mean square error between the first training feature vector and the second training feature vector is calculated.
[0030] Obtain the first correspondence between the potential distance, mean squared error, and learning reconstruction loss;
[0031] The learning reconstruction loss is obtained based on the potential distance between the 3D point cloud data and the reconstructed point cloud data, the mean square error between the first training feature vector and the second training feature vector, and the first correspondence.
[0032] In one embodiment, the step of determining the refinement loss based on the refined point cloud data and the chamfer distance between the 3D point cloud data and the reconstructed point cloud data includes:
[0033] Calculate the chamfer distance between the 3D point cloud data and the reconstructed point cloud data;
[0034] Obtain the second correspondence between chamfer distance, refined point cloud data, and refinement loss;
[0035] The refinement loss is obtained based on the chamfer distance between the 3D point cloud data and the reconstructed point cloud data, the refined point cloud data, and the second correspondence.
[0036] Furthermore, to achieve the above objectives, this application also proposes a three-dimensional reconstruction apparatus, which includes:
[0037] The learning module is used to input the surface point cloud data of the target object into the learning network for feature extraction to obtain the latent vector;
[0038] The reconstruction module is used to input the latent vector and the overall image of the target object into the reconstruction network for data reconstruction to obtain reconstructed point cloud data;
[0039] The refinement module is used to input the reconstructed point cloud data into the refinement network for refinement processing to obtain refined point cloud data.
[0040] The training module is used to determine the 3D reconstruction result of the target object based on the refined point cloud data.
[0041] In addition, to achieve the above objectives, this application also proposes a three-dimensional reconstruction device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the three-dimensional reconstruction method as described above.
[0042] In addition, to achieve the above objectives, the present invention also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the three-dimensional reconstruction method described above.
[0043] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the three-dimensional reconstruction method described above.
[0044] This application provides a 3D reconstruction method. The method involves inputting surface point cloud data of a target object into a learning network for feature extraction to obtain latent vectors. These latent vectors, along with the overall image of the target object, are then input into a reconstruction network for data reconstruction, resulting in reconstructed point cloud data. This reconstructed point cloud data is then input into a thinning network for thinning processing, yielding refined point cloud data. Based on the refined point cloud data, the 3D reconstruction result of the target object is determined. This application decouples the 3D reconstruction task from the parametric learning task, making the 3D reconstruction of arbitrary topological objects easier to process. It enables efficient structured or parametric restoration and accurate geometric reconstruction, simplifying the 3D reconstruction task, improving its efficiency, and solving the technical problem of computational complexity and reduced reconstruction efficiency when using point clouds for 3D reconstruction. Attached Figure Description
[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0047] Figure 1 This is a flowchart illustrating an embodiment of the three-dimensional reconstruction method of this application;
[0048] Figure 2A schematic diagram of the learning network structure of the three-dimensional reconstruction method provided in Embodiment 1 of this application;
[0049] Figure 3 This is a schematic diagram of the folded decoder structure of the three-dimensional reconstruction method provided in Embodiment 1 of this application;
[0050] Figure 4 This is a schematic diagram of the reconstruction network structure of the three-dimensional reconstruction method provided in Embodiment 1 of this application;
[0051] Figure 5 A detailed network structure diagram of the three-dimensional reconstruction method provided in Embodiment 1 of this application;
[0052] Figure 6 This is a flowchart illustrating Embodiment 2 of the three-dimensional reconstruction method of this application;
[0053] Figure 7 This is a schematic diagram of the mean square error calculation process for the three-dimensional reconstruction method provided in Embodiment 2 of this application.
[0054] Figure 8 This is a schematic diagram of the modular structure of the three-dimensional reconstruction device according to an embodiment of this application;
[0055] Figure 9 This is a schematic diagram of the device structure of the hardware operating environment involved in the three-dimensional reconstruction method in this application embodiment.
[0056] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0057] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.
[0058] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.
[0059] The main solution of this application embodiment is as follows: input the surface point cloud data of the target object into the learning network for feature extraction to obtain the latent vector; input the latent vector and the overall image of the target object into the reconstruction network for data reconstruction to obtain the reconstructed point cloud data; input the reconstructed point cloud data into the thinning network for thinning processing to obtain the thinned point cloud data; and determine the three-dimensional reconstruction result of the target object based on the thinned point cloud data.
[0060] Currently, point cloud-based 3D reconstruction is commonly used. Point clouds can be used to represent complex geometric and topological structures as well as fine geometric details, but they lack the concept of structure or parameterization and rely on complex end-to-end architectures. Therefore, it is difficult to use modern deep networks to regress from images, which affects the efficiency of reconstruction.
[0061] This application provides a solution that decouples the 3D reconstruction task from the parametric learning task, making the 3D reconstruction of arbitrary topological objects easier to process, enabling efficient structured or parametric restoration, and accurate geometric reconstruction. This simplifies the 3D reconstruction task, improves the efficiency of 3D reconstruction, and solves the technical problem of computational complexity and reduced reconstruction efficiency when using point clouds for 3D reconstruction.
[0062] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or 3D reconstruction device capable of performing the above functions. This embodiment does not specifically limit the scope of the embodiment. The following uses a 3D reconstruction device as an example to describe this embodiment and the following embodiments.
[0063] This application provides a three-dimensional reconstruction method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the three-dimensional reconstruction method of this application.
[0064] In this embodiment, the three-dimensional reconstruction method includes steps S10 to S40:
[0065] Step S10: Input the surface point cloud data of the target object into the learning network for feature extraction to obtain the latent vector;
[0066] It should be noted that the target object is the object to be reconstructed in 3D, and the surface point cloud data is a point cloud randomly sampled from the surface of the target object, without any concept of structure. The latent vector is the latent representation extracted from the surface point cloud data.
[0067] Additionally, it should be noted that the reference Figure 2 The learning network includes a point encoder, a first fully connected unit, a first quantization layer, and a first folded decoder connected in sequence. The first fully connected unit consists of two fully connected layers (FC).
[0068] In one feasible implementation, step S10 may include steps S101 to S104:
[0069] Step S101: Input the surface point cloud data of the target object into the point encoder for global feature extraction to obtain a global feature vector;
[0070] It is understood that this embodiment uses a point encoder to perform global feature extraction on the surface point cloud data, and the resulting feature vector is the global feature vector. In specific implementations, a global feature vector with a dimension of 1024 is typically extracted and passed to the first fully connected unit.
[0071] Step S102: Input the global feature vector into the first fully connected unit for nonlinear transformation to obtain the transformed vector;
[0072] It should be noted that both fully connected layers in the first fully connected unit consist of cascaded 512 units and use the ReLU (Rectified Linear Unit) activation function. The vector obtained after processing the global feature vector through the first fully connected unit is the transformed vector.
[0073] Step S103: Input the transformation vector into the first quantization layer for quantization to obtain the latent vector;
[0074] It should be noted that the first quantization layer uses the ReLU activation function. The input transformed vector is quantized using the first quantization layer, and the final vector obtained is the latent vector.
[0075] Step S104: Input the latent vector into the first folding decoder for decoding to obtain three-dimensional point cloud data.
[0076] It should be noted that this embodiment uses a first folding decoder to decode the latent vectors into a three-dimensional point cloud, i.e., three-dimensional point cloud data. A 45×45 point grid is used, with points evenly spaced within a size of [-0.3, 0.3]. 2 On the square. (Reference) Figure 3 The first folding decoder comprises six deconvolutional layers, enabling two deconvolution-based folding operations. The first folding operation uses three cascaded deconvolutional layers to map a fixed two-dimensional mesh to three-dimensional space, with output spaces of 512, 512, and 3, respectively. The second folding operation uses three cascaded deconvolutional networks to map the output of the first folding operation into a point cloud.
[0077] Understandably, if the learning network is well regularized during the training phase, the points generated by the learning network can be structured, and these structured points can be more easily reconstructed for the network to learn and understand.
[0078] Step S20: Input the latent vector and the overall image of the target object into the reconstruction network to reconstruct the data and obtain the reconstructed point cloud data;
[0079] It should be noted that the overall image of the target object refers to its overall RGB image, which is typically a monochrome RGB image with a size of 224×224. The 3D data obtained after reconstruction using a reconstruction network is the reconstructed point cloud data.
[0080] In its implementation, the reconstruction network maps a 224×224 RGB image onto a latent vector generated by the learning network. Therefore, both the latent vector and the overall image can be considered as inputs to the reconstruction network.
[0081] Additionally, it should be noted that the reference Figure 4 The reconstruction network consists of an image encoder, an average pooling layer, a second fully connected unit, a second quantization layer, and a second folded decoder connected in sequence. The second fully connected unit is composed of three cascaded fully connected layers (FC).
[0082] In one feasible implementation, step S20 may include: inputting the overall image of the target object into the image encoder for feature extraction to obtain an image feature vector; inputting the image feature vector into the average pooling layer for processing to obtain a target feature vector; inputting the target feature vector and the latent vector into the second fully connected unit, and based on the second fully connected unit, mapping the target feature vector onto the latent vector to obtain a mapped latent vector; inputting the mapped latent vector into the second quantization layer for quantization to obtain a target latent vector; and inputting the target latent vector into the second folding decoder for decoding to obtain the reconstructed point cloud data.
[0083] It should be noted that this embodiment uses an image encoder (graphic encoder) to extract features from the entire image, and the resulting feature vector is the image feature vector. Inputting the image feature vector into an average pooling layer for processing yields a 7×7×512 latent representation, which is the target feature vector.
[0084] Understandably, the target feature vector is processed using three cascaded fully connected layers, resulting in the mapped latent vector. Each fully connected layer uses the ReLU activation function. The first fully connected layer maps the flattened target feature vector to the second fully connected layer, which in turn maps it to the third fully connected layer. The third fully connected layer then maps the target feature vector to the latent vector generated by the learning network, yielding the mapped latent vector. The second quantization layer quantizes this mapped latent vector to obtain the latent vector used during reconstruction, i.e., the target latent vector.
[0085] It should be understood that the second fold decoder has the same structure and function as the first fold decoder, and will not be described in detail here. By using the second fold decoder to decrypt the target latent vector, reconstructed point cloud data can be obtained.
[0086] Step S30: Input the reconstructed point cloud data into a thinning network for thinning processing to obtain thinned point cloud data;
[0087] It should be noted that the reconstruction network is trained under the supervision of the learning network, therefore its performance is limited by the performance of the learning network. To overcome this limitation, this embodiment proposes a refinement network to refine the data generated by the reconstruction network. The data output after processing by the refinement network is the refined point cloud data.
[0088] It is understandable that the refinement network receives the point cloud data output by the reconstruction network as input and outputs a set of refined points. In this embodiment, reference is made to... Figure 5 The thinning network consists of 6 graph convolutional layers and 1 surface pooling layer. The 6 graph convolutional layers have 16, 64, 128, 64, 64, 16, and 3 kernels respectively, and the surface pooling layer can extract surface points from the thinned image.
[0089] Step S40: Based on the refined point cloud data, determine the three-dimensional reconstruction result of the target object.
[0090] Understandably, the 3D reconstruction result is the final reconstructed data. To ensure the accuracy of the reconstruction, the learning network, reconstruction network, and refinement network need to be trained to obtain the final 3D reconstruction result.
[0091] This embodiment provides a 3D reconstruction method. The surface point cloud data of the target object is input into a learning network for feature extraction to obtain a latent vector. The latent vector and the overall image of the target object are then input into a reconstruction network for data reconstruction to obtain reconstructed point cloud data. This reconstructed point cloud data is then input into a thinning network for thinning processing to obtain refined point cloud data. Based on the refined point cloud data, the 3D reconstruction result of the target object is determined. This embodiment decouples the 3D reconstruction task from the parametric learning task, making the 3D reconstruction of arbitrary topological objects easier to process. It enables efficient structured or parametric restoration and accurate geometric reconstruction, simplifying the 3D reconstruction task and improving its efficiency.
[0092] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 6 Step S40 may include steps S401 to S405:
[0093] Step S401: Obtain the 3D point cloud data corresponding to the decoded potential vector;
[0094] It should be noted that by inputting the latent vector into the first folded decoder in the learning network, the corresponding 3D point cloud data can be obtained.
[0095] Step S402: Determine the learning reconstruction loss based on the potential distance between the 3D point cloud data and the reconstructed point cloud data;
[0096] It should be noted that the learning reconstruction loss is the loss incurred by both the learning network and the reconstruction network. In this embodiment, the latent distance is used to calculate the learning reconstruction loss, which is then used to train both the learning network and the reconstruction network.
[0097] In one feasible implementation, step S402 may include: calculating the potential distance between the 3D point cloud data and the reconstructed point cloud data; performing feature extraction on the 3D point cloud data and the reconstructed point cloud data respectively to obtain a first training feature vector and a second training feature vector, and calculating the mean square error between the first training feature vector and the second training feature vector; obtaining a first correspondence between the potential distance, the mean square error, and the learning reconstruction loss; and obtaining the learning reconstruction loss based on the potential distance between the 3D point cloud data and the reconstructed point cloud data, the mean square error between the first training feature vector and the second training feature vector, and the first correspondence.
[0098] It should be noted that the formula for calculating the potential distance is as follows:
[0099]
[0100] In the formula, x1 represents the 3D point cloud data, x rec This represents the reconstruction of point cloud data, LD(x1, x...). rec H represents the potential distance between the 3D point cloud data and the reconstructed point cloud data, H represents the point encoder of the learning network, and H(x1) represents the output of the 3D point cloud data after inputting into the point encoder. rec ) represents the output after reconstructing the point cloud data input to the point encoder.
[0101] Additionally, it should be noted that the latent distance remains unchanged regardless of the ordering of the input points. However, the reconstruction network does not need to address the unstructured nature of the point cloud representation. Therefore, the structural information recovered by the learning network can be transferred to the reconstruction network. In this embodiment, this is achieved by matching the latent vectors of the learning network with the latent vectors of the reconstruction network using mean squared error (MSE). For specific implementation details, refer to... Figure 7The 3D point cloud data and the reconstructed point cloud data are respectively input into the point encoder for feature extraction to obtain the potential vectors used to calculate the mean square error, namely the first training feature vector and the second training feature vector.
[0102] Understandably, the primary correspondence between latent distance, mean squared error, and learning reconstruction loss—that is, the formula for calculating the learning reconstruction loss—is as follows:
[0103] Loss1 = LD(x1, x rec )+MSE(Lt2(I),H(x1))
[0104] In the formula, Loss1 represents the learning reconstruction loss, LD(x1, x rec Let H(x1) represent the potential distance between the 3D point cloud data and the reconstructed point cloud data, H(x1) represent the first training feature vector, Lt2(I) represent the second training feature vector, and MSE(Lt2(I),H(x1)) represent the mean squared error between the first and second training feature vectors. The potential distance between the 3D point cloud data and the reconstructed point cloud data, as well as the mean squared error between the first and second training feature vectors, are calculated separately. Substituting these values into the first correspondence mentioned above, the learning reconstruction loss can be calculated.
[0105] Step S403: Determine the refinement loss based on the refined point cloud data and the chamfer distance between the 3D point cloud data and the reconstructed point cloud data;
[0106] It should be noted that the formula for calculating the chamfer distance is as follows:
[0107]
[0108] In the formula, x1 represents three-dimensional point cloud data, a∈x1, x rec This represents reconstructed point cloud data, b∈x rec CD(x1,x) rec The chamfer distance between the 3D point cloud data and the reconstructed point cloud data is represented by ).
[0109] Additionally, it should be noted that the chamfer distance does not establish a one-to-one mapping between the predicted point and the target's 3D point, making it insensitive to subtle deformations. Since the reconstruction network is trained under the supervision of the learning network, it inherits the chamfer distance limitation from the learning network. To overcome this limitation, this embodiment uses an adversarial loss function to train the refinement network, adding the chamfer loss to the loss term to ensure that the refinement network reconstructs the correct object.
[0110] Understandably, the second correspondence between the chamfer distance, the refined point cloud data, and the refinement loss—that is, the calculation formula for the refinement loss—is as follows:
[0111] Loss2 = BCE(Disc(Rf(x) rec )),y real )+1000×CD(x1,x rec )
[0112] In the formula, Loss2 represents the refinement loss, x1 represents the 3D point cloud data, and x rec This represents reconstructed point cloud data, BCE represents the binary cross-entropy function, Rf is the refined network, and Rf(x) represents the reconstructed point cloud data. rec ) represents refined point cloud data, y real These are annotations used to train the refined network, CD(x1,x) rec The chamfer distance between the 3D point cloud data and the reconstructed point cloud data is represented by . Multiplying the chamfer distance by 1000 is to align it with the size of the BCE, thus avoiding modal collapse. The chamfer distance between the 3D point cloud data and the reconstructed point cloud data is calculated. Substituting the chamfer distance between the 3D point cloud data and the reconstructed point cloud data, along with the refined point cloud data, into the second correspondence mentioned above, the refinement loss can be calculated.
[0113] Step S404: Train the reconstruction network and the learning network based on the learning reconstruction loss, and train the refinement network based on the refinement loss;
[0114] Understandably, if the learning reconstruction loss is greater than the set threshold, the parameters of the reconstruction network and the learning network are updated, the 3D point cloud data and the reconstructed point cloud data are redefined, and the learning reconstruction loss is recalculated; if the learning reconstruction loss is less than or equal to the set threshold, it means that the reconstruction network and the learning network have been trained and can be applied.
[0115] It should be understood that if the refinement loss is greater than the set threshold, the parameters of the refinement network will be updated, the refinement point cloud data will be redefined, and the refinement loss will be recalculated; if the refinement loss is less than or equal to the set threshold, it means that the refinement network has been trained and can be applied.
[0116] Step S405: Upon completion of training, the output of the refined network is used as the 3D reconstruction result of the target object.
[0117] Understandably, once the learning network, reconstruction network, and refinement network have all been trained, the output of the refinement network is the final required 3D reconstruction result.
[0118] This embodiment provides a 3D reconstruction method that acquires 3D point cloud data corresponding to the latent vector decoding; determines the learning reconstruction loss based on the latent distance between the 3D point cloud data and the reconstructed point cloud data; determines the thinning loss based on the refined point cloud data and the chamfer distance between the 3D point cloud data and the reconstructed point cloud data; trains the reconstruction network and the learning network based on the learning reconstruction loss, and trains the thinning network based on the thinning loss; upon completion of training, the output of the thinning network is used as the 3D reconstruction result of the target object. This embodiment decouples the 3D reconstruction task from the parametric learning task, making the 3D reconstruction of arbitrary topological objects easier to process, enabling efficient structured or parametric restoration, and accurate geometric reconstruction, thus simplifying the 3D reconstruction task and improving the efficiency of 3D reconstruction.
[0119] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the three-dimensional reconstruction method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.
[0120] This application also provides a three-dimensional reconstruction device, please refer to... Figure 8 The three-dimensional reconstruction device includes:
[0121] Learning module 10 is used to input the surface point cloud data of the target object into the learning network for feature extraction to obtain the latent vector;
[0122] Reconstruction module 20 is used to input the latent vector and the overall image of the target object into the reconstruction network for data reconstruction to obtain reconstructed point cloud data;
[0123] The refinement module 30 is used to input the reconstructed point cloud data into the refinement network for refinement processing to obtain refined point cloud data;
[0124] Training module 40 is used to determine the three-dimensional reconstruction result of the target object based on the refined point cloud data.
[0125] In one feasible implementation, the learning module 10 is further configured to input the surface point cloud data of the target object into the point encoder for global feature extraction to obtain a global feature vector.
[0126] The global feature vector is input into the first fully connected unit for nonlinear transformation to obtain the transformed vector;
[0127] The transformed vector is input into the first quantization layer for quantization to obtain the latent vector;
[0128] The latent vector is input into the first folding decoder for decoding to obtain three-dimensional point cloud data.
[0129] In one feasible implementation, the reconstruction module 20 is further configured to input the overall image of the target object into the image encoder for feature extraction to obtain an image feature vector;
[0130] The image feature vector is input into the average pooling layer for processing to obtain the target feature vector;
[0131] The target feature vector and the latent vector are input into the second fully connected unit. Based on the second fully connected unit, the target feature vector is mapped onto the latent vector to obtain the mapped latent vector.
[0132] The mapped latent vector is input into the second quantization layer for quantization to obtain the target latent vector;
[0133] The target latent vector is input into the second folding decoder for decoding to obtain the reconstructed point cloud data.
[0134] In one feasible implementation, the training module 40 is further configured to acquire the three-dimensional point cloud data corresponding to the decoded latent vector;
[0135] Based on the potential distance between the 3D point cloud data and the reconstructed point cloud data, the learning reconstruction loss is determined;
[0136] Based on the refined point cloud data and the chamfer distance between the 3D point cloud data and the reconstructed point cloud data, the refinement loss is determined;
[0137] The reconstruction network and the learning network are trained based on the learning reconstruction loss, and the refinement network is trained based on the refinement loss;
[0138] Upon completion of training, the output of the refined network is used as the 3D reconstruction result of the target object.
[0139] In one feasible implementation, the training module 40 is further configured to calculate the potential distance between the 3D point cloud data and the reconstructed point cloud data;
[0140] Feature extraction is performed on the three-dimensional point cloud data and the reconstructed point cloud data respectively to obtain a first training feature vector and a second training feature vector, and the mean square error between the first training feature vector and the second training feature vector is calculated.
[0141] Obtain the first correspondence between the potential distance, mean squared error, and learning reconstruction loss;
[0142] The learning reconstruction loss is obtained based on the potential distance between the 3D point cloud data and the reconstructed point cloud data, the mean square error between the first training feature vector and the second training feature vector, and the first correspondence.
[0143] In one feasible implementation, the training module 40 is further configured to calculate the chamfer distance between the three-dimensional point cloud data and the reconstructed point cloud data;
[0144] Obtain the second correspondence between chamfer distance, refined point cloud data, and refinement loss;
[0145] The refinement loss is obtained based on the chamfer distance between the 3D point cloud data and the reconstructed point cloud data, the refined point cloud data, and the second correspondence.
[0146] The 3D reconstruction apparatus provided in this application, employing the 3D reconstruction method described in the above embodiments, can solve the technical problem of computational complexity and reduced reconstruction efficiency when using point clouds for 3D reconstruction. Compared with the prior art, the beneficial effects of the 3D reconstruction apparatus provided in this application are the same as those of the 3D reconstruction method provided in the above embodiments, and other technical features of the 3D reconstruction apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.
[0147] This application provides a three-dimensional reconstruction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the three-dimensional reconstruction method in the first embodiment described above.
[0148] The following is for reference. Figure 9 The diagram illustrates a structural schematic of a three-dimensional reconstruction device suitable for implementing embodiments of this application. The three-dimensional reconstruction device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 9 The 3D reconstruction device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0149] like Figure 9As shown, the 3D reconstruction device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the 3D reconstruction device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the 3D reconstruction equipment to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows 3D reconstruction equipment with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.
[0150] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.
[0151] The 3D reconstruction device provided in this application, employing the 3D reconstruction method described in the above embodiments, can solve the technical problem of computational complexity and reduced reconstruction efficiency when using point clouds for 3D reconstruction. Compared with the prior art, the beneficial effects of the 3D reconstruction device provided in this application are the same as those of the 3D reconstruction method provided in the above embodiments, and other technical features of this 3D reconstruction device are the same as those disclosed in the previous embodiment method, and will not be repeated here.
[0152] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0153] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0154] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the three-dimensional reconstruction method in the above embodiments.
[0155] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0156] The aforementioned computer-readable storage medium may be included in the 3D reconstruction device; or it may exist independently and not be assembled into the 3D reconstruction device.
[0157] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the 3D reconstruction device, the 3D reconstruction device: inputs the surface point cloud data of the target object into a learning network for feature extraction to obtain a latent vector; inputs the latent vector and the overall image of the target object into a reconstruction network for data reconstruction to obtain reconstructed point cloud data; inputs the reconstructed point cloud data into a thinning network for thinning processing to obtain thinned point cloud data; and determines the 3D reconstruction result of the target object based on the thinned point cloud data.
[0158] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0159] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0160] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.
[0161] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described 3D reconstruction method. This solves the technical problem of computational complexity and reduced reconstruction efficiency when using point clouds for 3D reconstruction. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the 3D reconstruction method provided in the above embodiments, and will not be elaborated upon here.
[0162] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the three-dimensional reconstruction method described above.
[0163] The computer program product provided in this application can solve the technical problem that the computation is complex and affects the reconstruction efficiency when using point clouds for 3D reconstruction. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the 3D reconstruction method provided in the above embodiments, and will not be repeated here.
[0164] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.
Claims
1. A three-dimensional reconstruction method, characterized in that, The method includes: The surface point cloud data of the target object is input into the learning network for feature extraction to obtain the latent vector; The latent vector and the overall image of the target object are input into a reconstruction network to reconstruct data, thereby obtaining reconstructed point cloud data. The reconstructed point cloud data is input into a thinning network for thinning processing to obtain thinned point cloud data; Based on the refined point cloud data, the 3D reconstruction result of the target object is determined.
2. The method as described in claim 1, characterized in that, The learning network includes a point encoder, a first fully connected unit, a first quantization layer, and a first folded decoder connected in sequence. The step of inputting the surface point cloud data of the target object into the learning network for feature extraction to obtain a latent vector includes: The surface point cloud data of the target object is input into the point encoder for global feature extraction to obtain a global feature vector; The global feature vector is input into the first fully connected unit for nonlinear transformation to obtain the transformed vector; The transformed vector is input into the first quantization layer for quantization to obtain the latent vector; The latent vector is input into the first folding decoder for decoding to obtain three-dimensional point cloud data.
3. The method as described in claim 1, characterized in that, The reconstruction network includes an image encoder, an average pooling layer, a second fully connected unit, a second quantization layer, and a second folded decoder connected in sequence. The step of inputting the latent vector and the overall image of the target object into the reconstruction network for data reconstruction to obtain reconstructed point cloud data includes: The overall image of the target object is input into the image encoder for feature extraction to obtain the image feature vector; The image feature vector is input into the average pooling layer for processing to obtain the target feature vector; The target feature vector and the latent vector are input into the second fully connected unit. Based on the second fully connected unit, the target feature vector is mapped onto the latent vector to obtain the mapped latent vector. The mapped latent vector is input into the second quantization layer for quantization to obtain the target latent vector; The target latent vector is input into the second folding decoder for decoding to obtain the reconstructed point cloud data.
4. The method as described in claim 1, characterized in that, The step of determining the 3D reconstruction result of the target object based on the refined point cloud data includes: Obtain the 3D point cloud data corresponding to the decoded potential vector; Based on the potential distance between the 3D point cloud data and the reconstructed point cloud data, the learning reconstruction loss is determined; Based on the refined point cloud data and the chamfer distance between the 3D point cloud data and the reconstructed point cloud data, the refinement loss is determined; The reconstruction network and the learning network are trained based on the learning reconstruction loss, and the refinement network is trained based on the refinement loss; Upon completion of training, the output of the refined network is used as the 3D reconstruction result of the target object.
5. The method as described in claim 4, characterized in that, The step of determining the learning reconstruction loss based on the potential distance between the 3D point cloud data and the reconstructed point cloud data includes: Calculate the potential distance between the 3D point cloud data and the reconstructed point cloud data; Feature extraction is performed on the three-dimensional point cloud data and the reconstructed point cloud data respectively to obtain a first training feature vector and a second training feature vector, and the mean square error between the first training feature vector and the second training feature vector is calculated. Obtain the first correspondence between the potential distance, mean squared error, and learning reconstruction loss; The learning reconstruction loss is obtained based on the potential distance between the 3D point cloud data and the reconstructed point cloud data, the mean square error between the first training feature vector and the second training feature vector, and the first correspondence.
6. The method as described in claim 4, characterized in that, The step of determining the refinement loss based on the refined point cloud data and the chamfer distance between the 3D point cloud data and the reconstructed point cloud data includes: Calculate the chamfer distance between the 3D point cloud data and the reconstructed point cloud data; Obtain the second correspondence between chamfer distance, refined point cloud data, and refinement loss; The refinement loss is obtained based on the chamfer distance between the 3D point cloud data and the reconstructed point cloud data, the refined point cloud data, and the second correspondence.
7. A three-dimensional reconstruction device, characterized in that, The device includes: The learning module is used to input the surface point cloud data of the target object into the learning network for feature extraction to obtain the latent vector; The reconstruction module is used to input the latent vector and the overall image of the target object into the reconstruction network for data reconstruction to obtain reconstructed point cloud data; The refinement module is used to input the reconstructed point cloud data into the refinement network for refinement processing to obtain refined point cloud data. The training module is used to determine the 3D reconstruction result of the target object based on the refined point cloud data.
8. A three-dimensional reconstruction device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the three-dimensional reconstruction method as described in any one of claims 1 to 6.
9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the three-dimensional reconstruction method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the three-dimensional reconstruction method as described in any one of claims 1 to 6.