Rapid three-dimensional reconstruction system based on deep learning
By adopting a five-layer modular architecture based on deep learning and the DUSt3R model, the problems of high cost, poor real-time performance and insufficient accuracy of 3D reconstruction equipment are solved, realizing low-cost, low-threshold and fast 3D reconstruction, and supporting efficient reconstruction in a variety of scenarios.
Patent Information
- Application Number
- CN202512049382.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-31
- Publication Date
- 2026-04-10
AI Technical Summary
Existing 3D reconstruction technologies and equipment are costly, lack real-time performance, are inefficient, and have insufficient accuracy, especially passive reconstruction methods, which are inefficient and lack accuracy.
It adopts a five-layer modular architecture based on deep learning, including a data layer, an algorithm layer, a business logic layer, a representation layer, and a communication layer. It uses the DUSt3R model for feature extraction, depth estimation, point cloud generation, and mesh reconstruction, and combines RANSAC and ICP algorithms for point cloud processing. It uses ordinary cameras and computers for reconstruction.
It reduces equipment costs to 1/100-1/10 of LiDAR, has a low operating threshold, supports reconstruction in multiple scenarios, provides real-time visualization of multimodal intermediate results, shortens reconstruction time by 40%, and has a wide range of applications.
Smart Images

Figure CN121837550A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of 3D reconstruction technology, and in particular to a fast 3D reconstruction system based on deep learning. Background Technology
[0002] 3D reconstruction technology is mainly divided into "active" and "passive" methods. Active reconstruction schemes measure the distance to the target by emitting laser beams, generate high-density point clouds, and then convert them into 3D models. Representative products include TrimbleTX8 and LeicaScanStation. However, these devices are expensive, have poor portability, cannot enter narrow spaces, and have a large data processing volume. The processing time for a single scene (such as a 1,000-square-meter coal bunker) exceeds 4 hours, and the real-time performance is poor. Passive reconstruction schemes can be achieved by using ordinary cameras. However, existing passive methods (such as MVS based on geometric calculations) suffer from low efficiency and insufficient accuracy. Summary of the Invention
[0003] The purpose of this invention is to provide a fast 3D reconstruction system based on deep learning, which solves the problems of high equipment cost, poor real-time performance, low efficiency and insufficient accuracy in existing technologies.
[0004] To achieve the above objectives, this invention provides a rapid 3D reconstruction system based on deep learning, employing a five-layer modular architecture where each layer communicates through a standardized interface, including: The data layer module is used to input multi-view image data, configure image size, calculate device parameters, and read, write, and manage the output point cloud, mesh, and depth map. The algorithm layer module is used to perform calculations based on 3D reconstruction algorithms, which include depth estimation, point cloud generation, global alignment, and mesh reconstruction. The business logic layer module is used to coordinate the execution of the entire automated reconstruction process by various modules, and to build the entire process from multi-view image data input to model output; The presentation layer module provides an interactive interface for users, enabling them to interact with the system in real time. The communication layer module communicates with the data layer module, algorithm layer module, business logic layer module, and presentation layer module, and is used to complete data transmission and state synchronization between the modules.
[0005] In some embodiments of this application, the data layer module inputs multi-view image data in formats including JPG, PNG, and BMP, which can be imported in batches. EXIF information is read through the PIL library, and the image rotation direction is corrected using the code function _correct_orientation. The data layer module stores frequently used user parameters in JSON format, and enables fast reading and writing via the json.dump / json.load commands in the code.
[0006] In some embodiments of this application, the data layer module temporarily manages the output point cloud, mesh, and depth map, creates a temporary directory to store preprocessed images, avoids modification of the original images, and automatically cleans them up after reconstruction is completed.
[0007] In some embodiments of this application, the algorithm layer module performs feature extraction, depth estimation, confidence prediction, and model optimization sequentially based on the DUSt3R model; The DUSt3R model is a deep learning model based on the Transformer architecture, which excels at capturing the geometric relationships of multi-view images and generating high-precision depth maps.
[0008] The feature extraction includes: using a pre-trained ResNet-50 network to extract local features of the image and outputting a 512-dimensional feature map; The depth estimation includes: capturing long-distance dependencies across multiple views based on the Transformer decoder, and generating a depth map of the same size as the input image; The confidence prediction includes: outputting a confidence map with a grayscale value range of 0-255 in parallel, where white indicates that the depth estimation is reliable, and implementing this by calling the model using the inference function in the code; The model optimizations include: converting floating-point weights to integers using INT8 quantization to reduce memory usage; and using PyTorchJIT compilation to accelerate inference.
[0009] In some embodiments of this application, the multi-view stereo strategy of the algorithm layer module includes: The main strategy is to directly generate a depth map based on DUSt3R and then convert the depth map into an initial point cloud. The auxiliary strategy employs the RANSAC algorithm to remove outliers from the point cloud and the ICP algorithm to achieve global alignment of point clouds from multiple perspectives. This is implemented using the global_aligner function in the code and supports both the PointCloudOptimizer high-precision alignment mode and the PairViewer fast preview alignment mode. ICP (Iterative Closest Point) is a point cloud registration algorithm that achieves alignment of point clouds from multiple perspectives by iteratively minimizing the distance between two point clouds.
[0010] The algorithm layer module performs mesh reconstruction by: converting the optimized point cloud into a triangular mesh based on the Poisson reconstruction algorithm, improving lighting rendering through normal estimation, reducing the number of faces through mesh simplification, balancing details and file size, and using the code save_point_cloud_and_mesh function.
[0011] In some embodiments of this application, the business logic layer coordinates the various modules to perform fully automated reconstruction, including: The user imports at least two multi-view images, sets the multi-view image size to 224 / 512 pixels, and optimizes the number of iterations to 300-1000. The system then begins reconstruction. The reconstruction steps include: S1. Image preprocessing, which includes EXIF correction, resizing, and format conversion to RGB. S2, Model Inference, which includes loading the DUSt3R model and generating a depth map or confidence map; S3, point cloud processing, which includes depth map to point cloud, RANSAC denoising, and ICP global alignment. S4. Mesh reconstruction, which includes Poisson reconstruction, mesh simplification, and normal estimation. S5. Output the results, save the point cloud or mesh as PLY format, and save the depth map or confidence map as PNG format.
[0012] RANSAC is a robust estimation algorithm used to remove outliers (such as noise and mismatched points) from point clouds.
[0013] In some embodiments of this application, the business logic layer module implements background parallel processing through ProcessingThread code.
[0014] In some embodiments of this application, the interactive interface of the presentation layer module includes: The control area is located on the left side of the interactive interface, and the interactive options include model settings, global alignment settings, and operation buttons. The status area, located at the top of the interactive interface, includes interactive options such as displaying the current progress and system status. The results preview area is located in the center of the interactive interface and can be switched between displaying the original image, depth map, confidence map, and 3D model through interactive options. The log area, located at the bottom of the interactive interface, is used to output processing information in real time.
[0015] In some embodiments of this application, when the communication layer module performs data transfer and state synchronization between modules, it transmits data between modules based on NumPy arrays and JSON configuration parameters.
[0016] The advantages and beneficial effects of this invention compared to the prior art are: 1. No special hardware is required; reconstruction can be completed using a regular camera and computer. The equipment cost is only 1 / 100 to 1 / 10 of that of LiDAR, making it suitable for small and medium-sized enterprises and scientific research scenarios.
[0017] 2. The visual GUI replaces the command line, and users do not need professional training. They can complete the reconstruction in just three steps (import image → set parameters → click start), which effectively reduces the operation threshold.
[0018] 3. Real-time visualization of multimodal intermediate results (depth map, confidence map) allows users to judge the reconstruction quality in advance without waiting for the final result, improving the efficiency of error troubleshooting.
[0019] 4. It supports both small indoor scenarios and industrial scenarios such as mines and coal bunkers. Compared with LiDAR and simplified SFM, the scope of application of this invention is wider.
[0020] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0021] Figure 1 This is a structural diagram of a fast 3D reconstruction system based on deep learning, as described in an embodiment of the present invention. Figure 2 This is a diagram illustrating the overall technical framework of the deep learning-based fast 3D reconstruction system according to an embodiment of the present invention. Figure 3 This is a schematic diagram of the DUSt3R model of the deep learning-based fast 3D reconstruction system according to an embodiment of the present invention. Figure 4 This is a schematic diagram of another DUSt3R model of the deep learning-based fast 3D reconstruction system according to an embodiment of the present invention. Figure 5 This is a flowchart illustrating the fully automated workflow of a deep learning-based rapid 3D reconstruction system according to an embodiment of the present invention. Figure 6 This is a diagram of the main software interface of the deep learning-based fast 3D reconstruction system according to an embodiment of the present invention. Detailed Implementation
[0022] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product is in use. They are used only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," and "connect" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0023] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0024] like Figure 1 As shown, this invention provides a rapid 3D reconstruction system based on deep learning, employing a five-layer modular architecture where each layer communicates through a standardized interface, including: The data layer module is used to input multi-view image data, configure image size, calculate device parameters, and read, write, and manage the output point cloud, mesh, and depth map. The algorithm layer module is used to perform calculations based on 3D reconstruction algorithms, which include depth estimation, point cloud generation, global alignment, and mesh reconstruction. The business logic layer module is used to coordinate the execution of the entire automated reconstruction process by various modules, and to build the entire process from multi-view image data input to model output; The presentation layer module provides an interactive interface for users, enabling them to interact with the system in real time. The communication layer module communicates with the data layer module, algorithm layer module, business logic layer module, and presentation layer module, and is used to complete data transmission and state synchronization between the modules.
[0025] The overall technical framework adopted in this invention is as follows: Figure 2 As shown.
[0026] This invention requires no specialized hardware; reconstruction can be completed using a regular camera and a computer. The equipment cost is only 1 / 100 to 1 / 10 of that of LiDAR, making it suitable for small and medium-sized enterprises and research scenarios. This invention uses a visual GUI instead of a command line, requiring no professional training from the user. Reconstruction can be completed in just three steps (import image - set parameters - click start), significantly lowering the operational threshold. The multimodal intermediate results (depth map, confidence map) are visualized in real time, allowing users to assess reconstruction quality in advance without waiting for the final result, thus improving error troubleshooting efficiency. This invention supports both small indoor scenes and industrial scenarios such as mines and coal bunkers, making it more widely applicable than LiDAR and simplified SFM.
[0027] In some embodiments of this application, the data layer module inputs multi-view image data in formats including JPG, PNG, and BMP, which can be imported in batches. EXIF information is read through the PIL library, and the image rotation direction is corrected using the code function _correct_orientation. The data layer module stores commonly used user parameters (such as default computing device and output path) in JSON format, and enables fast reading and writing through the json.dump / json.load commands in the code.
[0028] In some embodiments of this application, the data layer module temporarily manages the output point cloud, mesh, and depth map, creates a temporary directory (tempfile.mkdtemp) to store preprocessed images to avoid modification of the original images, and automatically cleans up (shutil.rmtree) after reconstruction is completed.
[0029] In some embodiments of this application, such as Figure 3 , Figure 4 As shown, the algorithm layer module performs feature extraction, depth estimation, confidence prediction, and model optimization sequentially based on the DUSt3R model; The feature extraction includes: using a pre-trained ResNet-50 network to extract local features of the image and outputting a 512-dimensional feature map; The depth estimation includes: capturing long-distance dependencies across multiple views based on the Transformer decoder (ViT-Large architecture) to generate a depth map of the same size as the input image; The confidence prediction includes: outputting a confidence map with a grayscale value range of 0-255 in parallel, where white indicates that the depth estimation is reliable, and implementing this by calling the model using the inference function in the code; The model optimizations include: converting floating-point weights to integers using INT8 quantization to reduce memory usage; and using PyTorchJIT compilation to accelerate inference.
[0030] In one specific implementation, floating-point weights are converted to integers using INT8 quantization, reducing memory usage by 75% and accelerating inference by 30% using PyTorchJIT compilation.
[0031] In some embodiments of this application, the multi-view stereo strategy of the algorithm layer module includes: The main strategy is to directly generate a depth map based on DUSt3R and then convert the depth map into an initial point cloud. The auxiliary strategy uses the RANSAC algorithm to remove outliers in the point cloud (automatically triggered when the outlier ratio is >5%), and the ICP (Iterative Closest Point) algorithm to achieve global alignment of the point cloud from multiple views. It is implemented using the global_aligner function in the code and supports PointCloudOptimizer high-precision alignment mode and PairViewer fast preview alignment mode. The algorithm layer module performs mesh reconstruction by: converting the optimized point cloud into a triangular mesh based on the Poisson reconstruction algorithm (Open3D library), improving lighting rendering through normal estimation, reducing the number of faces through mesh simplification, balancing details and file size, and using the code save_point_cloud_and_mesh function.
[0032] In some embodiments of this application, such as Figure 5 As shown, the business logic layer coordinates the various modules to perform a fully automated reconstruction process, including: The user imports at least two multi-view images, sets the multi-view image size to 224 / 512 pixels, and optimizes the number of iterations to 300-1000. The system then begins reconstruction. The reconstruction steps include: S1. Image preprocessing, which includes EXIF correction, resizing, and format conversion to RGB. S2, Model Inference, which includes loading the DUSt3R model and generating a depth map or confidence map; S3, point cloud processing, which includes depth map to point cloud, RANSAC denoising, and ICP global alignment. S4. Mesh reconstruction, which includes Poisson reconstruction, mesh simplification, and normal estimation. S5. Output the results, save the point cloud or mesh as PLY format, and save the depth map or confidence map as PNG format.
[0033] PLY format is a universal 3D model file format that supports storing point clouds, meshes, and other data, and is compatible with mainstream 3D software.
[0034] In some embodiments of this application, the business logic layer module implements background parallel processing through ProcessingThread (which inherits from QThread) code to avoid interface lag, and the run method in the code defines the complete process logic.
[0035] In some embodiments of this application, such as Figure 6 As shown, the interactive interface of the presentation layer module includes: The control area, located on the left side of the interactive interface, includes interactive options such as model settings (fixed model path checkpoints / DUSt3R_ViTLarge_BaseDecoder_512_dpt.pth, computing device selection), global alignment settings (alignment mode, number of iterations, learning rate), and operation buttons (load model, select image, start reconstruction). This area component is created using the setupUi method of the Ui_Form class in the code. The status area is located at the top of the interactive interface. Interactive options include displaying the current progress (progress bar) and system status (such as "Model loaded successfully" and "Reconstruction completed"). The results preview area is located in the center of the interactive interface. It can switch between displaying the original image, depth map (color-coded, with different colors representing different depth values), confidence map (grayscale display), and 3D model (supporting rotation / scaling / panning interaction) through interactive options. The preview logic is implemented in the display_depth_maps / display_confidence_maps functions in the code. The log area, located at the bottom of the interactive interface, is used to output processing information in real time. The update_process_info function in the code implements log updates.
[0036] In one specific embodiment, processing information is output in real time, such as "13,571 matching points were found between images 3 and 4".
[0037] In some embodiments of this application, when the communication layer module performs data transfer and state synchronization between modules, it transmits data between modules based on NumPy arrays and JSON configuration parameters.
[0038] In one specific embodiment, the technical implementation includes: a signal-slot mechanism (PyQt5): for example, clicking the "Start Reconstruction" button triggers the start_reconstruction function, and the update_progress signal of ProcessingThread updates the progress bar; standardized data format: data is transferred between modules using numpy arrays (point cloud / depth map) and JSON (configuration parameters) to ensure compatibility.
[0039] The deep integration technology of the DUSt3R deep learning model and PyQt5GUI in this invention achieves synchronization between model inference and interface interaction through the signal slot mechanism, solving the problem of "no visual interface and high operation threshold" in existing deep learning reconstruction solutions. Non-professional users can complete the reconstruction in three steps.
[0040] This invention eliminates the need for manual intervention in its five core steps from image preprocessing to 3D model output. Parallel processing is achieved through a background thread called ProcessingThread, solving the problems of "step-by-step operation and low efficiency" inherent in traditional tools. Reconstruction time is reduced by 40% compared to COLMAP. It also includes automatic CPU / GPU switching, dynamic image size adjustment, and INT8 quantization of the model, addressing the issues of "poor hardware compatibility and low resource utilization" in existing tools. It supports full-scenario adaptation from laptops (i5 + 8GB RAM) to workstations (i7 + 32GB RAM + RTX 3060).
[0041] This invention provides adjustable parameters (image size 224 / 512 pixels, optimization iterations 300-1000, learning rate 0.0001-0.1), allowing users to choose between "high-precision mode" (512 pixels + 1000 iterations, error <5%) or "fast mode" (224 pixels + 300 iterations, processing time <30 minutes) to address the difficulty of balancing accuracy and speed in existing tools. It supports color encoding of depth maps, grayscale display of confidence maps, and interactive operation of 3D models, and outputs in PLY / PNG standard formats, resolving the issues of poor result controllability and closed formats in existing tools, facilitating subsequent analysis and secondary development.
[0042] In this application, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. In case of any inconsistency, the meaning set forth in this specification or derived from the content described herein shall prevail. Furthermore, the terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the scope of this application.
[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.
Claims
1. A fast 3D reconstruction system based on deep learning, characterized in that, A five-layer modular architecture is applied, with each layer communicating through standardized interfaces, including: The data layer module is used to input multi-view image data, configure image size, calculate device parameters, and read, write, and manage the output point cloud, mesh, and depth map. The algorithm layer module is used to perform calculations based on 3D reconstruction algorithms, which include depth estimation, point cloud generation, global alignment, and mesh reconstruction. The business logic layer module is used to coordinate the execution of the entire automated reconstruction process by various modules, and to build the entire process from multi-view image data input to model output; The presentation layer module provides an interactive interface for users, enabling them to interact with the system in real time. The communication layer module communicates with the data layer module, algorithm layer module, business logic layer module, and presentation layer module, and is used to complete data transmission and state synchronization between the modules.
2. The rapid 3D reconstruction system based on deep learning according to claim 1, characterized in that, The data layer module inputs multi-view image data in formats including JPG, PNG, and BMP, which can be imported in batches. It reads EXIF information through the PIL library and corrects the image rotation direction using the code function _correct_orientation. The data layer module stores frequently used user parameters in JSON format, and enables fast reading and writing via the json.dump / json.load commands in the code.
3. The rapid 3D reconstruction system based on deep learning according to claim 2, characterized in that, The data layer module temporarily manages the output point cloud, mesh, and depth map, creates a temporary directory to store preprocessed images to avoid modification of the original images, and automatically cleans up the directory after reconstruction is completed.
4. The rapid 3D reconstruction system based on deep learning according to claim 3, characterized in that, The algorithm layer module performs feature extraction, depth estimation, confidence prediction, and model optimization sequentially based on the DUSt3R model. The feature extraction includes: using a pre-trained ResNet-50 network to extract local features of the image and outputting a 512-dimensional feature map; The depth estimation includes: capturing long-distance dependencies across multiple views based on the Transformer decoder, and generating a depth map of the same size as the input image; The confidence prediction includes: outputting a confidence map with a grayscale value range of 0-255 in parallel, where white indicates that the depth estimation is reliable, and implementing this by calling the model using the inference function in the code; The model optimizations include: converting floating-point weights to integers using INT8 quantization to reduce memory usage; and using PyTorchJIT compilation to accelerate inference.
5. The rapid 3D reconstruction system based on deep learning according to claim 4, characterized in that, The multi-view stereo strategy of the algorithm layer module includes: The main strategy is to directly generate a depth map based on DUSt3R and then convert the depth map into an initial point cloud. The auxiliary strategy uses the RANSAC algorithm to remove outliers in the point cloud and the ICP algorithm to achieve global alignment of the point cloud from multiple views. It is implemented using the global_aligner function in the code and supports the PointCloudOptimizer high-precision alignment mode and the PairViewer fast preview alignment mode. The algorithm layer module performs mesh reconstruction by: converting the optimized point cloud into a triangular mesh based on the Poisson reconstruction algorithm, improving lighting rendering through normal estimation, reducing the number of faces through mesh simplification, balancing details and file size, and using the code save_point_cloud_and_mesh function.
6. The rapid 3D reconstruction system based on deep learning according to claim 5, characterized in that, The business logic layer coordinates the various modules to perform fully automated reconstruction, including: The user imports at least two multi-view images, sets the multi-view image size to 224 / 512 pixels, and optimizes the number of iterations to 300-1000. The system then begins reconstruction. The reconstruction steps include: S1. Image preprocessing, which includes EXIF correction, resizing, and format conversion to RGB. S2, Model Inference, which includes loading the DUSt3R model and generating a depth map or confidence map; S3, point cloud processing, which includes depth map to point cloud, RANSAC denoising, and ICP global alignment. S4. Mesh reconstruction, which includes Poisson reconstruction, mesh simplification, and normal estimation. S5. Output the results, save the point cloud or mesh as PLY format, and save the depth map or confidence map as PNG format.
7. The rapid 3D reconstruction system based on deep learning according to claim 6, characterized in that, The business logic layer module implements background parallel processing through ProcessingThread code.
8. The rapid 3D reconstruction system based on deep learning according to claim 7, characterized in that, The interactive interface of the presentation layer module includes: The control area is located on the left side of the interactive interface, and the interactive options include model settings, global alignment settings, and operation buttons. The status area, located at the top of the interactive interface, includes interactive options such as displaying the current progress and system status. The results preview area is located in the center of the interactive interface and can be switched between displaying the original image, depth map, confidence map, and 3D model through interactive options. The log area, located at the bottom of the interactive interface, is used to output processing information in real time.
9. The rapid 3D reconstruction system based on deep learning according to claim 8, characterized in that, When the communication layer module performs data transfer and state synchronization between modules, it transmits data between modules based on NumPy arrays and JSON configuration parameters.
