Three-dimensional reconstruction and reproduction system and method for oblique photography

Abstract

The invention relates to a three-dimensional reconstruction and reproduction system and method for oblique photography, and the system comprises a data collection module which is used for collecting image data of different-angle oblique scenes; the data preprocessing module is used for preprocessing the image data to obtain a preprocessed image; the aerial triangulation processing module is used for acquiring the position and posture of the preprocessed image in the three-dimensional space, and optimizing and aligning the preprocessed image based on the position and posture of the preprocessed image in the three-dimensional space; the point cloud generation module is used for calculating and optimizing depth information of the aligned image, generating an initial point cloud, optimizing the initial point cloud and generating a dense point cloud; the three-dimensional model reconstruction module is used for optimizing the dense point cloud, converting the optimized dense point cloud into a continuous three-dimensional surface, generating a triangular mesh model, extracting textures of the image data, mapping the textures to the triangular mesh model, and reconstructing a three-dimensional model; and the scene reproduction module is used for performing scene reproduction by using the reconstructed three-dimensional model.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and three-dimensional modeling, and in particular to a three-dimensional reconstruction and reproduction system and method thereof for oblique photography. Background Art

[0002] Oblique photogrammetry is an advanced remote sensing technology that uses a camera system mounted on an aircraft, drone or other flying platform to photograph the ground or buildings from different angles to obtain rich geometric and texture information. These image data can be used to generate 3D models with a high level of detail, which has important application value in urban planning, architectural visualization, archaeological research and environmental monitoring.

[0003] However, despite the significant advantages of oblique photogrammetry, existing 3D reconstruction systems still face a series of challenges when processing these data:

[0004] 1. Large amount of data: The amount of data generated by oblique photogrammetry technology is usually very large because the scene needs to be captured from multiple angles, and each angle may contain thousands of high-resolution images.

[0005] 2. High computational complexity: Extracting feature points, matching, and reconstructing 3D models from these large numbers of images requires complex computational processes, which places high demands on computing resources.

[0006] 3. Insufficient reconstruction accuracy: When dealing with complex scenes or in the presence of occlusion, reflection, and lighting changes, existing technologies may find it difficult to accurately reconstruct the 3D model, resulting in errors or distortion in the reconstruction results.

[0007] 4. Low degree of automation: The existing 3D reconstruction process often requires a lot of manual intervention, such as image screening, feature point marking, etc., which not only increases the workload but also limits the processing speed.

[0008] 5. Poor real-time performance: Due to the high computational complexity and low degree of automation, it is difficult for existing 3D reconstruction systems to achieve real-time or near real-time data processing and model generation. Summary of the invention

[0009] In order to solve the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a three-dimensional reconstruction and reproduction system and method for oblique photography, aiming to solve the above-mentioned problems existing in the prior art. By adopting advanced image processing algorithms, optimized calculation processes and automated data processing technologies, the system can efficiently and accurately reconstruct three-dimensional models from oblique photography data and achieve high-quality three-dimensional scene reproduction. In addition, the system also provides a user-friendly interactive interface, allowing users to easily interact with the three-dimensional scene, thereby significantly improving the efficiency of three-dimensional reconstruction and user experience.

[0010] To achieve the above object, the present invention provides the following solutions:

[0011] The 3D reconstruction and reproduction system for oblique photography includes:

[0012] A data acquisition module, used to collect image data of scenes tilted at different angles;

[0013] A data preprocessing module, used to preprocess the image data to obtain a preprocessed image;

[0014] an aerial triangulation processing module, used for obtaining the position and posture of the pre-processed image in the three-dimensional space, and optimizing and aligning the pre-processed image based on the position and posture of the pre-processed image in the three-dimensional space;

[0015] A point cloud generation module, used to calculate the depth information of the optimized aligned image, generate an initial point cloud, and optimize the initial point cloud to generate a dense point cloud;

[0016] A three-dimensional model reconstruction module is used to optimize the dense point cloud, convert the optimized dense point cloud into a continuous three-dimensional surface, generate a triangular mesh model, extract the texture of the image data and map it to the triangular mesh model, and reconstruct the three-dimensional model;

[0017] The scene reconstruction module is used to reproduce the scene using the reconstructed three-dimensional model.

[0018] Optionally, the data acquisition module includes:

[0019] A data acquisition unit, used to acquire high-resolution images of scenes tilted at different angles using calibrated sensors and oblique photography equipment;

[0020] A storage unit is used to store the high-resolution image.

[0021] Optionally, the data preprocessing module includes:

[0022] A screening unit, used to screen the image data to remove blurred, incomplete or abnormal images;

[0023] A denoising unit, used for performing denoising processing on the filtered image data;

[0024] The calibration unit is used to perform image correction on the denoised image data, register the corrected image, and compare the image data taken at multiple angles.

[0025] Optionally, the aerial triangulation processing module includes:

[0026] The aerial triangulation processing unit is used to obtain initial geometric information and positioning data in the pre-processed image, determine the position and posture of the pre-processed image in three-dimensional space based on the initial geometric information and positioning data, and import image control points to optimize the alignment of the pre-processed image.

[0027] Optionally, the point cloud generation module includes:

[0028] A feature point recognition unit is used to perform image matching on the optimized aligned images and recognize the same feature points in different images;

[0029] The point cloud generation unit is used to calculate the depth information of the optimized aligned image according to the same feature points, generate an initial point cloud, and optimize the initial point cloud to generate a dense point cloud.

[0030] Optionally, the three-dimensional model reconstruction module includes:

[0031] A point cloud optimization unit, configured to perform statistical filtering and voxel filtering on the dense point cloud;

[0032] The three-dimensional model reconstruction unit is used to convert the optimized dense point cloud into a continuous three-dimensional surface, generate a triangular mesh model, extract the texture of the image data and map it to the triangular mesh model to reconstruct the three-dimensional model.

[0033] Optionally, the scene reproduction module includes:

[0034] A model processing unit, used for simplifying, smoothing and denoising the three-dimensional model, and rendering it;

[0035] The format conversion unit is used to convert the format of the rendered three-dimensional model.

[0036] Optionally, the scene reproduction module further includes:

[0037] The user interaction unit is used to select the 3D model after format conversion for rotation, scaling and measurement operations, and display the updated 3D model according to the operation instructions.

[0038] To achieve the above object, the present invention also provides a three-dimensional reconstruction and reproduction method for oblique photography, comprising:

[0039] Collecting image data of scenes tilted at different angles, preprocessing the image data, and obtaining preprocessed images;

[0040] Acquire the position and posture of the preprocessed image in the three-dimensional space, and optimize and align the preprocessed image based on the position and posture of the preprocessed image in the three-dimensional space;

[0041] Calculating depth information of the optimized aligned image to generate an initial point cloud, and optimizing the initial point cloud to generate a dense point cloud;

[0042] The dense point cloud is optimized, the optimized dense point cloud is converted into a continuous three-dimensional surface, a triangular mesh model is generated, the texture of the image data is extracted and mapped to the triangular mesh model, the three-dimensional model is reconstructed, and the scene is reproduced using the reconstructed three-dimensional model.

[0043] The beneficial effects of the present invention are:

[0044] The present invention uses an unmanned camera system to capture high-resolution image data of the ground or buildings from different angles, thereby improving the speed and accuracy of three-dimensional reconstruction.

[0045] The present invention enhances the robustness of the system through image preprocessing technology, so that it can work stably in complex scenes.

[0046] The present invention is based on an image processing algorithm to improve the efficiency and accuracy of data processing.

[0047] The present invention generates dense point clouds through point cloud generation technology, namely, a multi-view stereo matching algorithm, thereby improving the accuracy of three-dimensional reconstruction.

[0048] The present invention provides a user-friendly interactive interface, and the interactive user interface allows the user to rotate and zoom the three-dimensional reconstruction, thereby enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 A schematic diagram of a three-dimensional reconstruction and reproduction system for oblique photography according to an embodiment of the present invention;

[0051] Figure 2 Schematic diagram of a real-time interaction module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] like Figure 1 As shown, the present embodiment discloses a 3D reconstruction and reproduction system for oblique photography, including: a data acquisition module, used to acquire image data of scenes inclined at different angles; a data preprocessing module, used to preprocess the image data and obtain a preprocessed image; an aerial triangulation module, used to obtain the position and posture of the preprocessed image in the three-dimensional space, and optimize the alignment of the preprocessed image based on the position and posture of the preprocessed image in the three-dimensional space; a point cloud generation module, used to calculate the depth information of the image after optimization and alignment, generate an initial point cloud, optimize the initial point cloud, and generate a dense point cloud; a 3D model reconstruction module, used to optimize the dense point cloud, convert the optimized dense point cloud into a continuous three-dimensional surface, generate a triangular mesh model, extract the texture of the image data and map it to the triangular mesh model, and reconstruct the three-dimensional model; a scene reproduction module, used to reproduce the scene using the reconstructed three-dimensional model.

[0055] In this embodiment, the speed and accuracy of 3D reconstruction enhance the robustness of the system, enabling it to work stably in complex scenes, and provides a user-friendly interactive interface, thereby enhancing the user experience.

[0056] Furthermore, the data acquisition module includes: a calibration unit for setting up the oblique photography equipment and calibrating the sensor; a data acquisition unit for acquiring high-resolution images of scenes inclined at different angles using the calibrated sensor and the oblique photography equipment.

[0057] Specifically, the data acquisition module:

[0058] (1) Set up oblique photography equipment to ensure that the scene is captured from different angles.

[0059] (2) Sensor calibration, ensuring that the intrinsic and extrinsic parameters of the sensor are known and stable to improve the accuracy of subsequent reconstruction.

[0060] (3) Obtain high-resolution image data.

[0061] (4) Store the image data for subsequent processing.

[0062] Furthermore, the data preprocessing module includes: a screening unit for screening the image data to remove blurred, incomplete or abnormal images; a denoising unit for denoising the screened image data; a calibration unit for performing image correction on the denoised image data, registering the corrected image, and comparing image data taken from multiple angles.

[0063] Specifically, the preprocessing module:

[0064] (1) Read the stored image data.

[0065] (2) Screen the collected image data to remove blurry, incomplete or abnormal images.

[0066] (3) Perform denoising on the image to reduce noise interference.

[0067] (4) Perform image correction to ensure the consistency of all influencing coordinate systems and reduce distortion.

[0068] (5) Perform image registration to align images taken at multiple angles.

[0069] After screening, denoising, image correction and registration of images taken from multiple angles, the quality and usability of the images are improved, laying a good foundation for the subsequent point cloud extraction.

[0070] Furthermore, the aerial triangulation processing module includes: an aerial triangulation processing unit, which is used to obtain initial geometric information and positioning data in the preprocessed image, determine the position and posture of the preprocessed image in the three-dimensional space based on the initial geometric information and positioning data, and import image control points to optimize the alignment of the preprocessed image.

[0071] Specifically, the aerial triangulation (relative orientation and aerial triangulation) processing module:

[0072] (1) Image information import, providing initial geometric information and positioning data.

[0073] (2) Image alignment: determine the position and posture of the image in three-dimensional space.

[0074] (3) Image control point import and editing. Image control points are points with known positions in the actual scene. They are marked in the image and used to correct and optimize geometric errors in the photogrammetry process. Image control points are usually obtained by field measurement or determined by other high-precision measurement methods. They act as control points in photogrammetry to help improve the precision and accuracy of reconstruction. Image control points are used to improve the accuracy of reconstruction and correct geometric errors.

[0075] (4) Optimize image alignment and minimize reprojection error.

[0076] Furthermore, the point cloud generation module includes: a point cloud generation unit, which is used to perform image matching on the optimized aligned images, identify the same feature points in different images, calculate the depth information of the optimized aligned images based on the same feature points, generate an initial point cloud, optimize the initial point cloud, and generate a dense point cloud.

[0077] Specifically, point cloud generation module:

[0078] (1) Read the processed image data.

[0079] (2) Perform image matching to identify and match the same feature points in different images. Feature points refer to points at the same physical location in different images that are identified by image matching algorithms. These points have the same geometric features in images from different perspectives, such as corners and edges. Feature points are used for preliminary image matching and point cloud generation.

[0080] Acquiring feature points includes: feature detection, feature description, feature matching and geometric verification.

[0081] (3) Calculate depth information and generate the initial point cloud.

[0082] (4) Optimize the point cloud and generate a dense point cloud through a multi-view stereo matching algorithm.

[0083] (5) Store the optimized point cloud.

[0084] Obtaining feature points includes:

[0085] Furthermore, the three-dimensional model reconstruction module includes: a point cloud optimization unit, which is used to perform statistical filtering and voxel filtering on the dense point cloud; a three-dimensional model reconstruction unit, which is used to convert the optimized dense point cloud into a continuous three-dimensional surface, generate a triangular mesh model, extract the texture of the image data and map it to the triangular mesh model, and reconstruct the three-dimensional model.

[0086] Specifically, the 3D model reconstruction module:

[0087] (1) Read the optimized point cloud.

[0088] (2) Use point cloud data processing techniques, such as statistical filtering and voxel filtering, to further optimize the point cloud.

[0089] (3) Meshing: converting dense point cloud data into a continuous three-dimensional surface and generating a triangular mesh model to represent the geometric shape of the scene.

[0090] (4) Texture mapping, mapping the texture information of the original image to the surface of the 3D model, making the model look more realistic and detailed. Texture mapping technology involves mapping a 2D image (called texture) to the surface of a 3D object to enhance the details and realism of the object surface. The principle is to map each pixel in the texture image to a corresponding position on the surface of the object, and to achieve the fusion of the texture image and the surface of the object by interpolating the pixel points on the surface of the object.

[0091] (5) Reconstructing 3D models. 3D reconstruction focuses on recovering the geometric representation of a scene from an image, including recovering the camera poses of each viewpoint and the sparse structure of the scene from photos taken from multiple viewpoints. The 3D model reconstruction usually results in a white model that does not contain color information, while texture mapping applies texture information to this white model, making the model have color and texture details, making it more realistic.

[0092] The generation of 3D models usually uses 3D reconstruction algorithms, such as triangulation, voxel-based methods, or deep learning-based methods.

[0093] (6) The reconstructed three-dimensional model is stored.

[0094] Furthermore, the scene reproduction module includes: a model processing unit for simplifying, smoothing, denoising, and rendering the three-dimensional model; and a format conversion unit for converting the format of the rendered three-dimensional model.

[0095] Specifically, the scene reproduction module:

[0096] (1) Read the reconstructed 3D model.

[0097] (2) Simplify, smooth and denoise the generated 3D model to reduce the number of faces and improve the rendering speed and efficiency of the model.

[0098] (3) Store the rendering results and output the 3D model to a commonly used format (such as OBJ, 3DS, FBX, etc.) for subsequent use in different application platforms.

[0099] Furthermore, the scene reproduction module also includes: a user interaction unit, which is used to select the three-dimensional model after the format conversion for rotation, scaling and measurement operations, and display the updated three-dimensional model according to the operation instructions.

[0100] Specifically, the user interaction unit:

[0101] (1) Start the user interface and provide an interactive operating environment, such as Figure 2 shown.

[0102] (2) The user selects a 3D scene.

[0103] (3) Users perform operations such as rotation, scaling, and measurement.

[0104] (4) Update the 3D scene display based on user operations.

[0105] Step 8, End:

[0106] After the user completes all interactive operations, the process ends.

[0107] This embodiment also discloses a three-dimensional reconstruction and reproduction method for oblique photography, including: collecting image data of scenes inclined at different angles, preprocessing the image data, and obtaining a preprocessed image; obtaining the position and posture of the preprocessed image in three-dimensional space, and optimizing and aligning the preprocessed image based on the position and posture of the preprocessed image in three-dimensional space; calculating the depth information of the optimized aligned image, generating an initial point cloud, optimizing the initial point cloud, and generating a dense point cloud; optimizing the dense point cloud, converting the optimized dense point cloud into a continuous three-dimensional surface, generating a triangular mesh model, extracting the texture of the image data and mapping it to the triangular mesh model, reconstructing the three-dimensional model, and using the reconstructed three-dimensional model to reproduce the scene.

[0108] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A three-dimensional reconstruction and reproduction system for oblique photography, characterized in that: include: A data acquisition module, used to collect image data of scenes tilted at different angles; A data preprocessing module, used to preprocess the image data to obtain a preprocessed image; an aerial triangulation processing module, used for obtaining the position and posture of the pre-processed image in the three-dimensional space, and optimizing and aligning the pre-processed image based on the position and posture of the pre-processed image in the three-dimensional space; A point cloud generation module, used to calculate the depth information of the optimized aligned image, generate an initial point cloud, and optimize the initial point cloud to generate a dense point cloud; A three-dimensional model reconstruction module is used to optimize the dense point cloud, convert the optimized dense point cloud into a continuous three-dimensional surface, generate a triangular mesh model, extract the texture of the image data and map it to the triangular mesh model, and reconstruct the three-dimensional model; The scene reconstruction module is used to reproduce the scene using the reconstructed three-dimensional model.

2. The three-dimensional reconstruction and reproduction system for oblique photography according to claim 1, characterized in that: The data acquisition module comprises: A data acquisition unit, used to acquire high-resolution images of scenes tilted at different angles using calibrated sensors and oblique photography equipment; A storage unit is used to store the high-resolution image.

3. The three-dimensional reconstruction and reproduction system for oblique photography according to claim 1, characterized in that: The data preprocessing module comprises: A screening unit, used to screen the image data to remove blurred, incomplete or abnormal images; A denoising unit, used for performing denoising processing on the filtered image data; The calibration unit is used to perform image correction on the denoised image data, register the corrected image, and compare the image data taken at multiple angles.

4. The three-dimensional reconstruction and reproduction system for oblique photography according to claim 1, characterized in that: The aerial triangulation processing module includes: The aerial triangulation processing unit is used to obtain initial geometric information and positioning data in the pre-processed image, determine the position and posture of the pre-processed image in three-dimensional space based on the initial geometric information and positioning data, and import image control points to optimize the alignment of the pre-processed image.

5. The three-dimensional reconstruction and reproduction system for oblique photography according to claim 4, characterized in that: The point cloud generation module comprises: A feature point recognition unit is used to perform image matching on the optimized aligned images and recognize the same feature points in different images; The point cloud generation unit is used to calculate the depth information of the optimized aligned image according to the same feature points, generate an initial point cloud, and optimize the initial point cloud to generate a dense point cloud.

6. The three-dimensional reconstruction and reproduction system for oblique photography according to claim 1, characterized in that: The three-dimensional model reconstruction module comprises: A point cloud optimization unit, configured to perform statistical filtering and voxel filtering on the dense point cloud; The three-dimensional model reconstruction unit is used to convert the optimized dense point cloud into a continuous three-dimensional surface, generate a triangular mesh model, extract the texture of the image data and map it to the triangular mesh model to reconstruct the three-dimensional model.

7. The three-dimensional reconstruction and reproduction system for oblique photography according to claim 1, characterized in that: The scene reproduction module comprises: A model processing unit, used for simplifying, smoothing and denoising the three-dimensional model, and rendering it; The format conversion unit is used to convert the format of the rendered three-dimensional model.

8. The three-dimensional reconstruction and reproduction system for oblique photography according to claim 7, characterized in that: The scene reproduction module also includes: The user interaction unit is used to select the 3D model after format conversion for rotation, scaling and measurement operations, and display the updated 3D model according to the operation instructions.

9. A three-dimensional reconstruction and reproduction method for oblique photography, using the three-dimensional reconstruction and reproduction system according to any one of claims 1 to 8 to perform three-dimensional reconstruction and reproduction, characterized in that: include: Collecting image data of scenes tilted at different angles, preprocessing the image data, and obtaining preprocessed images; Acquire the position and posture of the preprocessed image in the three-dimensional space, and optimize and align the preprocessed image based on the position and posture of the preprocessed image in the three-dimensional space; Calculating depth information of the optimized aligned image to generate an initial point cloud, and optimizing the initial point cloud to generate a dense point cloud; The dense point cloud is optimized, the optimized dense point cloud is converted into a continuous three-dimensional surface, a triangular mesh model is generated, the texture of the image data is extracted and mapped to the triangular mesh model, the three-dimensional model is reconstructed, and the scene is reproduced using the reconstructed three-dimensional model.

Images