Construction site three-dimensional scene reconstruction method based on unmanned aerial vehicle image and monitoring video

Abstract

The invention discloses a construction site three-dimensional scene reconstruction method based on an unmanned aerial vehicle image and a monitoring video. The method comprises the following steps: acquiring a construction site scene multi-view image; a sparse three-dimensional point cloud and a camera pose are generated through a feature matching and motion recovery structure algorithm, and dense reconstruction is carried out to obtain a global three-dimensional point cloud; aligning the global three-dimensional point cloud with a world coordinate system by using geographic position information; estimating the position of a shooting camera in the three-dimensional point cloud, sampling a candidate view angle and rendering a virtual RGB image; based on two-dimensional feature matching of the shot image and the virtual RGB image, internal parameters and external parameters of the shooting camera are iteratively solved through triangulation and a pose optimization algorithm; monocular depth estimation is carried out on the shot image, and the shot image is converted to a measurement scale through static region depth alignment; and projecting the depth of the shot image to the three-dimensional point cloud, updating the dynamic object in real time, and fusing the dynamic object into a complete three-dimensional scene model. The method has the advantage that the real-time three-dimensional reconstruction of the dynamic scene of the construction site is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and three-dimensional reconstruction, and particularly relates to a construction site three-dimensional scene reconstruction method based on unmanned aerial vehicle images and monitoring videos. BACKGROUND

[0002] In the construction site scene, accurate three-dimensional reconstruction is crucial for progress tracking, safety monitoring, and digital twin applications. In recent years, unmanned aerial vehicles have been widely used in civil engineering and construction industries due to their flexibility, efficiency, and ability to access dangerous or hard-to-reach areas. Traditional unmanned aerial vehicle-based three-dimensional reconstruction processes mainly rely on offline photogrammetry techniques, which perform well in capturing static geometric shapes. However, since unmanned aerial vehicle photogrammetry methods are inherently difficult to capture dynamic scene changes, they are not suitable for time-critical or continuously evolving construction site environments, such as construction sites under construction.

[0003] Meanwhile, recent advances in real-time three-dimensional perception mainly focus on sensors such as laser radars, stereo cameras, or RGB-D cameras. Although these devices can provide direct and accurate depth measurements, they are often expensive, require extensive infrastructure support, and are not practical for widespread deployment on large-scale construction sites.

[0004] On the other hand, monitoring cameras have been widely installed on construction sites, providing continuous and large-scale monitoring at a relatively low cost. However, there are still many challenges in using these cameras for accurate three-dimensional reconstruction. Monitoring cameras often lack depth sensors, are often installed without calibration, and have pan-tilt-zoom functionality, which can cause changes in viewing angles and unstable camera parameters. To compensate for the lack of depth information, learning-based monocular depth estimation methods have emerged in recent years, allowing depth to be inferred from a single RGB image. Despite remarkable progress, accurately predicting metric-scale depth in outdoor scenes without auxiliary cues such as camera poses or scene geometry remains a fundamental challenge. SUMMARY

[0005] The present application aims to address the shortcomings of the prior art by providing a construction site three-dimensional scene reconstruction method based on unmanned aerial vehicle images and monitoring videos. This method fuses unmanned aerial vehicle imagery with monitoring images to address the problem of traditional unmanned aerial vehicle photogrammetry being unable to capture dynamic changes in the construction site scene, while monitoring cameras lack depth perception and accurate calibration. This provides accurate three-dimensional reconstruction support for progress tracking, safety management, and digital twin applications in construction sites.

[0006] The present application achieves the above-mentioned purpose by the following technical solutions: The present application achieves the above-mentioned purpose by the following technical solutions:

[0007] A construction site three-dimensional scene reconstruction method based on unmanned aerial vehicle images and monitoring videos, the method comprising:

[0008] S1: acquiring multi-view images of a construction site scene;

[0009] S2: based on the images, generating a sparse three-dimensional point cloud and camera pose through feature matching and motion structure recovery algorithm, and performing dense reconstruction to obtain a global three-dimensional point cloud;

[0010] S3: aligning the global three-dimensional point cloud with a world coordinate system using geographic location information;

[0011] S4: estimating the shooting camera position in the three-dimensional point cloud, sampling candidate view angles, and rendering virtual RGB images;

[0012] S5: based on two-dimensional feature matching of the shooting images and virtual RGB images, iteratively solving the internal and external parameters of the shooting camera through triangulation and pose optimization algorithm;

[0013] S6: performing monocular depth estimation on the shooting images, and converting to metric scale through static area depth alignment;

[0014] S7: projecting the depth of the shooting images to the three-dimensional point cloud, updating dynamic objects in real time and fusing into a complete three-dimensional scene model.

[0015] In the step S1, multi-view images of a construction site scene are acquired by an unmanned aerial vehicle or a general camera; when the multi-view images of the construction site scene are acquired by the unmanned aerial vehicle, the step S1 comprises:

[0016] Acquiring multi-view images of a construction site scene by an unmanned aerial vehicle; performing quality screening on the images, and eliminating images with blur or abnormal exposure.

[0017] The step S2 comprises:

[0018] Using SIFT algorithm to extract image feature points and calculate feature descriptors; optimizing camera pose and three-dimensional point position through incremental SfM algorithm, generating a sparse three-dimensional point cloud; performing multi-view stereo matching based on the sparse point cloud, generating a dense three-dimensional point cloud.

[0019] The step S2 further comprises:

[0020] Performing Poisson surface reconstruction on the dense three-dimensional point cloud, generating a mesh model; mapping the unmanned aerial vehicle image texture to the mesh model to enhance visual realism.

[0021] In the step S3:

[0022] The geographic position information is acquired by a GPS / RTK device; and a rigid transformation matrix is estimated by a least square method to align the three-dimensional point cloud with a world coordinate system.

[0023] In the step S4, the shooting camera is a monitoring camera or a general camera; when the shooting camera is a monitoring camera, the step S4 comprises:

[0024] Candidate view angles are uniformly sampled on a hemisphere around the estimated position of the monitoring camera; a virtual pinhole camera is defined for each candidate view angle and a virtual RGB image is rendered.

[0025] The step S5 comprises:

[0026] Feature points of the shooting image and the virtual RGB image are extracted by using a SuperPoint algorithm; three-dimensional positions of the matching points are recovered by triangulation; and shooting camera parameters are iteratively optimized by combining a PnP algorithm and a bundle adjustment method.

[0027] The step S6 comprises:

[0028] Monocular depth estimation is performed by using a deep learning model; a static scene region is segmented; and a linear transformation relationship between monocular depth and rendered depth is fitted by using a RANSAC algorithm: In the formula, is a converted metric depth, D is a monocular depth estimation result, s is a scale factor, and t is an offset.

[0029] The dynamic object updating in the step S7 comprises:

[0030] According to the depth map and the camera parameters, a position of the dynamic object in a three-dimensional space is calculated in real time; and a dynamic object model is fused with a static scene point cloud.

[0031] The advantages of the application are as follows: by fusing the unmanned aerial vehicle image and the monitoring image, real-time three-dimensional reconstruction of a dynamic scene of a construction site is realized, which plays an important and multi-dimensional role in site management; compared with a traditional management mode relying on two-dimensional drawings, planar images and artificial inspection, a precise three-dimensional site model can be generated in real time, key information such as construction progress, equipment operation state and material stacking position is comprehensively displayed, a management personnel can intuitively and comprehensively master the site dynamic without going to the site, and the efficiency and accuracy of information acquisition are improved; meanwhile, the system has real-time monitoring and intelligent early warning functions, can timely alarm when an abnormality occurs, and prevent safety accidents; the system provides strong data support for reasonable allocation of construction resources and optimization of construction processes, significantly improves the scientific nature and efficiency of site management, provides a solid guarantee for smooth progress of a construction project, and helps the site to develop in a high-efficiency, safe and intelligent direction. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 Flow chart of the construction site three-dimensional scene reconstruction method based on UAV images and monitoring videos of the present application;

[0033] Figure 2 Comparison chart of the positioning image and the monitoring image under different weather conditions of the present application;

[0034] Figure 3 Comparison chart of the positioning image and the monitoring image after the scene changes of the present application;

[0035] Figure 4 Comparison chart of the positioning image, the positioning image by the traditional method and the monitoring image of the present application;

[0036] Figure 5 Process schematic diagram of step S5 of the present application;

[0037] Figure 6 Process schematic diagram of step S6 of the present application. DETAILED DESCRIPTION

[0038] The features and other related features of the present application are further described in detail below with reference to the accompanying drawings, so as to facilitate the understanding of the same by the skilled in the art:

[0039] Embodiment: As shown in the figure, the present embodiment relates to a construction site three-dimensional scene reconstruction method based on UAV images and monitoring videos, which mainly includes the following steps: Figure 1

[0040] S1: Obtain the entire construction site scene picture.

[0041] In the present embodiment, the UAV (or a general camera) is used to obtain the multi-view images of the construction site. Specifically, the UAV is used to carry a high-resolution camera to take pictures of the construction site in all directions, obtain multi-view image data, and preliminarily screen the collected image data to remove blurred, overexposed or underexposed images to ensure data quality.

[0042] S2: Based on the images, generate sparse three-dimensional point cloud and camera pose through feature matching and motion recovery structure algorithm, and perform dense reconstruction to obtain global three-dimensional point cloud.

[0043] In the present embodiment, step S2 specifically includes the following steps:

[0044] S2.1: According to the image feature matching algorithm and sparse reconstruction method, obtain globally consistent sparse three-dimensional point cloud and UAV camera pose.

[0045] ​In this embodiment, the COLMAP software is used to extract features from the screened unmanned aerial scene pictures. The SIFT algorithm is used to extract feature points in the image and calculate feature descriptors. Through the similarity measurement between feature descriptors, feature matching is performed to establish the correspondence between images. The incremental SfM (Structure from Motion) algorithm is used to start from the initial image pair, gradually add new images, and through iterative optimization of camera pose and three-dimensional point position, a globally consistent sparse three-dimensional point cloud is generated, and the accurate pose of the unmanned aerial vehicle camera is determined.

[0046] S2.2: Use the COLMAP toolbox to perform dense reconstruction on the sparse point cloud to obtain a comprehensive three-dimensional point cloud.

[0047] In this embodiment, based on the sparse reconstruction result, the PatchMatch-based algorithm in the MVS (Multi-View Stereo) module of COLMAP is used for dense reconstruction. This algorithm estimates the depth value of each pixel by matching at the pixel level between adjacent images, and generates a dense depth map. The dense depth maps of each view are fused into a comprehensive three-dimensional point cloud.

[0048] S2.3: Use filtering methods for post-processing to improve the quality of the three-dimensional point cloud.

[0049] In this embodiment, the filtering method is used for post-processing operation to remove noise points and outliers, and to improve the quality of the point cloud. The Poisson surface reconstruction algorithm is used to process the dense point cloud, converting the discrete three-dimensional point cloud into a continuous mesh model. This algorithm solves the Poisson equation to reconstruct a smooth surface based on the normal vector information of the point cloud. By projecting the unmanned aerial image onto the mesh surface, texture mapping is performed according to the color information of the pixels to generate high-resolution textures and improve the visual realism of the model.

[0050] S3: Use the geographic position information obtained by the GPS / PTK device to align the reconstructed three-dimensional model with the world coordinate system.

[0051] In this embodiment, the geographic position information of the image is obtained by using a GPS / RTK device during the flight of the unmanned aerial vehicle. At the same time, a certain number of ground control points are set on the construction site, and the precise coordinates thereof are obtained by using a high-precision measuring device (such as a total station). In combination with the GPS / RTK measurement data and the ground control point coordinates, a rigid transformation matrix is estimated by a least square method (which is an optimization method widely used in mathematics, engineering and computer vision, and is used to minimize the fitting error) optimization algorithm, the reconstructed three-dimensional model is aligned with the world coordinate system, and it is ensured that the model has a metric scale and meets the geodetic measurement alignment requirements.

[0052] S4: The estimated approximate positions of the shooting cameras in the three-dimensional model are uniformly sampled to obtain a certain number of candidate view angles, and RGB images are rendered.

[0053] In this embodiment, the shooting camera is a monitoring camera (or a general camera can also be used). Specifically, the rough estimation of the position of the monitoring camera is obtained by using the installation position information or prior knowledge of the monitoring camera. 100 candidate view angles are uniformly sampled on a hemisphere with the estimated position as the center and a radius of 2 meters. For each sampled view angle, a virtual pinhole camera is defined by using the preset camera internal parameters. By using the PyTorch3D library, the RGB images are rendered from the global three-dimensional grid generated in step S2.3 to generate a set of rendered images I r , each of which is associated with a known camera pose P r = [R | T], wherein R is a rotation matrix and T is a translation vector.

[0054] S5: The internal and external parameters of the shooting camera are iteratively optimized by using the SuperPoint algorithm and the triangulation method.

[0055] In this embodiment, the SuperPoint algorithm (a general algorithm for end-to-end feature point detection and descriptor extraction based on convolutional neural networks, commonly used for image matching, SLAM, SfM, etc.) is used to extract features from the monitoring image and the rendered image, respectively, to obtain feature points and their descriptors. By calculating the Hamming distance or Euclidean distance between the feature descriptors, a two-dimensional-two-dimensional correspondence relationship u between the monitoring image and the rendered image is established. Based on the known camera internal and external parameters of the rendered view, a linear triangulation method is used to recover the three-dimensional position of the matching points. For each correspondence observed in at least two views, a homogeneous linear system is constructed according to the camera projection matrix, and singular value decomposition (a basic tool in numerical calculation, commonly used to solve the least squares solution of underdetermined or overdetermined equations, especially for vision problems such as triangulation and PnP) is used to solve it, generating a sparse three-dimensional point cloud X aligned with the synthetic view. Using the sparse three-dimensional point cloud X obtained by triangulation and their two-dimensional correspondence relationship u between the monitoring image and the rendered image, the PnP algorithm (a general algorithm for predicting camera pose from a set of input three-dimensional points and their projected points in the image) is applied to obtain the initial estimate of the monitoring camera's external and internal parameters. The objective function of the bundle adjustment method (a general optimization algorithm that jointly optimizes the camera pose and the position of three-dimensional points to minimize the reprojection error) is constructed, and the Levenberg-Marquardt (a standard method for solving nonlinear least squares problems, combining the advantages of Gauss-Newton method and gradient descent method, commonly used for optimization of the objective function in bundle adjustment method) nonlinear optimization algorithm is used to solve the objective function. The internal and external parameters of the monitoring camera and the position of the three-dimensional points are iteratively updated until the objective function converges, obtaining the accurate internal and external parameters of the monitoring camera.

[0056] S6: Monocular depth estimation is performed on the captured image, and a static region depth alignment is converted to a metric scale.

[0057] In this embodiment, step S6 specifically includes the following steps:

[0058] S6.1: Depth estimation is performed on the depth image using a monocular depth estimation method, and aligned with the actual world scale.

[0059] In this embodiment, a monocular depth estimation network is used to predict the depth of the monitoring image. This network learns the mapping relationship from monocular image to depth map by training on a large-scale depth dataset, and outputs a monocular depth map D. A segmentation model SAM is used to perform semantic segmentation on the monitoring image, identifying static scene elements such as buildings, roads, etc. By applying a binary mask to the rendered depth map, the identified static regions are isolated. In these static regions, a random sample consensus (RANSAC) algorithm is used to robustly fit the relationship between monocular depth and rendered depth:

[0060]

[0061] wherein, is the converted metric depth, D is the monocular depth estimation result, s is the scale factor, and t is the offset. Through the transformation, the monocular depth is converted into the metric depth, thereby ensuring its consistency with the actual world scale while taking advantage of monocular depth prediction.

[0062] S6.2: Continuously collect monitoring images and perform monocular depth estimation and depth alignment according to step S6.1 to obtain a real-time updated metric scale depth map.

[0063] S7: Project the estimated depth of the photographed image scene into the three-dimensional map and real-time update the dynamically changing objects in the scene to obtain a complete dynamic scene three-dimensional model.

[0064] In this embodiment, the internal and external parameters of the monitoring camera obtained in step S5 are used to project the real-time collected monitoring images and their corresponding metric scale depth maps obtained in step S6.2 into the three-dimensional map. For dynamic scene elements, their positions and shapes in three-dimensional space are updated in real time according to their depth information and camera parameters, realizing real-time three-dimensional reconstruction of the dynamic scene, and the reconstruction result is fused with the static scene model to form a complete construction site dynamic scene three-dimensional model.

[0065] As shown in Figure 2 and Figure 3 shown, this embodiment not only shows good robustness in complex environments such as structural changes and weather changes, but also is suitable for long-term deployment in construction sites, meeting various application requirements such as progress tracking, safety management and digital twin. Through efficient integration and automatic processing of multi-source information, this embodiment significantly improves the timeliness, accuracy and intelligence level of three-dimensional reconstruction of construction sites, providing a solid technical support for intelligent construction.

[0066] As shown in Figure 4 compared to traditional methods that rely on a single data source, this embodiment combines large-scale, high-precision images obtained by drones with continuous perspective information provided by fixed monitoring cameras, significantly improving the perception ability and reconstruction efficiency of dynamic changes.

[0067] As shown in Figure 5 In this embodiment, by innovatively combining feature matching, triangulation and bundle adjustment method, the internal and external parameters of the monitoring camera are effectively optimized by using the sampled several perspective images, and the internal and external parameters of the monitoring camera are obtained, and based on the obtained parameters of the monitoring camera, the RGB image and the depth map under the corresponding real monitoring camera perspective in the three-dimensional map can be rendered.

[0068] As Figure 6 shown, in this embodiment, by innovatively using the RANSAC algorithm on the static area in the depth map, the robustness of scale and offset estimation can be effectively improved, and then the alignment accuracy is improved and adapted to changes in dynamic scenes.

[0069] The beneficial technical effects of the embodiment are: by fusing the unmanned aerial vehicle image and the monitoring image, real-time three-dimensional reconstruction of the construction site dynamic scene is realized, which plays an important and multi-dimensional role in site management; compared with the traditional management mode relying on two-dimensional drawings, planar images and artificial patrol, accurate three-dimensional site models can be generated in real time, key information such as construction progress, equipment operation status, material stacking position, etc. is fully displayed, so that the management personnel can intuitively and comprehensively master the site dynamics without going to the site, and the efficiency and accuracy of information acquisition are improved; at the same time, the system has real-time monitoring and intelligent early warning functions, and can alarm in time when an abnormality occurs, preventing safety accidents; it provides strong data support for the rational allocation of construction resources and the optimization of construction process, significantly improves the scientific nature and efficiency of site management, provides a solid guarantee for the smooth progress of the construction project, and helps the site to develop in the direction of high efficiency, safety and intelligence.

[0070] Although the above embodiments have been described in detail with reference to the accompanying drawings for the purpose of the conception and embodiments of the present application, those skilled in the art can recognize that various improvements and changes can be made to the present application without departing from the scope defined by the claims, and therefore, detailed description is not given here.

Claims

1. A method for construction site three-dimensional scene reconstruction based on images of unmanned aerial vehicles and surveillance videos, characterized in that The method comprises: S1: acquiring multi-view images of a construction site scene; S2: based on the images, generating a sparse three-dimensional point cloud and camera pose through feature matching and motion structure recovery algorithm, and performing dense reconstruction to obtain a global three-dimensional point cloud; S3: aligning the global three-dimensional point cloud with a world coordinate system by using geographic location information; S4: estimating the shooting camera position in the three-dimensional point cloud, sampling candidate viewpoints and rendering virtual RGB images; S5: based on two-dimensional feature matching of the shooting images and virtual RGB images, iteratively solving the internal and external parameters of the shooting camera through triangulation and pose optimization algorithm; S6: performing monocular depth estimation on the shooting images, and converting to metric scale through static area depth alignment; S7: projecting the depth of the shooting images to the three-dimensional point cloud, updating dynamic objects in real time and fusing into a complete three-dimensional scene model. 2.The construction site three-dimensional scene reconstruction method based on UAV images and monitoring videos of claim 1, characterized in that In step S1, multi-view images of a construction site scene are acquired by using a UAV or a general camera; When the multi-view images of the construction site scene are acquired by using a UAV, step S1 comprises: acquiring multi-view images of the construction site scene by using a UAV; performing quality screening on the images, and removing images with blur or abnormal exposure. 3.The construction site three-dimensional scene reconstruction method based on UAV images and monitoring videos of claim 1, wherein Step S2 comprises: extracting image feature points and calculating feature descriptors using SIFT algorithm; optimizing camera pose and three-dimensional point position through incremental SfM algorithm to generate a sparse three-dimensional point cloud; performing multi-view stereo matching based on the sparse point cloud to generate a dense three-dimensional point cloud. 4.The construction site three-dimensional scene reconstruction method based on UAV images and monitoring videos of claim 1, wherein Step S2 further comprises: performing Poisson surface reconstruction on the dense three-dimensional point cloud to generate a mesh model; and mapping the UAV image texture to the mesh model to enhance visual realism.

5. The method of claim 1, wherein the method further comprises: determining a location of the UAV based on the UAV image; and determining a location of the camera based on the monitoring video. In step S3: the geographic location information is acquired by using a GPS / RTK device; and a rigid transformation matrix is estimated by using least squares method to align the three-dimensional point cloud with the world coordinate system. 6.The construction site three-dimensional scene reconstruction method based on UAV images and monitoring videos of claim 1, characterized in that In step S4, the shooting camera is a surveillance camera or a general camera; When the shooting camera is a surveillance camera, step S4 comprises: uniformly sampling candidate viewpoints on a hemisphere around the estimated position of the surveillance camera; defining a virtual pinhole camera for each candidate viewpoint and rendering a virtual RGB image.

7. The method of claim 1, wherein the method further comprises: determining a location of the UAV based on the UAV image; and determining a location of the camera based on the monitoring video. Step S5 comprises: extracting feature points of the shooting images and virtual RGB images using SuperPoint algorithm; recovering the three-dimensional positions of the matching points through triangulation; and iteratively optimizing the shooting camera parameters by combining PnP algorithm and bundle adjustment method. 8.The construction site three-dimensional scene reconstruction method based on UAV images and monitoring videos of claim 1, wherein Step S6 comprises: A deep learning model is used for monocular depth estimation; a static scene region is segmented, and a linear transformation relationship between monocular depth and rendering depth is fitted by a RANSAC algorithm: In the formula, is the converted metric depth, D is the monocular depth estimation result, s is a scale factor, and t is an offset. 9.The construction site 3D scene reconstruction method based on UAV images and surveillance videos of claim 1, wherein Dynamic object updating in step S7 comprises: calculating the position of dynamic objects in three-dimensional space in real time according to the depth map and camera parameters; and fusing the dynamic object model with the static scene point cloud.

Images