A construction site three-dimensional map construction method fusing multi-source visual sensors

Abstract

The application belongs to the technical field of image processing, and particularly relates to a construction site three-dimensional map construction method fusing multiple source visual sensors, which arranges a multiple source visual sensor network at key structural parts of a construction area, ensures that the fields of view of at least two sensors in the same monitoring area overlap with each other, and performs space-time synchronization and coordinate correction on collected image data, so that image sequences under a unified space-time reference are obtained, subsequent feature matching, disparity calculation and point cloud fusion are ensured to be performed under a consistent time and space framework, the accuracy and consistency of three-dimensional map construction are ensured, local sparse point clouds and local dense point clouds are generated and registered and fused, the problem of insufficient accuracy or missing details of a single reconstruction method is effectively solved, the fused point clouds are converted to an absolute coordinate system, a dynamically updatable construction site three-dimensional digital map is finally generated, and the visualization of a construction scene is realized.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically a method for constructing a three-dimensional map of a construction site by integrating multiple visual sensors. Background Technology

[0002] With the rapid development of Building Information Modeling (BIM) and intelligent construction technologies, digital management of construction sites has become a key means to improve project quality, safety, and efficiency. Among these, constructing high-precision, high-fidelity 3D maps of construction sites is fundamental to achieving visualized monitoring of the construction process, progress analysis, and risk early warning.

[0003] In existing technologies, 3D maps are generated by importing multi-source data such as laser scanning, oblique photography, and IoT sensors into a 3D graphics engine and manually registering them using manual marking or a predefined coordinate system.

[0004] However, the existing technology still has the following limitations: 1. Multiple sensors are often deployed at construction sites. Due to the lack of effective time and space synchronization and coordinate correction mechanisms, the data collected by each sensor has time delay and inconsistent spatial coordinates, which makes it difficult to integrate the data later and form a unified scene expression, affecting the integrity and consistency of the 3D map.

[0005] 2. Sparse point clouds generated by traditional methods can only reflect the general outline of the scene and cannot express the fine geometric features of the component surface. Dense point clouds, although rich in details, are prone to noise and errors when lacking constraints, resulting in the reconstructed 3D model failing to meet the requirements of high-precision construction monitoring. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention provides a method for constructing a three-dimensional map of a construction site by integrating multiple visual sensors, which can effectively solve the problems mentioned in the prior art.

[0007] The objective of this invention can be achieved through the following technical solution: a method for constructing a three-dimensional map of a construction site by integrating multiple source visual sensors, comprising: deploying a network of multiple source visual sensors at key structural locations in the construction area, so that the fields of view of at least two visual sensors in the same monitoring area overlap with each other.

[0008] The acquired image data is spatiotemporally synchronized and coordinate corrected to obtain an image sequence under a unified spatiotemporal reference.

[0009] Two-dimensional feature points are extracted from each image sequence and cross-view matching is performed. The three-dimensional coordinates of the feature points are calculated to generate a local sparse point cloud.

[0010] Based on the stereo matching results between images acquired by multiple visual sensors within the same monitoring area, corresponding parallax information is obtained, a dense depth map is generated, and a local dense point cloud is obtained by backprojection.

[0011] The local sparse point cloud and the local dense point cloud are registered and fused, and the fused point cloud is transformed to the absolute coordinate system by combining ground control points to obtain the point cloud model of the construction site.

[0012] The point cloud model is reconstructed and textured to generate a 3D mesh model. The construction elements are semantically identified and classified, and associated with building information data to form a dynamically updatable 3D digital map of the construction site.

[0013] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects: (1) The present invention deploys a multi-source visual sensor network in key structural parts of the construction area to ensure that the fields of view of at least two sensors in the same monitoring area overlap, and performs spatiotemporal synchronization and coordinate correction on the collected image data to obtain an image sequence under a unified spatiotemporal reference. This ensures that subsequent feature matching, disparity calculation and point cloud fusion are all carried out in a consistent time and space framework, thus guaranteeing the accuracy and consistency of the three-dimensional map construction.

[0014] (2) By combining the generation and registration of local sparse point clouds and local dense point clouds, this invention effectively solves the problem of insufficient accuracy or lack of detail in a single reconstruction method. The fused point cloud is then converted to an absolute coordinate system and subjected to surface reconstruction, texture mapping, semantic recognition and classification. Finally, a dynamically updatable three-dimensional digital map of the construction site is generated, realizing the visualization of the construction scene. Attached Figure Description

[0015] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0017] Figure 2 A flowchart for generating fused point clouds for this invention;

[0018] Figure 3 This is a logic diagram for determining whether a point is a fusion point in this invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Reference Figure 1 As shown, the present invention provides a method for constructing a three-dimensional map of a construction site by integrating multiple visual sensors, including: S1, deploying a network of multiple visual sensors at key structural parts of the construction area, so that the fields of view of at least two visual sensors in the same monitoring area overlap with each other.

[0021] Considering that subsequent 3D reconstruction relies on multi-view images for feature matching and disparity calculation, by making the fields of view of at least two visual sensors in the same monitoring area overlap, we can provide an observation basis for extracting corresponding feature points and generate sufficient disparity information to obtain depth data of the target area.

[0022] The specific process of deploying a multi-source visual sensor network at key structural parts in the construction area is as follows: First, obtain the building information model or structural design drawings of the construction area. Based on the marked engineering importance level and structural stress characteristics, extract key structural parts such as steel structure nodes and reinforced concrete core areas. Based on the outer contour boundary of each key structural part, combined with the distribution range of adjacent components and the monitoring distance of the visual sensors, delineate the monitoring area corresponding to each key structural part.

[0023] Secondly, geometrical measurements were used to obtain the geometrical dimensions and spatial morphological characteristics of each monitoring area. Based on the locations of adjacent components, construction machinery, scaffolding, temporary facilities, and material stacking, the sources, directions, and extent of shading in each monitoring area were determined, and areas of continuous shading and areas of phased shading were distinguished. Based on the construction period, the direction of natural light, and the layout of on-site lighting, the brightness distribution, shadow range, and backlight areas of each monitoring area were determined.

[0024] Next, based on the geometric dimensions, occlusion conditions, and lighting conditions of each monitoring area, the corresponding image acquisition range is determined. The image acquisition range includes the effective observation boundary, acquisition distance range, horizontal viewing angle range, and vertical viewing angle range.

[0025] Finally, based on the image acquisition range, the type, installation location, number, and installation posture of the multi-source vision sensors are configured, and the orientation, pitch angle, focal length, and acquisition parameters of each vision sensor are adjusted. The specific process is as follows: Select stable, fixed carriers with power supply and communication capabilities that do not affect on-site construction as candidate installation points around each monitoring area. Combine the results of the construction site survey and the spatial distribution information of the construction area to calculate the observation distance, visible range, observation angle, and occlusion of each candidate installation point to the corresponding monitoring area. Candidate installation points with obstructed lines of sight, insufficient installation stability, or inability to meet the acquisition range requirements are eliminated. The installation location and number of vision sensors are determined based on the condition that the target surface in the monitoring area is within the effective field of view of at least two vision sensors at the same time, and the corresponding images meet the field of view overlap condition.

[0026] The installation posture includes the installation height of the visual sensors, the direction of their optical axes, and the relative angle between them and the monitoring area. By adjusting the orientation and pitch angle of each visual sensor, the optical axis of the visual sensor is made to point to the center position of the corresponding monitoring area or the key structural surface, and an observation baseline relationship that meets the requirements of three-dimensional reconstruction is formed between adjacent visual sensors. The focal length parameter is determined according to the monitoring distance, the boundary range of the monitoring area, and the target resolution requirements, so that the key structural parts are completely included in the image acquisition range, and the target area has an imaging scale that meets the requirements of feature extraction in the image, thereby constructing a multi-source visual sensor network covering the key structural parts of the construction area.

[0027] It should be noted that the types of multi-source vision sensors include high-resolution visible light cameras, binocular stereo cameras, and wide-angle or panoramic cameras.

[0028] It should also be noted that by constructing a multi-source visual sensor network covering key structural parts of the construction site through step S1, the same monitoring area is effectively covered by at least two sensors, avoiding occlusion interference and uneven lighting during the construction process, ensuring the integrity and imaging quality of image acquisition in the monitoring area, and thus providing basic data for the generation of local sparse point clouds, reconstruction of local dense point clouds, and subsequent point cloud registration and fusion.

[0029] S2. The acquired image data is spatiotemporally synchronized and coordinate corrected to obtain an image sequence under a unified spatiotemporal reference.

[0030] In this embodiment, the multi-source vision sensors have time base differences and lens distortion during independent acquisition. Based on this, the image data needs to be spatiotemporally synchronized and coordinate corrected. The specific process is as follows: obtain the timestamp information corresponding to the images acquired by each vision sensor, and perform unified time synchronization for each sensor based on a network time protocol or a high-precision clock synchronization device, and perform time alignment of the image data acquired by each vision sensor according to the unified clock base.

[0031] After the vision sensors are installed, images of control feature points of known dimensions are collected on site to calibrate each vision sensor and obtain the corresponding intrinsic parameters, extrinsic parameters and distortion parameters.

[0032] The intrinsic parameters include focal length, principal point coordinates, and pixel scale parameters; the extrinsic parameters include rotation and translation parameters of the visual sensor relative to a unified spatial reference; and the distortion parameters include radial distortion parameters and tangential distortion parameters.

[0033] Based on the focal length, principal point coordinates, radial distortion parameters, and tangential distortion parameters obtained from the calibration of each vision sensor, an imaging model for the corresponding vision sensor is established; the pixel coordinates in the original image are transformed to the normalized imaging plane, and the pixel positions affected by lens distortion are reverse-corrected to obtain the corrected pixel coordinates; the original image is resampled based on the corrected pixel coordinates to generate a distortion-corrected image, thereby eliminating straight line bending, edge stretching, and local geometric offset caused by lens distortion.

[0034] Based on the extrinsic parameters, a pre-set local coordinate system of the construction area is used as a unified spatial reference to determine the spatial pose of each visual sensor relative to the unified spatial reference, and the imaging coordinate relationship of each visual sensor is transformed to the unified spatial reference. For image pairs with overlapping fields of view, an epipolar geometric relationship is constructed according to the intrinsic parameters and relative pose relationship of each visual sensor, the correction transformation matrix is ​​calculated, and epipolar correction is performed on the image pairs so that the corresponding points of the same spatial point in images from different viewpoints are located on the same epipolar line.

[0035] Multi-source image data that has been time-aligned and coordinate-corrected are sorted according to a unified time axis, and the correlation between each image and the pose parameters of the corresponding visual sensor is established to generate an image sequence under a unified spatiotemporal reference.

[0036] It should be noted that the preset local coordinate system is established through control points on the construction site. Specifically, at least three fixed and non-collinear control points are selected within the construction area. After measuring their spatial positions, a plane coordinate axis is established with one of the control points as the origin and the line connecting adjacent control points as the reference direction. An elevation axis is established in the vertical direction to form the preset local coordinate system of the construction area.

[0037] It should also be noted that each image frame in the image sequence processed above corresponds to a unique acquisition time and spatial pose information, thus providing a data foundation for subsequent cross-view feature matching, disparity calculation, depth recovery, local sparse point cloud generation, and local dense point cloud reconstruction.

[0038] S3. Extract two-dimensional feature points from each image sequence and perform cross-view matching, calculate the three-dimensional coordinates of the feature points, and generate a local sparse point cloud.

[0039] To recover the three-dimensional geometric information of the target scene from two-dimensional images, it is necessary to establish the correspondence between corresponding points in images from different perspectives and calculate their spatial coordinates based on the principle of multi-view geometry. Therefore, after obtaining the image sequence under a unified spatiotemporal reference, it is necessary to extract the two-dimensional feature points in each image and perform cross-view matching to obtain the pixel correspondence of corresponding points in images from different perspectives. Then, the three-dimensional coordinates of the feature points under the unified spatiotemporal reference are calculated based on the triangulation method to generate a local sparse point cloud that reflects the spatial distribution and geometric framework of key structural parts.

[0040] The specific process of extracting two-dimensional feature points from each image sequence and performing cross-view matching is as follows: Select time-synchronized and visually overlapping image pairs from the image sequence under a unified spatiotemporal reference as images to be matched.

[0041] The images to be matched are convolved with Gaussian kernel functions of different scales to generate a Gaussian pyramid image sequence; the Gaussian images of adjacent scales are then differentially processed to obtain the Gaussian difference scale space.

[0042] In the Gaussian difference scale space, each pixel is compared with its 8 neighboring pixels at the same scale and its 18 neighboring pixels at adjacent upper and lower scales. If the response value of the pixel is greater than or less than all neighboring pixels, it is marked as a candidate feature point.

[0043] Construct a gradient covariance matrix from the gray-level gradients of the neighborhood of candidate feature points, and determine their structural properties based on the eigenvalue distribution of the gradient covariance matrix. When the gradient covariance matrix has two non-zero eigenvalues, retain the corresponding candidate feature points. When the gradient covariance matrix has only one non-zero eigenvalue or no non-zero eigenvalues, remove the corresponding candidate feature points. Determine the retained candidate feature points as the final two-dimensional feature points.

[0044] Centered on a two-dimensional feature point, the gradient direction histogram of pixels in its neighborhood is calculated. The peak direction of the histogram is taken as the principal direction of the feature point, and its neighborhood is divided into several sub-regions after being rotated according to the principal direction. The gradient magnitude and direction of each pixel in each sub-region are calculated and accumulated to generate a gradient direction histogram vector. The histogram vectors of all sub-regions are concatenated and normalized to form a feature descriptor with illumination and rotation invariance.

[0045] For any two-dimensional feature point in the image to be matched, its corresponding feature descriptor is read and the distance is calculated with the feature descriptors of each two-dimensional feature point in another image to be matched. The two-dimensional feature point with the smallest feature descriptor distance is selected to establish an initial matching point pair. The initial matching point pair is verified by reverse matching. When the reverse matching result still points to the original two-dimensional feature point, the initial matching point pair is retained; otherwise, it is discarded as the preliminary matching result.

[0046] For each feature point pair in the preliminary matching result, its corresponding epipolar line in the other image is determined based on the homogeneous coordinates of the two-dimensional feature point in one image and the fundamental matrix between the two view images. The point-to-line distance from the matching feature point in the other image to the corresponding epipolar line is calculated, and the epipolar line consistency test is performed on the preliminary matching result based on the point-to-line distance. Feature point pairs that do not satisfy the epipolar line geometric relationship are eliminated to obtain the effective matching point pairs between the two view images.

[0047] For images with three or more images participating in the matching within the same monitoring area, the effective matching point pairs obtained from images from different perspectives are associated and integrated according to the relationship of corresponding points to form a cross-view feature matching set that corresponds consistently across multiple perspectives.

[0048] The process of calculating the three-dimensional coordinates of feature points and generating a local sparse point cloud is as follows: obtain the pixel coordinates of the same two-dimensional feature points in the cross-view feature matching set in the images of each view, and call the intrinsic and extrinsic parameters of the corresponding visual sensor; based on the intrinsic parameters, perform principal point correction and scale normalization on the pixel coordinates to obtain the corresponding normalized imaging coordinates.

[0049] Starting from the optical center of the corresponding visual sensor, the imaging direction of the two-dimensional feature point in the visual sensor coordinate system is determined according to the normalized imaging coordinates, forming an imaging line of sight from the optical center to the target space.

[0050] Based on the external parameters, the imaging lines of view in each view are transformed from the visual sensor coordinate system to a unified spatiotemporal reference to obtain the spatial observation relationship of the same two-dimensional feature points under different viewpoints. The pixel coordinates of the same two-dimensional feature points, the optical center position of the corresponding visual sensor, and the direction of the imaging line are associated to establish the projection relationship between the two-dimensional pixels in each view image and the target spatial point.

[0051] Based on the projection relationship corresponding to each viewpoint image, the optical center position of each visual sensor and the spatial imaging line of sight from the optical center to the corresponding pixel point are determined respectively. Two imaging lines are used for the solution. If the two imaging lines do not strictly intersect in space, the two closest points corresponding to the shortest line between the two imaging lines are obtained, and the midpoint of the two closest points is used as the initial three-dimensional coordinates of the target point.

[0052] Substitute each initial 3D coordinate into the projection relationship of each viewpoint image participating in the triangulation, and obtain the reprojection point coordinates in each viewpoint image; calculate the pixel distance between each reprojection point coordinate and the corresponding original matching pixel coordinate to obtain the reprojection error of each viewpoint.

[0053] Based on the coordinates of the reprojection point, the two-dimensional feature point closest to its position is retrieved from the set of two-dimensional feature points in the corresponding viewpoint image, and it is determined whether the retrieved two-dimensional feature point is still the original matching feature point participating in the triangulation. When the two-dimensional feature points retrieved in each viewpoint are consistent with the original matching feature points, it is determined that the corresponding initial three-dimensional coordinates satisfy the multi-view matching consistency and are retained.

[0054] When the two-dimensional feature points retrieved from some viewpoints are inconsistent with the original matching feature points, the observation data of the corresponding viewpoints are removed, and triangulation is performed again based on the remaining consistent viewpoints. If at least two viewpoints still maintain a consistent matching relationship after resolving, the re-solved three-dimensional coordinates are retained as valid three-dimensional feature points; otherwise, the corresponding three-dimensional coordinates are judged as abnormal three-dimensional points and removed. The retained valid three-dimensional feature points are collected to form a set of valid three-dimensional feature points.

[0055] All three-dimensional points in the effective three-dimensional feature point set are uniformly represented under a unified spatiotemporal reference. For each three-dimensional point, neighboring three-dimensional points are searched in its spatial neighborhood, and three-dimensional points with overlapping or close spatial positions are clustered.

[0056] For multiple 3D points belonging to the same class after clustering, the duplicate recovery results corresponding to the same target point are determined, and the coordinates of multiple 3D points are merged to generate a fused 3D point. For 3D points that do not form a cluster relationship with other 3D points, they are directly retained. All fused 3D points and retained 3D points are output together to form a local sparse point cloud.

[0057] It should be noted that by extracting two-dimensional feature points and performing cross-view matching and triangulation, a local sparse point cloud reflecting the spatial distribution of key structural parts can be generated under a unified spatiotemporal reference. This provides spatial constraints and position guidance for subsequent reconstruction of local dense point clouds, and provides an initial position reference for the registration and fusion of multi-source point clouds.

[0058] S4. Based on the stereo matching results between images acquired by multiple visual sensors in the same monitoring area, obtain the corresponding parallax information, generate a dense depth map, and back-project it to obtain a local dense point cloud.

[0059] Considering that although local sparse point clouds reflect the discrete feature point distribution of key structural parts, they are difficult to fully express the continuous geometric shape of the target surface; therefore, after obtaining sparse point clouds, it is necessary to calculate the disparity information of dense pixels and convert it into depth data based on pixel-level stereo matching between multi-view images in the same monitoring area, and then back-project to generate local dense point clouds with continuous surface details.

[0060] Based on this, the process of generating a dense depth map and back-projecting it to obtain a local dense point cloud is as follows: under a unified spatiotemporal reference, the image sequence is grouped according to the monitoring area number and the acquisition time, and multiple images belonging to the same monitoring area and with the same acquisition time are extracted as candidate images in the same batch.

[0061] Based on the external parameters, imaging direction, and field of view coverage of each visual sensor, the common observation of the same target surface in candidate images is analyzed, and images with common observation areas are retained.

[0062] By combining the differences in observation direction, common observation area, and image sharpness among the images, multiple images with relatively complete coverage of the target surface and complementary perspectives are selected for combination. Among them, the image with complete imaging range and less occlusion is selected as the reference image, and the remaining images are used as auxiliary images. The reference image and the auxiliary images are jointly determined as a dense reconstruction image group.

[0063] Based on the intrinsic and extrinsic parameters of each visual sensor, the imaging geometric relationship between the reference image and the auxiliary image is determined, and the corresponding epipolar lines of each pixel in the reference image in each auxiliary image are generated.

[0064] On the corresponding epipolar lines of each auxiliary image, a first pixel neighborhood is constructed in the reference image with the pixel to be matched as the center, and a second pixel neighborhood with the same size as the first pixel neighborhood is constructed in the auxiliary image with each candidate pixel as the center.

[0065] Establish a one-to-one correspondence between pixels with the same relative position in the neighborhood of two pixels, calculate the gray level difference between the corresponding pixels, and accumulate all gray level differences to obtain the matching difference value between the pixel to be matched and each candidate pixel. Compare the matching difference values ​​corresponding to each candidate pixel, and select the candidate pixel with the smallest value as the corresponding matching pixel of the pixel to be matched.

[0066] The pixels to be matched that pass the matching consistency test are associated with their corresponding matching pixels in each auxiliary image to obtain the pixel correspondence in the densely reconstructed image group.

[0067] Taking each reference pixel in the pixel correspondence relationship as the object, obtain its corresponding pixel position in each auxiliary image, and take the pixel coordinate difference between the reference pixel and the corresponding pixel in the corresponding epipolar direction as the disparity value of the reference pixel, i.e., disparity information.

[0068] When the optical axes of the visual sensors corresponding to the reference image and the auxiliary image are parallel and their imaging planes are coplanar, the depth value calculation formula is used based on the principle of triangulation. Perform the conversion, among which For depth value, The baseline distance between the optical centers of the two sensors. The effective focal length value is calculated by pixel-scale conversion based on the focal length parameters of the corresponding visual sensor for the reference image. This represents the disparity value.

[0069] It should be noted that for vision sensors with non-parallel optical axis configurations, the one-dimensional disparity along the epipolar direction is calculated based on the coordinate differences of the matching pixels on their respective image planes, combined with epipolar geometric constraints, and used as the disparity information of the reference pixel.

[0070] When the vision sensor is configured with a non-parallel optical axis, a projection matrix for stereo vision is constructed based on the intrinsic and extrinsic parameters corresponding to the reference and auxiliary images. The three-dimensional coordinates of the target point in the sensor coordinate system corresponding to the reference image are solved by the spatial forward intersection method, and its depth component is extracted as the depth information of the pixel.

[0071] When the same reference pixel has valid corresponding pixels in multiple auxiliary images, the depth value corresponding to each auxiliary image is calculated respectively, and the depth value is assigned a corresponding fusion weight according to the matching difference value of each correspondence. The initial depth value of the pixel is obtained by fusion calculation using a weighted average method. The smaller the matching cost, the greater the weight.

[0072] Based on the spatial resolution of the reference image, a data matrix with the same size as the reference image is constructed. According to the pixel coordinate index of the reference image, the initial depth value corresponding to each pixel is stored in the corresponding position cell of the data matrix to generate an initial dense depth map corresponding to the dense reconstructed image group.

[0073] For each pixel in the initial dense depth map, obtain its pixel coordinates and corresponding depth value. Based on the intrinsic parameters of the visual sensor corresponding to the reference image, convert the pixel coordinates into the imaging direction in the sensor coordinate system, and combine the depth value to determine the three-dimensional position of the pixel in the sensor coordinate system.

[0074] By calling the extrinsic parameters of the vision sensor, the three-dimensional position is transformed from the sensor coordinate system to a unified spatiotemporal reference, thus obtaining the global three-dimensional spatial coordinates corresponding to the pixel.

[0075] By iterating through all pixels in the depth map and performing the above transformations sequentially, all the obtained global 3D spatial coordinates are aggregated to form a local dense point cloud.

[0076] S5. Register and fuse the local sparse point cloud and the local dense point cloud, and combine the ground control points to transform the fused point cloud to the absolute coordinate system to obtain the point cloud model of the construction site.

[0077] Considering that local sparse point clouds and local dense point clouds are complementary in geometric representation, they need to be registered and fused. The process of registering and fusing local sparse point clouds and local dense point clouds is as follows: obtain local sparse point clouds and local dense point clouds corresponding to the same monitoring area; based on the spatial distribution of three-dimensional points in the local sparse point cloud, select three-dimensional points located at structural corners, edge intersections, or areas with significant curvature changes to form a benchmark feature point set.

[0078] For each 3D point in the baseline feature point set, based on the original image feature point information corresponding to that point during local sparse point cloud reconstruction, the pixel coordinates and feature descriptor of that point in the 2D image are obtained.

[0079] The 3D points in the local dense point cloud are projected onto the image plane corresponding to the same visual sensor to obtain the pixel coordinates of each 3D point. Based on the pixel coordinates and feature descriptors of each feature point in the reference feature point set in the image, 3D points that are spatially adjacent to the reference feature points and have similar feature descriptors are searched among the projection points of the local dense point cloud to establish the initial correspondence between the reference feature points and the points in the local dense point cloud.

[0080] Using the least squares method, the initial rotation matrix and translation vector of the local dense point cloud relative to the local sparse point cloud are calculated based on the coordinate correspondence between the reference feature point and the corresponding dense point. The initial rotation matrix and translation vector are then applied to the local dense point cloud to initially align the local dense point cloud to the coordinate system of the local sparse point cloud, thus completing the initial registration and positioning of the local dense point cloud.

[0081] Based on the spatial distribution of the local dense point cloud after initial registration, the nearest neighbor search method is used to search for the nearest neighbor point in the local dense point cloud for each 3D point in the local sparse point cloud.

[0082] When the nearest neighbor distance is less than the preset spatial overlap threshold, it is determined that the points in the local sparse point cloud fall into the overlapping area of ​​the two point clouds. All the points in the local sparse point cloud that meet the above conditions and their corresponding neighboring points in the local dense point cloud together constitute the overlapping area point set.

[0083] The preset spatial overlap threshold is obtained through statistical methods: after coarse registration is completed, some points in the local sparse point cloud are randomly selected, and the distance distribution between each point and its nearest neighbor in the local dense point cloud is statistically analyzed. The median of this distance distribution is multiplied by 1.5 to 2.5 times as the preset spatial overlap threshold.

[0084] For each point in the local sparse point cloud of the overlapping region point set, its nearest neighbor in the local dense point cloud is taken as a candidate corresponding point to form an initial set of corresponding point pairs; the angle between the normal vectors of the two points in each corresponding point pair is calculated, and corresponding point pairs with the same normal vector direction are retained.

[0085] Using the least squares method, the rotation matrix and translation vector that align the local dense point cloud to the local sparse point cloud are calculated based on the coordinate differences between corresponding point pairs, and this is used as the coarse registration result.

[0086] Based on the rotation matrix and translation vector determined by coarse registration, the local dense point cloud is transformed to the coordinate system of the local sparse point cloud, establishing an initial correspondence between the two point clouds. For each 3D point in the local sparse point cloud, its corresponding 3D point is determined in the transformed local dense point cloud through spatial nearest neighbor search, forming an initial set of corresponding point pairs.

[0087] We construct a joint optimization objective that integrates the corresponding points measured by Euclidean distance, the normal constraint measured by the cosine of the angle between the normal vectors at the two points, and the local geometric features measured by the distribution of the eigenvalues ​​of the covariance matrix of the point clouds in the local neighborhood of the two points.

[0088] The least squares method is used to solve for the rotation matrix and translation vector that minimize the distance between corresponding points. These are used as the transformation parameters for this iteration and applied to update the spatial position of the local dense point cloud.

[0089] Repeat the above process of establishing correspondences and solving transformation parameters to form an iterative optimization loop. After each iteration, record the rotation matrix and translation vector obtained in this iteration, as well as the average distance between all corresponding point pairs, and calculate the change in rotation angle, the change in translation vector, and the decrease in average distance of the current iteration relative to the previous iteration.

[0090] When the changes in rotation angle and translation vector both decrease continuously and no longer show a decreasing trend in multiple consecutive iterations, and the decrease in the average distance between corresponding point pairs approaches the minimum value allowed by the calculation accuracy, the optimization process is determined to have converged, the iteration is terminated, and the rotation matrix and translation vector obtained at the time of convergence are taken as the fine registration result.

[0091] Reference Figure 2 and Figure 3 As shown, the coordinates of the local dense point cloud and the local sparse point cloud are unified according to the fine registration results, and the two are fused according to the preset fusion rules. The preset fusion rules are as follows: in the overlapping area of ​​the local sparse point cloud and the local dense point cloud, for each three-dimensional point in the local sparse point cloud, the corresponding three-dimensional point in the local dense point cloud is determined by the nearest neighbor search to form a corresponding point pair, and the distance between the corresponding points is calculated.

[0092] If the distance between points is less than a preset distance threshold, the point data in the local dense point cloud is retained, and the corresponding points are marked as fusion points.

[0093] Otherwise, the corresponding point is judged as an outlier and removed.

[0094] The point data in the local dense point cloud marked as fusion points are merged with the point data in the local sparse point cloud that does not fall into the overlapping region and the point data in the local dense point cloud that does not fall into the overlapping region to generate a fused point cloud.

[0095] It should be noted that the process of obtaining the preset distance threshold is as follows: after the fine registration is completed, the distance values ​​of all corresponding point pairs in the overlapping area are obtained to form a distance distribution set, and the average value of the set plus 1 to 2 times the standard deviation is used as the preset distance threshold.

[0096] The process of combining ground control points to transform the fused point cloud into an absolute coordinate system and obtain the point cloud model of the construction site is as follows: obtain the coordinates of several ground control points at the construction site in the absolute coordinate system.

[0097] Based on the spatial distribution of ground control points, a local search is performed in the corresponding area of ​​the fused point cloud, and the three-dimensional point in the point cloud that is closest to the measured location of the control point is used as the initial matching point.

[0098] Centered on the initial matching point, a neighborhood point set with a preset radius, such as twice the average point spacing of the point cloud, is selected, and the geometric centroid of the point set is calculated as the feature point coordinates of the control point in the local coordinate system.

[0099] Establish a one-to-one correspondence between the characteristic point coordinates of each ground control point in the local coordinate system and their measured coordinates in the absolute coordinate system to form the observation data for transform parameter calculation.

[0100] Using rotation parameters, translation parameters, and scale parameters as unknowns to be solved, a transformation relationship from the local coordinate system to the absolute coordinate system is established based on the principle of spatial coordinate transformation.

[0101] Based on the transformation relationship, the feature point coordinates of each control point are expressed as a function containing rotation parameters, translation parameters, and scale parameters, thus obtaining the calculated coordinate expression of the transformed control point; the difference between the calculated coordinates and the measured coordinates is used as the error equation of the control point.

[0102] For all control points, corresponding error equations are established, forming a system of error equations. The objective function is constructed with the goal of minimizing the sum of the squares of all error equations.

[0103] The objective function is solved using the least squares method. The values ​​of rotation, translation, and scale parameters are adjusted through iterative calculations until the objective function value converges to the minimum value. The corresponding parameter values ​​at this point are the optimal estimates of the rotation, translation, and scale parameters, which are used as coordinate transformation parameters from the local coordinate system to the absolute coordinate system.

[0104] For each 3D point in the fused point cloud, its coordinate value in the local coordinate system is read, and this coordinate value is used as the point to be transformed.

[0105] Based on the coordinate transformation parameters, the current 3D point is sequentially subjected to scaling, coordinate axis rotation, and origin translation to calculate the transformed coordinates of the 3D point in the absolute coordinate system.

[0106] Traverse all 3D points in the merged point cloud and perform the coordinate transformation operation described above in sequence to transform the local coordinates of each point to the absolute coordinate system one by one.

[0107] All the transformed 3D points are organized according to the original spatial topology to form a construction site point cloud model unified in an absolute coordinate system.

[0108] It should be noted that the ground control points include, but are not limited to, the forced centering marks and structural corner points of the construction control network, and there shall be no fewer than three of them, which shall not be collinear.

[0109] S6. Perform surface reconstruction and texture mapping on the point cloud model to generate a three-dimensional mesh model. Perform semantic recognition and classification on construction elements and associate them with building information data to form a dynamically updatable three-dimensional digital map of the construction site.

[0110] The process of semantic recognition and classification of construction elements and their association with building information data is as follows: First, a three-dimensional mesh model generated after surface reconstruction and texture mapping is obtained. This three-dimensional mesh model consists of several geometric units, each corresponding to a physical component or structural surface on the construction site. A semantic segmentation network is then used to process the three-dimensional mesh model. The semantic segmentation network uses the spatial geometric features, texture features, and adjacency relationships of each geometric unit as input feature vectors. Multi-scale features are extracted through an encoder-decoder structure, and the probability distribution of each geometric unit belonging to a preset construction element category is output through a fully connected layer and a Softmax classifier. The category corresponding to the highest probability is taken as the semantic classification result for that geometric unit. The preset construction element categories include structural components, temporary facilities, construction machinery, and material storage areas.

[0111] Secondly, based on the semantic segmentation results, semantic tags are assigned to each geometric unit. These semantic tags include at least the following fields: type identifier (used to distinguish construction element categories), location identifier (used to uniquely determine the spatial location of the construction element), and status identifier (used to mark the current construction stage or usage status of the construction element). A one-to-one correspondence is established between the semantic tags and the unique identifiers of the corresponding components in the Building Information Model (BIM). Specifically, using the location identifier in the semantic tags as the matching benchmark, components with overlapping spatial locations or distances less than a preset threshold are retrieved from the BIM. An example preset threshold is 0.1 meters. Their unique identifiers are then bound to the current geometric unit. After binding, each geometric unit inherits the attribute information of the corresponding component, including material properties, design parameters, and construction stage information.

[0112] Then, based on the newly added image sequence, the geometric shape and spatial position of the current construction elements are extracted and compared with the geometric attributes and position information of the corresponding construction elements in the three-dimensional mesh model to identify the construction elements that have changed. For the identified changed elements, their geometric attributes, position information and construction progress attributes in the three-dimensional mesh model are updated so that the three-dimensional mesh model can reflect the actual state of the construction site in real time.

[0113] Finally, the updated 3D mesh model is synchronized with the associated data in the building information model to output a 3D digital map of the construction site containing geometric information, semantic information, and dynamic update status, forming a dynamically updatable 3D digital map of the construction site.

[0114] The above description is merely an example and illustration of the structure of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the structure of the invention or exceed the scope defined by the present invention, they should all fall within the protection scope of the present invention.

Claims

1. A method for constructing a 3D map of a construction site by integrating multiple visual sensors, characterized in that, include: A multi-source visual sensor network is deployed at key structural locations in the construction area, so that the fields of view of at least two visual sensors in the same monitoring area overlap. The acquired image data is spatiotemporally synchronized and coordinate corrected to obtain an image sequence under a unified spatiotemporal reference. Two-dimensional feature points are extracted from each image sequence and cross-view matching is performed. The three-dimensional coordinates of the feature points are calculated to generate a local sparse point cloud. Based on the stereo matching results between images acquired by multiple visual sensors in the same monitoring area, the corresponding parallax information is obtained, a dense depth map is generated, and a local dense point cloud is obtained by back projection. The local sparse point cloud and the local dense point cloud are registered and fused, and the fused point cloud is transformed to the absolute coordinate system by combining ground control points to obtain the point cloud model of the construction site. The point cloud model is reconstructed and textured to generate a 3D mesh model. The construction elements are semantically identified and classified, and associated with building information data to form a dynamically updatable 3D digital map of the construction site.

2. The method for constructing a 3D map of a construction site by fusing multiple visual sensors according to claim 1, characterized in that, The specific process of deploying a multi-source vision sensor network at key structural locations in the construction area is as follows: Based on the structural distribution of the construction area, key structural parts and their corresponding monitoring areas are identified, and the geometric dimensions, occlusion conditions and lighting conditions of each monitoring area are analyzed to determine the image acquisition range. The type, installation location, number, and installation posture of the multi-source vision sensors are determined based on the image acquisition range, so that adjacent vision sensors can form cross-observations of the same monitoring area; Adjust the orientation, pitch angle, and acquisition parameters of each visual sensor to enable at least two visual sensors within the same monitoring area to obtain image data with overlapping fields of view, thus constructing a multi-source visual sensor network.

3. The method for constructing a 3D map of a construction site by fusing multiple visual sensors according to claim 1, characterized in that, The spatiotemporal synchronization and coordinate correction include: Obtain the timestamp information corresponding to the images acquired by each vision sensor, and perform time alignment of the image data acquired by each vision sensor according to a unified clock reference; Each vision sensor is calibrated to obtain the corresponding intrinsic, extrinsic, and distortion parameters. Based on the intrinsic, extrinsic, and distortion parameters, distortion correction and coordinate transformation are performed on the image data. The corrected multi-source image data is mapped to a unified coordinate reference to obtain an image sequence under a unified spatiotemporal reference.

4. The method for constructing a 3D map of a construction site by fusing multiple visual sensors according to claim 3, characterized in that, The specific process of extracting two-dimensional feature points from each image sequence and performing cross-view matching is as follows: Select time-synchronized and visually overlapping image pairs from the image sequence as images to be matched; Two-dimensional feature points are extracted from the images to be matched, and corresponding feature descriptors are generated. Based on feature descriptors, preliminary feature matching is performed on images of the same monitoring area collected by different visual sensors. False matching points are eliminated using epipolar geometric constraints, and a cross-view feature matching set is output.

5. The method for constructing a 3D map of a construction site by fusing multiple visual sensors according to claim 4, characterized in that, The process of calculating the three-dimensional coordinates of feature points and generating a local sparse point cloud is as follows: Based on the pixel coordinates of the same two-dimensional feature points in the cross-view feature matching set in different view images, and combined with the intrinsic and extrinsic parameters of the corresponding visual sensor, the projection relationship of each view image is constructed. Based on the projection relationship, triangulation is performed on the corresponding two-dimensional feature points in the cross-view feature matching set to solve their initial three-dimensional coordinates under a unified spatiotemporal reference. Accuracy is determined based on the reprojection error of the initial 3D coordinates on images from different viewpoints. Combined with the consistency test results of the reprojection points and the original matching feature points under each viewpoint, abnormal 3D points with inconsistent matching are eliminated to obtain an effective set of 3D feature points. The effective three-dimensional feature point set is converged and fused according to its spatial location to form a local sparse point cloud.

6. The method for constructing a 3D map of a construction site by fusing multiple visual sensors according to claim 3, characterized in that, The process of generating a dense depth map and back-projecting it to obtain a local dense point cloud is as follows: Multi-view images that are time-synchronized and have overlapping fields of view are selected from the image sequence as a dense reconstruction image group; Based on the intrinsic and extrinsic parameters of each visual sensor and the epipolar constraints between images, pixel-level stereo matching is performed on densely reconstructed image groups to obtain pixel correspondence. The disparity information of each pixel is calculated based on the pixel correspondence, and combined with the imaging parameters to convert it into depth information, thereby generating an initial dense depth map corresponding to the dense reconstructed image group. By combining the intrinsic and extrinsic parameters of the corresponding visual sensor, the depth information of each pixel in the initial dense depth map is back-projected to a three-dimensional space under a unified spatiotemporal reference, and the three-dimensional spatial coordinates corresponding to each pixel are calculated. All three-dimensional spatial coordinates are converged to form a locally dense point cloud.

7. The method for constructing a 3D map of a construction site by fusing multiple visual sensors according to claim 1, characterized in that, The process of registering and fusing local sparse point clouds and local dense point clouds is as follows: Acquire local sparse point clouds and local dense point clouds corresponding to the same monitoring area, and determine the initial registration position and attitude of the local dense point cloud based on the spatial distribution of the local sparse point cloud. Based on the overlapping area between the local sparse point cloud and the local dense point cloud, the coarse registration result between the two is obtained; Based on the coarse registration results, the distance between corresponding points, normal constraints and local geometric features between point clouds are iteratively optimized to obtain fine registration results between locally sparse point clouds and locally dense point clouds. Based on the fine registration results, the coordinates of the locally dense point cloud and the locally sparse point cloud are unified, and the two are fused according to the preset fusion rules to generate a fused point cloud.

8. The method for constructing a 3D map of a construction site by fusing multiple visual sensors according to claim 7, characterized in that, The preset fusion rules are as follows: Within the overlapping area of ​​a local sparse point cloud and a local dense point cloud, the distance between corresponding points is calculated according to the spatial correspondence after registration. If the distance between points is less than a preset distance threshold, the point data in the local dense point cloud is retained, and the corresponding points are marked as fusion points. Otherwise, the corresponding point is judged as an outlier and removed.

9. The method for constructing a 3D map of a construction site by fusing multiple visual sensors according to claim 1, characterized in that, The process of combining ground control points to transform the fused point cloud to an absolute coordinate system to obtain the point cloud model of the construction site is as follows: Obtain the coordinates of several ground control points at the construction site in the absolute coordinate system, and extract the coordinates of feature points corresponding to the ground control points in the local coordinate system corresponding to the fused point cloud. Based on the coordinates of ground control points and the corresponding feature points, the coordinate transformation parameters from the local coordinate system to the absolute coordinate system are solved. Based on the coordinate transformation parameters, perform coordinate transformation on all 3D points in the fused point cloud and convert them to the absolute coordinate system. All the transformed 3D points are organized according to the original spatial topology to form a construction site point cloud model unified in an absolute coordinate system.

10. The method for constructing a 3D map of a construction site by fusing multiple visual sensors according to claim 1, characterized in that, The process of semantically recognizing and classifying construction elements and associating them with building information data is as follows: Obtain a 3D mesh model and perform semantic segmentation on its geometric units to identify and classify various construction elements within the construction area; Assign semantic tags to various construction elements, establish the correspondence between various construction elements and corresponding components in the building information model, and realize the association and binding between the three-dimensional mesh model and building information data; Based on the subsequently acquired image sequence, the associated construction elements are subjected to state change detection, their geometric attributes, location information and construction progress attributes are updated, and the three-dimensional digital map data of the construction site is output.

Images