A vision-based method, system, device, and medium for integrity enhancement of a three-dimensional model of a building

Abstract

The application discloses a kind of based on vision's building three-dimensional model integrity enhancement method, system, equipment and medium, the method is first to the original aerial survey image data obtained by unmanned aerial vehicle aerial survey processing, obtain the camera internal parameter and sparse point cloud of building scene, then based on camera internal parameter and sparse point cloud generates the dense point cloud of building scene, and the dense point cloud is grid processing, extracts the target plane of building, then identify the defect area in target plane, generate the optimal supplementary shooting viewpoint covering defect area, according to supplementary shooting viewpoint path planning is generated, generates optimal supplementary shooting path, finally controls unmanned aerial vehicle along optimal supplementary shooting path to defect area is image supplementary shooting, obtains supplementary shooting image data, after merging supplementary shooting image data with original aerial survey image data, the reconstruction of building three-dimensional model is carried out, and the integrity of enhanced building three-dimensional model is obtained.Using the present application can improve the precision and integrity of building three-dimensional model.

Description

Technical Field

[0001] This invention relates to the field of 3D model construction technology, and in particular to a vision-based method, system, device, and medium for enhancing the integrity of 3D building models. Background Technology

[0002] In 3D modeling of buildings, UAV aerial surveying has become the mainstream technology for generating 3D models due to its advantages such as flexibility, low cost, and wide coverage. However, due to limitations in UAV flight altitude and angle, as well as the complexity of building structures, such as varying facades and protruding components, the generated 3D models often have integrity defects. This is especially true in key areas such as building facades, where data loss and blurred details are common, affecting the model's representation of the real form and reducing its application value. Therefore, in the current UAV-based modeling workflow, after generating 3D point clouds, it is necessary to re-capture defective areas to enhance the integrity of the building model.

[0003] Existing methods for supplementary photography involve manual low-altitude shooting or circling flight missions after the aerial survey is completed. For the initial building model generated by the aerial survey, operators visually inspect the building facade to determine if there are any areas with substandard data. They then manually operate the drone to conduct localized supplementary photography of these areas, or set up a circling flight path to perform supplementary photography. This method relies heavily on human experience to determine the viewpoint, resulting in low efficiency, excessive time and manpower consumption, and difficulty in ensuring effective model integrity restoration. It often leads to missing or inaccurate information even after supplementary photography. Summary of the Invention

[0004] The purpose of this invention is to provide a vision-based method, system, device, and medium for enhancing the integrity of 3D building models, thereby improving the accuracy and integrity of 3D building models.

[0005] To achieve the above objectives, the present invention provides a vision-based method for enhancing the integrity of 3D building models, comprising: The raw aerial survey image data acquired by UAV aerial survey is processed to obtain camera intrinsic parameters and sparse point clouds of building scenes; A dense point cloud of the building scene is generated based on the camera intrinsic parameters and the sparse point cloud; The dense point cloud is meshed to extract the target plane of the building; Identify the defective region in the target plane, generate the optimal reshoot viewpoint covering the defective region, and perform path planning based on the reshoot viewpoint to generate the optimal reshoot path; The UAV is controlled to take supplementary images of the defective area along the optimal supplementary shooting path to obtain supplementary image data; the supplementary image data is merged with the original aerial survey image data, and the three-dimensional model of the building is reconstructed based on the merged data to obtain a three-dimensional model of the building with enhanced integrity.

[0006] Optionally, identifying the defect area in the target plane, generating an optimal reshooting viewpoint covering the defect area, and performing path planning based on the reshooting viewpoint to generate an optimal reshooting path includes: The area-weighted centroid of each target plane is projected onto a two-dimensional polygon, and the target plane whose projection point is located within the region of interest of the two-dimensional polygon is selected to obtain the first target plane; Calculate the image coverage of the first target plane; Select a target plane whose image coverage is lower than a preset image coverage threshold to obtain a second target plane; Based on the second target plane, an effective area mask and a camera detection mark coverage area mask are obtained. A defect mask is obtained based on the difference between the effective area mask and the coverage area mask. A defect area is identified based on the defect mask to obtain a defect plane. For each of the defect planes, generate candidate reshoot viewpoints covering the defect area; Based on the candidate reshoot viewpoints, the optimal reshoot viewpoint is selected using a multi-objective optimization algorithm; Based on the optimal reshoot viewpoint, a sampling-based path planning algorithm is used to plan the path and generate the optimal reshoot path.

[0007] Optionally, calculating the image coverage of the first target plane includes: For each of the first target plane, sampling points are uniformly sampled according to area; For each sampling point, perform extrinsic parameter transformation and intrinsic parameter projection on each camera in sequence to obtain the image pixel coordinates. If the image pixel coordinates are within the effective range of the image and are not occluded by other objects, then the sampling point is determined to be covered. The image coverage of the first target plane is obtained by summing the coverage ratios of all covered sampling points according to area weight.

[0008] Optionally, generating candidate reshoot viewpoints covering the defect area for each of the defect planes includes: For each defect plane, the centroid of each cluster of representative surfaces in the defect plane is selected as a sampling point, the normal vector of the representative surface is used as the center direction, and the observation distance is determined by combining the image size and camera parameters. Multiple reshoot viewpoints are generated in the center direction and in the conical direction around the center direction with a fixed pitch angle. Invalid viewpoints are removed based on visibility constraints, and candidate reshoot viewpoints for the coverage defect area are obtained.

[0009] Optionally, the processing of the raw aerial survey image data acquired by the UAV to obtain camera intrinsic parameters and sparse point clouds of the building scene includes: In the raw aerial survey image data obtained by UAV aerial survey, a set of multi-view images with an overlap rate greater than or equal to a preset overlap rate threshold are obtained; Extract local feature points from each image and establish the correspondence between feature points of different images; Based on the correspondence of the feature points, the camera intrinsic parameters of the building scene are solved and a sparse point cloud of the building scene is generated.

[0010] Optionally, generating a dense point cloud of the building scene based on the camera intrinsic parameters and the sparse point cloud includes: Each image is processed to remove distortion based on the camera intrinsic parameters, generating a distortion-free image. Based on the distortion-free images, a relative depth map for each image is generated using a depth estimation model; Based on the sparse point cloud, the relative depth in the relative depth map is converted into absolute depth using the least squares method to generate an absolute depth map for each image. The absolute depth maps of each image are fused to generate a dense point cloud of the building scene.

[0011] Optionally, the step of meshing the dense point cloud to extract the target plane of the building scene includes: Based on the dense point cloud, a surface reconstruction algorithm is used to generate a watertight mesh model; The watertight mesh model is simplified by iteratively shrinking the vertices using a quadratic error metric. Region growing is performed based on a simplified watertight mesh model to extract the target plane of the building scene.

[0012] To achieve the above objectives, the present invention also provides a vision-based system for enhancing the integrity of 3D building models, comprising: The motion structure recovery module is used to process the raw aerial survey image data acquired by UAV aerial surveying to obtain camera intrinsic parameters and sparse point clouds of building scenes; A dense point cloud construction module is used to generate a dense point cloud of a building scene based on the camera intrinsic parameters and the sparse point cloud. The planar segmentation module is used to perform meshing processing on the dense point cloud and extract the target plane of the building. The viewpoint generation and path planning module is used to identify defect areas in the target plane, generate the optimal reshoot viewpoint covering the defect area, and perform path planning based on the reshoot viewpoint to generate the optimal reshoot path. The complete model reconstruction module is used to control the UAV to take supplementary images of the defective area along the optimal supplementary shooting path to obtain supplementary image data; the supplementary image data is merged with the original aerial survey image data, and the three-dimensional model of the building is reconstructed based on the merged data to obtain a three-dimensional model of the building with enhanced integrity.

[0013] To achieve the above objectives, the present invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the vision-based building 3D model integrity enhancement method as described above.

[0014] To achieve the above objectives, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the vision-based building 3D model integrity enhancement method as described above.

[0015] Compared with existing technologies, this invention provides a vision-based method, system, device, and medium for enhancing the integrity of 3D building models. The method first uniformly samples the area of ​​the building plane based on dense point clouds and planar segmentation results. It then accurately calculates image coverage using camera intrinsic and extrinsic parameters and occlusion detection to automatically filter defect areas below a threshold. Next, using the centroid of the defect area patch as the sampling point and its normal vector as the center direction, candidate reshoot viewpoints are generated based on image size and camera parameters. A multi-objective optimization algorithm is then used to select the optimal reshoot viewpoint that efficiently covers all defect areas. Finally, the optimal reshoot viewpoint is input into a sampling-based path planning algorithm to obtain the optimal UAV reshoot flight path. This significantly improves the defect detection accuracy and reshoot efficiency of 3D building models, reduces secondary reshoots and data redundancy, and ensures the integrity and accuracy of the 3D model construction. Attached Figure Description

[0016] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1This is a flowchart of a vision-based method for enhancing the integrity of a 3D building model, provided in an embodiment of the present invention. Figure 2 This is another flowchart of a vision-based method for enhancing the integrity of a 3D building model provided in this embodiment of the invention; Figure 3 This is a structural block diagram of a vision-based 3D building model integrity enhancement system provided in an embodiment of the present invention; Figure 4 This is a structural block diagram of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] See Figure 1 , Figure 1 This is a flowchart of a vision-based method for enhancing the integrity of a 3D building model, provided by an embodiment of the present invention. The vision-based method for enhancing the integrity of a 3D building model includes steps S1 to S5: S1. Process the raw aerial survey image data acquired by UAV aerial survey to obtain camera intrinsic parameters and sparse point cloud of building scene; For example, from a series of disordered or ordered two-dimensional image sequences, without relying on prior information of camera intrinsic and extrinsic parameters, the camera's attitude parameters and intrinsic parameters are solved in reverse by analyzing the correspondence and motion trajectory between images, and the three-dimensional sparse point cloud of the scene is reconstructed simultaneously.

[0020] In one alternative embodiment, S1 includes: S101. From the original aerial survey image data acquired by UAV aerial survey, acquire a set of multi-view images with an overlap rate greater than or equal to a preset overlap rate threshold; preferably, the preset overlap rate threshold is 60%; S102. Extract local feature points from each image and establish the correspondence between feature points of different images; S103. Solve the camera intrinsic parameters of the building scene and generate a sparse point cloud of the building scene based on the correspondence of the feature points.

[0021] S2. Generate a dense point cloud of the building scene based on the camera intrinsic parameters and the sparse point cloud; For example, based on the camera intrinsic and extrinsic parameters, image sequence and sparse point cloud output in step S1, the matching search space is limited by epipolar geometric constraints, reliable matches are selected by combining photometric and geometric consistency criteria, dense depth information of the scene is generated by using a depth estimation algorithm, and a dense point cloud containing detailed features is constructed and output after 3D processing.

[0022] In one alternative embodiment, S2 includes: S201. Perform distortion correction processing on each image according to the camera intrinsic parameters to generate a distortion-free image; S202. Based on the distortion-free image, a relative depth map for each image is generated using a depth estimation model; S203. Based on the sparse point cloud, the relative depth in the relative depth map is converted into absolute depth using the least squares method to generate an absolute depth map for each image. S204. The absolute depth maps of each image are fused to generate a dense point cloud of the building scene.

[0023] S3. Perform meshing on the dense point cloud and extract the target plane of the building scene; For example, a watertight mesh model is generated based on a dense point cloud, and the mesh model is simplified by iteratively shrinking vertices based on a quadratic error metric. The simplified mesh model is then segmented to identify and remove non-target or uninteresting regions, thereby focusing on the target reconstruction region to achieve planar segmentation.

[0024] In one alternative embodiment, S3 includes: S301. Based on the dense point cloud, a surface reconstruction algorithm is used to generate a watertight mesh model; S302. The watertight mesh model is simplified by iteratively shrinking the vertices through a quadratic error metric. S303. Based on the simplified watertight mesh model, perform region growing to extract the target plane of the building scene.

[0025] S4. Identify the defect area in the target plane, generate the optimal reshoot viewpoint covering the defect area, and perform path planning based on the reshoot viewpoint to generate the optimal reshoot path; For example, for the segmented planar target area, the coverage of its surface with all disordered or ordered two-dimensional image sequences is calculated to find defect areas. Based on the camera intrinsic parameters and the geometric information of the defect areas, a series of candidate reshoot viewpoints are generated for the defect areas. The optimal reshoot viewpoint that can cover the largest defect area with the fewest viewpoints is selected. The optimal reshoot viewpoint is used as the target point, and the path planning method is used to solve the optimal flight path of the UAV.

[0026] In one alternative embodiment, S4 includes: S401. Project the area-weighted centroid of each target plane onto a two-dimensional polygon, select the target plane whose projection point is located within the region of interest of the two-dimensional polygon, and obtain the first target plane. S402. Calculate the image coverage of the first target plane; S403. Select a target plane whose image coverage is lower than a preset image coverage threshold to obtain a second target plane; S404. Based on the second target plane, obtain an effective area mask and a camera detection mark coverage area mask. Obtain a defect mask based on the difference between the effective area mask and the coverage area mask. Identify the defect area based on the defect mask to obtain a defect plane. S405. Generate candidate reshoot viewpoints covering the defect area for each of the defect planes; S406. Based on the candidate reshoot viewpoints, the optimal reshoot viewpoint is selected using a multi-objective optimization algorithm; S407. Based on the optimal reshoot viewpoint, a sampling path planning algorithm is used to plan the path and generate the optimal reshoot path; preferably, the sampling path planning algorithm is the RRT* algorithm.

[0027] It is worth noting that the embodiments of the present invention use a sampling path planning algorithm to plan the flight path of the drone for the supplementary shooting viewpoints, which enables the drone to reach each supplementary shooting viewpoint in the shortest time and in the safest way to complete image acquisition. Compared with traditional hand-controlled drones or simple circling flight, this improves the efficiency and safety of the supplementary shooting process.

[0028] In one alternative embodiment, S402 includes: S4021. For each of the first target plane, sample points are uniformly collected according to area. S4022. For each sampling point, perform extrinsic parameter transformation and intrinsic parameter projection on each camera in sequence to obtain the image pixel coordinates. If the image pixel coordinates are within the effective range of the image and are not occluded by other objects, then it is determined that the sampling point is covered. S4023. Summarize the coverage ratio of all covered sampling points according to area weight to obtain the image coverage of the first target plane.

[0029] It is worth noting that the embodiments of the present invention identify defect areas by combining camera parameters to calculate the coverage of the plane and the original image. First, the target plane of the building is sampled evenly by area. Then, the sampling points are judged to be covered by steps such as camera extrinsic parameter transformation, intrinsic parameter projection, and occlusion detection. The coverage is obtained by summarizing the area weights. Finally, the area below the threshold is determined as the defect area. This method avoids the subjectivity and error of manual visual judgment and improves the accuracy of defect identification.

[0030] In one alternative embodiment, S405 includes: S4051. For each of the defect planes, the centroid of each cluster of representative surfaces in the defect plane is selected as a sampling point, the normal vector of the representative surface is used as the center direction, and the observation distance is determined by combining the image size and camera parameters. Multiple reshoot viewpoints are generated in the center direction and in the conical direction around the center direction with a fixed pitch angle. S4052. Based on visibility constraints, invalid viewpoints are deleted to obtain candidate reshoot viewpoints for the coverage defect area.

[0031] It is worth noting that, in this embodiment of the invention, for the identified defective area, the centroid of its surface is used as the sampling point, and the observation distance is determined by combining the image size and camera parameters. Candidate viewpoints are generated in its normal direction and surrounding direction. Then, a multi-objective optimization algorithm is used to select the optimal reshoot viewpoint. This can ensure that the viewpoint covers all defective areas at the lowest cost, reduce redundant data and processing burden caused by extra reshoots, and thus improve reshoot efficiency.

[0032] It is worth noting that existing methods fail to quantitatively extract defect areas from 3D models built using UAV aerial survey imagery, relying excessively on visual judgment and human experience. Furthermore, during the re-shooting of defect areas, the reliance on experience to determine the re-shooting viewpoint and angle easily leads to missed shots or significant data redundancy, resulting in secondary re-shooting or repeated data calculations and wasted resources. This invention, however, determines model defect areas by calculating the coverage of the original image over the building plane, achieving quantitative defect detection, which is more scientific than manual visual judgment. For the selected defect areas, candidate re-shooting viewpoints are calculated based on their geometric information, image size, and camera parameters. A multi-objective optimization algorithm selects the optimal re-shooting viewpoint from the candidate viewpoints, which completely covers the defect area and has minimal redundancy. Finally, the algorithm is used to plan the UAV flight path from the optimal viewpoint, maximizing both re-shooting efficiency and safety.

[0033] S5. Control the UAV to take supplementary images of the defective area along the optimal supplementary shooting path to obtain supplementary image data; merge the supplementary image data with the original aerial survey image data, and reconstruct the three-dimensional model of the building based on the merged data to obtain a three-dimensional model of the building with enhanced integrity.

[0034] For example, the drone is controlled to perform image re-capture on the defective area, and the original image and the re-captured image are merged as a new input dataset. The modeling process is repeated to finally generate a complete, high-quality 3D model.

[0035] In summary, this invention addresses the problems of existing UAV aerial surveying methods for identifying defect areas in 3D building models, which rely on human experience, lack scientific optimization of reshoot viewpoints and path planning, and suffer from low reshoot efficiency. It provides a vision-based method for enhancing the integrity of 3D building models. First, based on dense point clouds and planar segmentation results, the building plane is uniformly sampled. Then, combined with camera intrinsic and extrinsic parameters and occlusion detection, the image coverage is accurately calculated to automatically filter defect areas below a threshold. Next, using the centroid of the defect area patch as the sampling point and its normal vector as the center direction, candidate reshoot viewpoints are generated based on image size and camera parameters. A multi-objective optimization algorithm is used to select the optimal reshoot viewpoint that efficiently covers all defect areas. Finally, the optimal reshoot viewpoint is input into a sampling-based path planning algorithm to obtain the optimal UAV reshoot flight path. Compared to existing methods that rely on human visual judgment and experience for reshooting, this invention significantly improves the defect detection accuracy and reshoot efficiency of 3D building models, reduces secondary reshoots and data redundancy, and ensures the integrity and accuracy of the 3D model construction.

[0036] To make the specific implementation process of the embodiments of the present invention clearer to those skilled in the art, further explanation is provided below.

[0037] See Figure 2 , Figure 2 This is another flowchart of a vision-based method for enhancing the integrity of 3D building models provided in this embodiment of the invention. Figure 2 As shown, the vision-based method for enhancing the integrity of 3D building models includes the following steps: The recovery of the moving structure can be broken down into the following steps: Step 1: Image Acquisition: Acquire a set of multi-view images with an overlap rate of 60% or higher to be processed.

[0038] Step 2, Feature Extraction and Matching: Feature extraction algorithms such as SIFT (Scale Invariant Feature Transform) are used to detect and describe local feature points in each image. Then, a feature descriptor matching algorithm is used to establish the correspondence between feature points across different images. The nearest neighbor distance ratio is: (1); In the formula, For the descriptors of feature points in image A, In image B and The most similar and second most similar candidate descriptors This is an empirical threshold (usually 0.8); if it exceeds this threshold, the feature descriptor matching is considered unreliable.

[0039] Step 3, Sparse 3D Reconstruction: Using the feature point correspondence in Step 2, the incremental SFM (Structure of Motion) algorithm is used to solve for the camera's precise pose, intrinsic parameters, and generate a sparse point cloud.

[0040] It should be noted that in the incremental SFM (Structure of Motion) process, the deviation between the pixel position obtained by "projecting" the 3D spatial point (candidate point of the sparse point cloud) back to the image plane using the currently estimated camera parameters (pose, intrinsic parameters) and the pixel position of the actual detected feature point in step 2 is the reprojection error. To make the estimation of camera parameters and sparse point cloud more accurate, the algorithm iteratively optimizes and continuously adjusts the camera pose, intrinsic parameters, and 3D coordinates of the sparse point cloud to minimize the sum of the reprojection errors of all feature points, ultimately obtaining accurate camera pose, intrinsic parameters, and sparse point cloud.

[0041] Among these, the reprojection error is minimized: (2); In the formula, The variables to be optimized include the rotation matrices of all cameras. Translation vector and the coordinates of all three-dimensional points j is the index of a point in three-dimensional space; Let i be the camera pose, and i be the camera index. For projection function, For camera internal parameters, This represents the pixel coordinates of the actual observed feature point of the i-th camera for the j-th 3D empty point.

[0042] Furthermore, the construction of dense point clouds can be broken down into the following steps: Step 4, Image Distortion: Generate a distortion-free image using the estimated camera intrinsic parameters (including distortion).

[0043] Step 5, Depth Estimation: Perform depth estimation on each image using a depth estimation model (such as Depth_Anything_V2) to generate a relative depth map.

[0044] Step 6, Depth Scale Recovery: Solve for the scale factor using the least squares method with SFM sparse point cloud. and offset Convert relative depth in the depth map to absolute depth: (3); In the formula, For relative depth, Where N is the absolute depth, and N is the total number of data points participating in the depth transformation. It is a vector consisting of all relative depth values. It is a vector consisting of all absolute depth values.

[0045] Step 7: Multi-view depth map fusion: Multi-view depth maps are globally fused using geometric consistency constraints to generate a unified and globally consistent dense point cloud. The geometric consistency constraints are as follows: (4); In the formula, For reference depth, For point The computational depth The depth consistency threshold, This is a collection of visible views.

[0046] Furthermore, planar segmentation can be decomposed into the following steps: Step 8, Surface Reconstruction: Based on the dense point cloud, a watertight mesh model is generated using the TSDF (Truncate Signed Distance Function) surface reconstruction algorithm. Among other things, (5); In the formula, voxels The final "truncated signed distance value", As weight, The distance from the voxel to the surface. The cutoff radius is [value].

[0047] Step 9, Mesh Simplification: Based on a quadratic error metric, vertex shrinking is used to reduce model complexity and data volume while maintaining model topology and visual quality. The quadratic error is: (6); In the formula, As vertex The second-order error measure, For the vertex The related set of triangular facets, Let be the quadratic error matrix of the triangular facet.

[0048] Step 10, Model Segmentation: Calculate the flatness of each facet and select a seed face accordingly. Fit an initial plane to its neighborhood. Then, perform region growth based on the criterion that "the distance from the vertex to the current plane is less than or equal to a threshold". Refit the plane after each round of absorption until it is impossible to continue.

[0049] (7); In the formula, To fit the plane, It is a plane normal vector. This is the distance parameter between the plane and the origin. This is the distance threshold. Furthermore, viewpoint generation and path planning can be broken down into the following steps: Step 11, Target Plane Selection: Represent the region of interest of a two-dimensional polygon using the AOI set of vertices arranged in order. Find the area-weighted centroid of each dividing plane. And projected onto Plane ,like This plane is preserved within the AOI: (8); In the formula, If the value is 1, the point is outside the AOI; if the value is 0, the point is inside the AOI.

[0050] Step 12, Image Coverage Calculation: For each selected plane, uniformly sample points on its triangular facets by area; for each sample point, perform "extrinsic parameter transformation → intrinsic parameter projection → in-image determination → occlusion detection" using each camera. If any camera satisfies the condition (sample point is in-image and unoccluded), it is considered covered. Finally, the coverage ratio is summed according to area weights to obtain the image coverage of that plane. (9); In the formula, For a planar set, Area weight.

[0051] Step 13, Defect Region Identification: Based on the image coverage of each plane, select regions with coverage below a threshold. First, establish a local coordinate system on each plane. With the center of mass Using the origin as the reference point, project the triangular facets of this plane onto a two-dimensional plane to obtain the effective area mask. For the center point of the regular grid Point-by-point, the process involves "extrinsic parameter transformation → intrinsic parameter projection → in-image determination → occlusion detection" using each camera. Mesh visible to any camera is marked as overlay, and finally, a defect mask is obtained. The contours are extracted and reconstructed into three-dimensional space as the defect area. (10); In the formula, , , , For the camera rotation matrix, For the camera optical center, For camera focal length, For camera principal point offset, Define the width and height of the image.

[0052] Step 14: Candidate viewpoint generation: For each defect plane, take the centroid of each cluster of representative surfaces as the sampling point P, and use its normal vector as the center direction to obtain the observation distance through image size and camera parameters. In the central direction and its fixed pitch angle Add in the direction of the cone A number of candidate viewpoints are selected, and invalid viewpoints are removed based on visibility constraints to obtain candidate reshoot viewpoints. (11); In the formula, The observation direction vector, The position of the camera's optical center. Indicates surrounding the normal direction The unit observation direction obtained by rotation.

[0053] Step 15, Optimal Viewpoint Selection: A heuristic viewpoint selection method based on a multi-objective optimization algorithm is used, combined with redundancy and reconstructability constraints, to select the optimal reshoot viewpoint. Among these, (12); In the formula, As weight, For the degree of viewpoint coverage, Redundancy between adjacent viewpoints This involves determining the coverage of key parts of the scene.

[0054] Step 16, Path Planning: Using the optimal reshoot viewpoint as the target point, the RRT* (Path Based on Sampling Points) algorithm is employed for path planning to find the optimal flight path for the UAV that minimizes flight time and avoids obstacles. (13); In the formula, The optimal parent node for the new node. This is the set of neighboring path points of the new node. As a cumulative cost, The cost of the edge.

[0055] Furthermore, the complete model reconstruction can be broken down into the following steps: Step 17, Image Recapping: The UAV takes recapped images of the identified defective areas according to the planned flight path to obtain new multi-view image data.

[0056] Step 18, Model Reconstruction: Merge the original image with the newly acquired re-enhanced image as a new input dataset, and repeat steps 2 through 9. By fusing image information from more perspectives, a complete, high-quality 3D model is finally generated.

[0057] Based on the above method items, the present invention provides corresponding system items embodiments.

[0058] See Figure 3 , Figure 3 This is a structural block diagram of a vision-based 3D building model integrity enhancement system provided in an embodiment of the present invention. The vision-based 3D building model integrity enhancement system includes: The motion structure recovery module 21 is used to process the raw aerial survey image data acquired by UAV aerial survey to obtain camera intrinsic parameters and sparse point clouds of the building scene. Dense point cloud construction module 22 is used to generate a dense point cloud of a building scene based on the camera intrinsic parameters and the sparse point cloud; Planar segmentation module 23 is used to perform meshing processing on the dense point cloud and extract the target plane of the building; The viewpoint generation and path planning module 24 is used to identify the defect area in the target plane, generate the optimal reshoot viewpoint covering the defect area, and perform path planning based on the reshoot viewpoint to generate the optimal reshoot path. The complete model reconstruction module 25 is used to control the UAV to take supplementary images of the defective area along the optimal supplementary shooting path to obtain supplementary image data; merge the supplementary image data with the original aerial survey image data, and reconstruct the three-dimensional model of the building based on the merged data to obtain a three-dimensional model of the building with enhanced integrity.

[0059] In one optional embodiment, the motion structure recovery module 21 is configured to: In the raw aerial survey image data obtained by UAV aerial survey, a set of multi-view images with an overlap rate greater than or equal to a preset overlap rate threshold are obtained; Extract local feature points from each image and establish the correspondence between feature points of different images; Based on the correspondence of the feature points, the camera intrinsic parameters of the building scene are solved and a sparse point cloud of the building scene is generated.

[0060] In one alternative embodiment, the dense point cloud construction module 22 is configured to: Each image is processed to remove distortion based on the camera intrinsic parameters, generating a distortion-free image. Based on the distortion-free images, a relative depth map for each image is generated using a depth estimation model; Based on the sparse point cloud, the relative depth in the relative depth map is converted into absolute depth using the least squares method to generate an absolute depth map for each image. The absolute depth maps of each image are fused to generate a dense point cloud of the building scene.

[0061] In one alternative embodiment, the plane segmentation module 23 is configured to: Based on the dense point cloud, a surface reconstruction algorithm is used to generate a watertight mesh model; The watertight mesh model is simplified by iteratively shrinking the vertices using a quadratic error metric. Region growing is performed based on a simplified watertight mesh model to extract the target plane of the building scene.

[0062] In one optional embodiment, the viewpoint generation and path planning module 24 includes: The first target plane filtering unit is used to project the area-weighted centroid of each target plane onto a two-dimensional polygon, select the target plane whose projection point is located within the region of interest of the two-dimensional polygon, and obtain the first target plane. An image coverage calculation unit is used to calculate the image coverage of the first target plane; The second target plane filtering unit is used to select target planes with image coverage lower than a preset image coverage threshold to obtain the second target plane; The defect region identification unit is used to obtain an effective region mask and a camera detection mark coverage region mask based on the second target plane, obtain a defect mask according to the difference between the effective region mask and the coverage region mask, identify the defect region according to the defect mask, and obtain the defect plane. A candidate reshoot viewpoint generation unit is used to generate a candidate reshoot viewpoint covering the defect area for each defect plane. The optimal reshoot viewpoint selection unit is used to select the optimal reshoot viewpoint based on the candidate reshoot viewpoints using a multi-objective optimization algorithm. The reshooting path planning unit is used to plan the path based on the optimal reshooting viewpoint using a sampling path planning algorithm to generate the optimal reshooting path.

[0063] In one alternative embodiment, the image coverage calculation unit is configured to: For each of the first target plane, sampling points are uniformly sampled according to area; For each sampling point, perform extrinsic parameter transformation and intrinsic parameter projection on each camera in sequence to obtain the image pixel coordinates. If the image pixel coordinates are within the effective range of the image and are not occluded by other objects, then the sampling point is determined to be covered. The image coverage of the first target plane is obtained by summing the coverage ratios of all covered sampling points according to area weight.

[0064] In one optional embodiment, the candidate reshoot viewpoint generation unit is configured to: For each defect plane, the centroid of each cluster of representative surfaces in the defect plane is selected as a sampling point, the normal vector of the representative surface is used as the center direction, and the observation distance is determined by combining the image size and camera parameters. Multiple reshoot viewpoints are generated in the center direction and in the conical direction around the center direction with a fixed pitch angle. Invalid viewpoints are removed based on visibility constraints, and candidate reshoot viewpoints for the coverage defect area are obtained.

[0065] It should be noted that the vision-based 3D building model integrity enhancement system provided in this embodiment of the invention is used to execute all the process steps of the vision-based 3D building model integrity enhancement method in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0066] This invention also provides a terminal device, such as... Figure 4 The diagram shown is a structural block diagram of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor 31, a memory 32, and a computer program stored in the memory 32 and configured to be executed by the processor 31. When the processor 31 executes the computer program, it implements the vision-based method for enhancing the integrity of three-dimensional building models as described in any of the above embodiments.

[0067] In addition, embodiments of the present invention also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the vision-based building 3D model integrity enhancement method as described in any of the above embodiments.

[0068] When the processor 31 executes the computer program, it implements the steps in the above-described embodiment of the vision-based 3D building model integrity enhancement method, for example... Figure 1 All steps of the vision-based 3D building model integrity enhancement method shown. Alternatively, when the processor 31 executes the computer program, it implements the functions of each module in the above-described vision-based 3D building model integrity enhancement system embodiment, for example... Figure 3 The functions of each module in the vision-based 3D building model integrity enhancement system are shown.

[0069] Preferably, the computer program can be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor 31 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the terminal device.

[0070] The processor 31 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 31 can be any conventional processor. The processor 31 is the control center of the terminal device, connecting various parts of the terminal device through various interfaces and lines.

[0071] The memory 32 mainly includes a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory 32 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, and a flash card, etc., or the memory 32 can also be other volatile solid-state storage devices.

[0072] It should be noted that the aforementioned terminal devices may include, but are not limited to, processors and memory, as will be understood by those skilled in the art. Figure 4 The structural block diagram shown is merely a structural example of the terminal device described above and does not constitute a limitation on the structure of the terminal device. The terminal device may include more or fewer components than shown, or combine certain components, or use different components.

[0073] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A vision-based method for enhancing the integrity of 3D building models, characterized in that, include: The raw aerial survey image data acquired by UAV aerial survey is processed to obtain camera intrinsic parameters and sparse point clouds of building scenes; A dense point cloud of the building scene is generated based on the camera intrinsic parameters and the sparse point cloud; The dense point cloud is meshed to extract the target plane of the building; Identify the defective region in the target plane, generate the optimal reshoot viewpoint covering the defective region, and perform path planning based on the reshoot viewpoint to generate the optimal reshoot path; The UAV is controlled to take supplementary images of the defective area along the optimal supplementary shooting path to obtain supplementary image data; the supplementary image data is merged with the original aerial survey image data, and the three-dimensional model of the building is reconstructed based on the merged data to obtain a three-dimensional model of the building with enhanced integrity.

2. The vision-based method for enhancing the integrity of 3D building models as described in claim 1, characterized in that, The process of identifying defective regions in the target plane, generating optimal reshoot viewpoints covering the defective regions, and performing path planning based on the reshoot viewpoints to generate optimal reshoot paths includes: The area-weighted centroid of each target plane is projected onto a two-dimensional polygon, and the target plane whose projection point is located within the region of interest of the two-dimensional polygon is selected to obtain the first target plane; Calculate the image coverage of the first target plane; Select a target plane whose image coverage is lower than a preset image coverage threshold to obtain a second target plane; Based on the second target plane, an effective area mask and a camera detection mark coverage area mask are obtained. A defect mask is obtained based on the difference between the effective area mask and the coverage area mask. A defect area is identified based on the defect mask to obtain a defect plane. For each of the defect planes, generate candidate reshoot viewpoints covering the defect area; Based on the candidate reshoot viewpoints, the optimal reshoot viewpoint is selected using a multi-objective optimization algorithm; Based on the optimal reshoot viewpoint, a sampling-based path planning algorithm is used to plan the path and generate the optimal reshoot path.

3. The vision-based method for enhancing the integrity of 3D building models as described in claim 2, characterized in that, The calculation of the image coverage of the first target plane includes: For each of the first target plane, sampling points are uniformly sampled according to area; For each sampling point, perform extrinsic parameter transformation and intrinsic parameter projection on each camera in sequence to obtain the image pixel coordinates. If the image pixel coordinates are within the effective range of the image and are not occluded by other objects, then the sampling point is determined to be covered. The image coverage of the first target plane is obtained by summing the coverage ratios of all covered sampling points according to area weight.

4. The vision-based method for enhancing the integrity of 3D building models as described in claim 2, characterized in that, The step of generating candidate reshoot viewpoints covering the defect area for each defect plane includes: For each defect plane, the centroid of each cluster of representative surfaces in the defect plane is selected as a sampling point, the normal vector of the representative surface is used as the center direction, and the observation distance is determined by combining the image size and camera parameters. Multiple reshoot viewpoints are generated in the center direction and in the conical direction around the center direction with a fixed pitch angle. Invalid viewpoints are removed based on visibility constraints, and candidate reshoot viewpoints for the coverage defect area are obtained.

5. The vision-based method for enhancing the integrity of 3D building models as described in claim 1, characterized in that, The process of processing the raw aerial survey image data acquired by the UAV to obtain camera intrinsic parameters and sparse point clouds of the building scene includes: In the raw aerial survey image data obtained by UAV aerial survey, a set of multi-view images with an overlap rate greater than or equal to a preset overlap rate threshold are obtained; Extract local feature points from each image and establish the correspondence between feature points of different images; Based on the correspondence of the feature points, the camera intrinsic parameters of the building scene are solved and a sparse point cloud of the building scene is generated.

6. The vision-based method for enhancing the integrity of 3D building models as described in claim 1, characterized in that, The generation of a dense point cloud for the building scene based on the camera intrinsic parameters and the sparse point cloud includes: Each image is processed to remove distortion based on the camera intrinsic parameters, generating a distortion-free image. Based on the distortion-free images, a relative depth map for each image is generated using a depth estimation model; Based on the sparse point cloud, the relative depth in the relative depth map is converted into absolute depth using the least squares method to generate an absolute depth map for each image. The absolute depth maps of each image are fused to generate a dense point cloud of the building scene.

7. The vision-based method for enhancing the integrity of 3D building models as described in claim 1, characterized in that, The step of meshing the dense point cloud and extracting the target plane of the building scene includes: Based on the dense point cloud, a surface reconstruction algorithm is used to generate a watertight mesh model; The watertight mesh model is simplified by iteratively shrinking the vertices using a quadratic error metric. Region growing is performed based on a simplified watertight mesh model to extract the target plane of the building scene.

8. A vision-based system for enhancing the integrity of 3D building models, characterized in that, include: The motion structure recovery module is used to process the raw aerial survey image data acquired by UAV aerial surveying to obtain camera intrinsic parameters and sparse point clouds of building scenes; A dense point cloud construction module is used to generate a dense point cloud of a building scene based on the camera intrinsic parameters and the sparse point cloud. The planar segmentation module is used to perform meshing processing on the dense point cloud and extract the target plane of the building. The viewpoint generation and path planning module is used to identify defect areas in the target plane, generate the optimal reshoot viewpoint covering the defect area, and perform path planning based on the reshoot viewpoint to generate the optimal reshoot path. The complete model reconstruction module is used to control the UAV to take supplementary images of the defective area along the optimal supplementary shooting path to obtain supplementary image data; the supplementary image data is merged with the original aerial survey image data, and the three-dimensional model of the building is reconstructed based on the merged data to obtain a three-dimensional model of the building with enhanced integrity.

9. A terminal device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the vision-based method for enhancing the integrity of a three-dimensional building model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the vision-based method for enhancing the integrity of a three-dimensional building model as described in any one of claims 1 to 7.

Images