A method and system for three-dimensional mounting and closed-loop inspection of infrastructure defects
By constructing a 3D model and state diagram in the UAV inspection system, and using dual-link intersection technology and credibility assessment to generate cause codes, the problems of image positioning offset and mount credibility were solved, and efficient closed-loop inspection was achieved.
Patent Information
- Application Number
- CN202611292834.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-25
- Publication Date
- 2026-11-17
AI Technical Summary
Existing drone inspection systems struggle to achieve automatic and accurate 3D positioning of images captured by uninvolved entities, and the mounting results lack reliable quantitative interpretation, making it impossible to form a closed-loop inspection system.
By planning flight routes and acquiring images, a 3D model and state diagram are constructed. A supplementary flight route is generated to acquire close-up images. Combined with aerial triangulation reconstruction and metadata direct ray link, image recognition is performed, spatial intersection is calculated, the reliability of the mount is evaluated, and a cause code is generated to drive feedback actions to update the state diagram.
It achieves high-precision spatial positioning of high-definition images, quantitative self-diagnosis of mounting results, reduces the workload of manual review and forms a closed-loop inspection, thus reducing the field rework rate.
Smart Images

Figure REF-OBJ-1787559452393-000002 
Figure REF-OBJ-1787559452393-000003 
Figure REF-OBJ-1787559452393-000004
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and drone inspection technology, specifically to a method and system for three-dimensional mounting and closed-loop inspection of infrastructure defects. Background Technology
[0002] With the increasing use of drone technology in infrastructure inspection, digital management of structural defects by constructing 3D models and combining them with intelligent recognition algorithms has become a trend. However, existing inspection systems typically separate modules such as flight path planning, 3D modeling, defect identification, spatial mounting, and manual verification.
[0003] For high-resolution disease images acquired at close range, since they are usually not involved in aerial triangulation reconstruction, the system has difficulty in achieving automatic and accurate three-dimensional spatial positioning. If only the rough GPS and attitude data of the UAV are relied upon, serious positioning offsets are likely to occur, and a reliable spatial projection area cannot be obtained.
[0004] Furthermore, the existing automatic disease loading results lack a reliable interpretation mechanism, and the results of manual review cannot achieve data feedback, making it difficult to optimize algorithm parameters or guide the next round of data collection, thus failing to form a closed loop for inspection.
[0005] Therefore, how to achieve accurate three-dimensional positioning of defects in images that were not taken during the reconstruction and how to build a closed-loop inspection mechanism with reliable quantification and verification control are problems that urgently need to be solved in this field. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this invention proposes a three-dimensional mounting and closed-loop inspection method and system for infrastructure defects, which solves the technical problems in existing technologies such as the difficulty in automatically and accurately mounting images that were not taken during construction, the lack of reliable quantitative interpretation of mounting results, and the inability of manual verification to achieve data feedback and form a closed loop.
[0007] The technical solution adopted in this invention is as follows: In the modeling stage, flight routes are planned and images are acquired to construct a 3D model and a state diagram; in the reshooting stage, reshooting flight routes for specific units are generated based on the state diagram and close-range reshooting images are acquired; the modeling and reshooting images are identified to obtain 2D defect results; depending on whether the images containing defect results participate in aerial triangulation reconstruction, an aerial triangulation projection link or a metadata direct ray link is automatically selected, and the 2D results are intersected with the 3D model in a layered hit-backspace intersection to solve for the 3D defect location and measure the spatial hit quality; the mounting credibility is evaluated by combining the defect identification quality, spatial hit quality, and ray geometric parameters, and a cause code is generated to drive feedback actions to update the state diagram, thereby achieving closed-loop inspection.
[0008] In the first feasible approach, a method for three-dimensional mounting and closed-loop inspection of infrastructure defects is provided, including: Modeling images are acquired according to the planned modeling route, and a three-dimensional model of the infrastructure and a surface unit state diagram for recording the state of each surface unit are constructed based on the modeling images. A reshooting route is generated based on the surface unit state diagram, and close-up reshooting images are obtained based on the reshooting route; The modeling image and the close-up re-shot image are used to identify defects, and a two-dimensional defect result including the range of defect pixels and the quality of defect identification is obtained. Depending on whether the image containing the two-dimensional defect results participates in aerial triangulation reconstruction, an aerial triangulation projection link or a metadata direct ray link is selected, and the two-dimensional defect results are spatially intersected with the three-dimensional model to obtain the three-dimensional defect location and record the corresponding spatial hit quality. A multi-dimensional credibility assessment is performed based on the defect identification quality, the spatial hit quality, and the camera ray geometry parameters in spatial intersection, and the mounting credibility score is calculated. Extract the evaluation index sub-score that has the greatest negative contribution to the mounting credibility score to determine the dominant deviation source and generate a cause code to characterize the specific cause of the positioning deviation. The surface unit state diagram is updated based on the cause code that drives the closed-loop feedback action.
[0009] Furthermore, the surface unit state diagram records the modeling coverage, texture quality, occlusion status, disease risk, existing mount credibility, historical review conclusions, and reshoot history of each surface unit. Based on the surface unit state diagram, the reshoot priority of each surface unit is utilized. Identify the target for reshoots:
[0010] In the formula, e is a surface element. For model quality score, This represents the disease risk value. The existing mount credibility score, For manual review of abnormalities or rejections, This refers to historical anomaly frequencies or historical risk values. , , , , These are the corresponding weighting factors for each item.
[0011] Furthermore, when generating the reshoot route, the following multi-objective comprehensive optimization objective function is used. The reshoot route is evaluated and selected from the candidate route set:
[0012] In the formula, As candidate routes, To complete the model's returns, The quality score is based on the identifiable disease. This is for the subsequent automatic mounting quality prediction value. To cover the benefits of high-risk components, For flight risk values, For flight time or energy consumption, to These are the weights of each corresponding item.
[0013] Furthermore, the following steps are used to determine whether the surface unit needs to be re-photographed: When the trigger flag of the surface unit meets the following condition, it is determined that it will enter the generation queue of the reshoot route:
[0014] In the formula, As a trigger flag, To improve the quality of disease identification, To assess mount reliability or predict mount quality, This represents the disease risk value. , , These are the corresponding identification quality threshold, mounting credibility threshold, and disease risk threshold. The logic states that if and only if, This indicates a logical OR.
[0015] Furthermore, depending on whether the image containing the two-dimensional defect results participates in aerial triangulation reconstruction, an aerial triangulation projection link or a metadata direct ray link is selected, including:
[0016] In the formula, I represents the two-dimensional image of the disease result. For the selected mount mode, This is a collection of images that have already been used in aerial triangulation reconstruction. In response to The aforementioned spatial triangulation projection link is selected at that time; In response to The metadata direct ray link is selected at that time.
[0017] Furthermore, by selecting the aerial triangulation projection link, the two-dimensional defect results are spatially intersected with the three-dimensional model to obtain the three-dimensional defect location and spatial hit quality, including: The original camera's calibration intrinsic parameters are scaled proportionally to the actual size of the current inference image, and the scaled camera intrinsic parameters are calculated as follows:
[0018] In the formula, , These are the original calibrated focal lengths. , These are the original principal point coordinates. , The original calibration image width and height, , To infer the actual width and height of the image; Combined with the camera rotation matrix calculated from the photogrammetric attitude angle Construct the empty three-pixel back-projection ray r(t):
[0019]
[0020] In the formula, The normalized unit ray direction. , The coordinates of the diseased pixels. The coordinates of the camera center in the aerial triangulation results. This is the distance parameter.
[0021] Furthermore, by selecting the metadata direct ray link, the two-dimensional disease results are spatially intersected with the three-dimensional model to obtain the three-dimensional disease location and spatial hit quality, including: Calculate the tolerance of the ray angle :
[0022] In the formula, , , These represent the real-time dynamic differential positioning accuracy error, gimbal attitude angle error, and intrinsic parameter uncertainty, respectively. The shooting distance is the distance from the camera to the target surface. , , , The corresponding error term weights; Constructing an uncertainty cone :
[0023] In the formula, As spatial candidate points, These are the camera's starting coordinates in the model coordinate system. The metadata direct ray direction is constructed from the pixel coordinates of the defect, the camera optical axis direction, and intrinsic parameters. This indicates the angle between the direction of the candidate point and the direction of the straight ray.
[0024] Furthermore, a layered hit-back mechanism is adopted to spatially intersect the two-dimensional defect results with the three-dimensional model to obtain the three-dimensional defect location and spatial hit quality, including: During the intersection calculation, the camera ray or the uncertainty cone is intersected with the real triangular mesh of the 3D model; If no match is found, hierarchical hit backoff and intersection are performed sequentially to the component semantic layer, local bounding box, and global bounding box to obtain the final hit level, and the location of the three-dimensional defect is determined based on the final hit level. Based on the final hit level, calculate the model hit quality sub-score in the spatial hit quality. This serves as the quantitative basis for the quality of the space hit:
[0025] In the formula, The base score for hit validity, For the hit level variable, This is the penalty coefficient that varies with the hit level.
[0026] Furthermore, the credibility score is calculated using the following formula:
[0027] In the formula, These are the normalized evaluation sub-scores for each dimension. The weights for the corresponding scoring items; The normalized sub-scores for each dimension include: Metadata integrity sub-score calculated using positioning and attitude parameters recorded in the close-up recaptured images ; Geometric stability sub-fraction calculated based on the angle between the camera's line of sight and the target surface normal. ; The model hit mass fraction determined based on the final hit level. ; The defect identification quality sub-score is calculated based on the confidence level and pixel area output by the defect detection or segmentation algorithm. ; Consistency sub-score calculated based on backprojection discreteness of defect multi-sampling points in 3D space ; The weights corresponding to the above evaluation sub-scores are, and .
[0028] In conjunction with the first feasible approach, a second feasible approach provides a three-dimensional mounting and closed-loop inspection system for infrastructure defects, including: The 3D model and state diagram construction module is used to acquire modeling images according to the planned modeling route, and construct a 3D model of the infrastructure and a surface unit state diagram for recording the state of each surface unit based on the modeling images. The reshooting planning module is used to generate a reshooting route based on the surface unit state diagram, and to acquire close-up reshooting images based on the reshooting route; The two-dimensional defect identification module is used to identify defects in the modeled image and the close-up re-shot image to obtain two-dimensional defect results that include the defect pixel range and defect identification quality. The dual-link intersection mounting module is used to select either the aerial triangulation projection link or the metadata direct ray link based on whether the image containing the two-dimensional disease results participates in aerial triangulation reconstruction, to perform spatial intersection between the two-dimensional disease results and the three-dimensional model, obtain the three-dimensional disease location, and record the corresponding spatial hit quality. The credibility assessment module is used to perform multi-dimensional credibility assessment based on the defect identification quality, the spatial hit quality, and the camera ray geometric parameters in spatial intersection, and to calculate the mounting credibility score. The cause code generation module is used to extract the evaluation index sub-score that has the greatest negative contribution to the mounting credibility score to determine the dominant deviation source and generate a cause code to characterize the specific cause of the positioning deviation. The closed-loop feedback control module is used to drive the closed-loop feedback action according to the cause code and update the surface unit state diagram.
[0029] As can be seen from the above technical solution, the beneficial technical effects of the present invention are as follows: 1. By using dual-link spatial intersection calculation, the problem of automatic high-precision spatial positioning of close-range re-captured images that did not participate in 3D reconstruction was overcome, significantly improving the utilization rate of high-definition local images.
[0030] 2. By using layered hit rollback and multi-factor reliability scoring, the system achieves quantitative self-diagnosis of mounting accuracy and outputs cause codes, avoiding the inclusion of mismounted defects in the repair process and reducing the workload of manual verification.
[0031] 3. By using cause codes and state diagrams to drive closed-loop feedback actions, a closed-loop flow of inspection defects and field-collected parameters was achieved, effectively reducing the field rework rate and enabling adaptive system iteration. Attached Figure Description
[0032] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.
[0033] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention; Figure 2 This is a system structure diagram of Embodiment 2 of the present invention; Figure label: 2-A three-dimensional mounting and closed-loop inspection system for infrastructure defects, 21-Three-dimensional model and state diagram construction module, 22-Reshoot planning module, 23-Two-dimensional defect identification module, 24-Dual-link intersection mounting module, 25-Reliability assessment module, 26-Cause code generation module, 27-Closed-loop feedback control module. Detailed Implementation
[0034] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are merely illustrative of the technical solution of the present invention and are therefore intended to limit the scope of protection of the present invention.
[0035] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.
[0036] Example 1
[0037] This embodiment provides a method for three-dimensional mounting and closed-loop inspection of infrastructure defects. The working principle of Embodiment 1 is explained in detail below: The method flowchart of this embodiment is as follows: Figure 1 As shown, it includes the following steps: Modeling images are acquired according to the planned modeling route, and a three-dimensional model of the infrastructure and a surface unit state diagram for recording the state of each surface unit are constructed based on the modeling images. A reshooting route is generated based on the surface unit state diagram, and close-up reshooting images are obtained based on the reshooting route; The modeling image and the close-up re-shot image are used to identify defects, and a two-dimensional defect result including the range of defect pixels and the quality of defect identification is obtained. Depending on whether the image containing the two-dimensional defect results participates in aerial triangulation reconstruction, an aerial triangulation projection link or a metadata direct ray link is selected, and the two-dimensional defect results are spatially intersected with the three-dimensional model to obtain the three-dimensional defect location and record the corresponding spatial hit quality. A multi-dimensional credibility assessment is performed based on the defect identification quality, the spatial hit quality, and the camera ray geometry parameters in the spatial intersection, and the mounting credibility score is calculated to characterize the credibility of the three-dimensional defect location result. Extract the evaluation index sub-score that has the greatest negative contribution to the mounting credibility score to determine the dominant deviation source and generate a cause code to characterize the specific cause of the positioning deviation. The surface unit state diagram is updated based on the cause code that drives the closed-loop feedback action.
[0038] In this embodiment, further, in the first stage, namely the modeling stage, the system first needs to receive inspection task constraint parameters, which include the inspection space boundary, target component list, restricted flight area constraints, the type of disease to be detected, historical risk distribution area, and existing rough three-dimensional mesh model.
[0039] Next, the system performs modeling and route planning. To obtain image sequences capable of stable, high-quality 3D reconstruction, the system employs a joint planning process: candidate viewpoint generation, candidate route combination, multi-objective scoring, constraint filtering, and path optimization. Candidate viewpoint generation: Based on the 3D bounding box extent of the infrastructure, the surface normal vector of the components, and the set minimum safety distance, candidate shooting viewpoints covering all exposed surfaces of the infrastructure are generated.
[0040] Candidate route combination: Based on the preset heading overlap rate, lateral overlap rate, camera viewing angle, and expected ground sampling distance (GSD), adjacent candidate viewpoints are topologically connected and combined into multiple candidate routes.
[0041] Multi-objective scoring and constraint filtering: Constructing a multi-objective comprehensive optimization objective function Each candidate route P is quantitatively scored, and candidate routes that do not meet the requirements of obstacle avoidance boundary, minimum safe distance, battery power limit, gimbal reach angle range and overlap rate lower limit are eliminated. Finally, the first-stage modeling route with the highest comprehensive score or that meets the preset threshold is output.
[0042] Among them, the multi-objective integrated optimization objective function used in route selection is The formula is as follows:
[0043] In the formula, Let J(P) be the candidate route, and J(P) be the comprehensive optimization objective value of the candidate route. This means selecting the route that maximizes the objective function value from the set of candidate routes. To complete the model's returns, The quality score is based on the identifiable disease. This is for the subsequent automatic mounting quality prediction value. To cover the benefits of high-risk components, T(P) represents the flight risk value, and T(P) represents the flight time or energy consumption. to These are the weights for each corresponding optimization objective. By adjusting the weight values, a balance can be achieved between model reconstruction quality, disease clarity, and flight cost.
[0044] Model completion benefit in the above objective function The following formula is used for refined calculation:
[0045] In the formula, k is the global number of the component or surface element. The importance weight of the k-th component or surface element. The candidate route P is expected to add or be visible to the k-th cell in terms of projected surface area. This represents the effective coverage area threshold that the k-th unit needs to achieve. The operator is used to limit the coverage gain of a single cell to 1 in order to avoid excessive gain amplification caused by repeated shooting of a single area, and to ensure that the flight path can prioritize the coverage of data blind spots that have not yet been fully modeled.
[0046] Meanwhile, the subsequent automatic mount quality prediction value in the objective function Prior estimation is performed using the following formula:
[0047] In the formula, The RTK positioning quality score is determined offline based on the satellite search status and the differential status of the base station. The image metadata integrity score is denoted by , where n is the normal vector of the target surface unit. Let be the camera's line-of-sight vector. This indicates the viewing direction of the camera pointing towards the target surface. Let be the cosine of the angle between the surface normal and the viewing direction, and D be the expected shooting distance from the camera to the target surface. This represents the maximum allowed mounting and shooting distance of the system. The operator is used to truncate and zero out negative values in the calculation results, such as when the shooting angle is greater than 90 degrees or the distance exceeds the limit. This formula takes into account the impact of RTK positioning accuracy, shooting incident angle, and shooting distance on the geometric stability of the subsequent 3D spatial projection during the flight path generation stage.
[0048] To ensure the physical safety of flight routes, the system forcibly introduces flight safety and executable constraint formulas in multi-objective optimization:
[0049] In the formula, Let j be the coordinates of the j-th waypoint. For the m-th obstacle or no-fly zone boundary, The Euclidean distance from the waypoint to the obstacle. B(P) represents the preset minimum safe distance limit, and B(P) represents the estimated power consumption of the route P. To limit the maximum safe discharge capacity of drones, and These represent the actual heading and the lateral overlap rate, respectively. and These are the corresponding safety lower limits for overlap rates. In this embodiment, the preferred values are... , , Let the gimbal tilt angle be the angle at the j-th waypoint. and These are the physical limits of the gimbal's allowable pitch and elevation angles, respectively.
[0050] The UAV executes the modeling flight path and acquires a sequence of modeling images. The system performs aerial triangulation reconstruction and dense mesh reconstruction on the modeling image sequence to generate a 3D infrastructure model with 3D extrinsic parameters (such as a tile model in 3DTiles or b3dm format).
[0051] While constructing the 3D model, the system divides the infrastructure into several surface units according to geometric components, spatial grids, or semantic regions, and establishes a surface unit state diagram to dynamically track and record the multidimensional observations and quality characteristics of each surface unit, including modeling coverage, texture quality, occlusion / missing surface status, potential defects risk, existing mount credibility score, manual historical review conclusions, and reshoot history information.
[0052] In this embodiment, further, after the state graph is constructed, the inspection task enters the second stage, namely the reshooting stage. In order to obtain high-quality detailed imaging of surface units with low reliability due to low texture, occlusion, or poor shooting angle, the system generates a reshooting route for close-up shooting based on the quality deficiencies in the state graph.
[0053] First, the system calculates the trigger flag. The system automatically identifies and filters target surface units x that require close-up reshooting. A surface unit x is included in the reshooting candidate queue only if it meets the following conditions:
[0054] In the formula, This indicates that surface element x meets the retake trigger condition. This indicates the logical "if and only if", To determine the confidence quality of the two-dimensional defect image for the current surface unit. This refers to the existing defect mounting confidence score for this unit or the mounting quality predicted based on physical location. This represents the disease risk value. , , These are the preset identification quality threshold, the lower limit threshold for mounting credibility and security, and the high-risk threshold. Represents the logical OR operator.
[0055] For each surface unit e that has been selected into the replacement candidate queue, the system reads its multidimensional feature attributes from the state diagram and calculates the replacement priority. And, in order of priority from high to low, waypoints and attitude planning are carried out in sequence:
[0056] In the formula, This is the model quality score; a lower value indicates more severe surface defects, low texture, or occlusion. This represents the historical probability of damage or historical defect risk for the surface unit. The existing mount credibility score, For manual review of abnormalities or historical rejections, This refers to the number of times the component area has been re-photographed or the frequency of missed inspections. to These are the positive weighting factors corresponding to each quality dimension.
[0057] The mapping relationship between the unit quality state in the state diagram and the reshoot route planning action is shown in Table 1: Table 1. Mapping Relationship between Unit Quality Status and Supplementary Flight Route Planning Actions
[0058] When planning specific waypoints and camera attitudes for reshoots, the system forcibly introduces ground sampling distance (GSD) constraints and mounted viewing angle constraints to ensure that the reshoot photos have sufficient spatial resolution and ideal projection angles.
[0059] The formula for ground sampling distance constraint is:
[0060] In the formula, Where is the actual ground sampling distance corresponding to the target surface, D is the measured shooting distance from the camera center to the target component surface, p is the physical pixel size of the camera sensor, and f is the physical focal length of the camera. In this embodiment, the maximum allowed sampling resolution is preferably defined as the upper limit of the system's maximum sampling resolution. This constraint ensures that small-scale defects such as microcracks and spalling can be clearly imaged and measured in the image.
[0061] The formula for constraining the mounting viewpoint is:
[0062] In the formula, The angle between the camera's optical axis (viewing direction) and the unit normal vector of the target surface is the angle of incidence, where n is the normal vector of the target surface. For the camera's observation direction, The maximum allowable viewing angle deviation is preferably 45°. This constraint avoids spatial projection distortion and inaccurate estimation of the lesion area caused by oblique perspective or large-angle tilt shooting.
[0063] The drone executes the planned reshoot route to acquire high-resolution close-up reshoot images.
[0064] In this embodiment, the system further inputs both the modeled image and the close-up re-enhanced image into the disease identification module. The disease identification module uses a deep learning object detection model or instance segmentation model (YOLO series, Mask R-CNN, or a segmentation network with an attention mechanism) to perform inference calculations on each input image and outputs a two-dimensional disease identification result. The two-dimensional disease result includes: the pixel category of the disease (including cracks, leaks, damage, exposed reinforcement, etc.), the identification confidence score (conf), the bounding box (bbox), the pixel-level segmentation mask, and the pixel area occupied in the image coordinate system. .
[0065] In this embodiment, further, after obtaining the two-dimensional disease results, since there are two types of images—modeled images participating in reconstruction and close-up re-captured images not participating in reconstruction—the system executes a dual-link automatic mounting algorithm. The system first checks whether the image I currently containing the disease results belongs to the aerial triangulation result set. And perform the following automatic link selection:
[0066] When the defect results exist in the modeling image that has been used for aerial triangulation reconstruction, the system prioritizes reading the camera center coordinates (external parameters) obtained from photogrammetry calculations, as well as the calibration intrinsic parameters.
[0067] To eliminate pixel back-projection bias that may be caused by image scaling during the inference stage, the original camera's calibration intrinsic parameters are first scaled proportionally:
[0068] In the formula, , These are the camera's original physical focal lengths in the x and y directions, respectively. , These are the original pixel coordinates of the camera principal point, respectively. , The physical width and height of the original calibration image. , The actual width and height of the image fed into the deep learning model for inference. , , , This is the scaled-down camera intrinsic parameter.
[0069] Next, the rotation matrix from the camera to the 3D model coordinate system, calculated from the photogrammetric extrinsic parameters, is used. Construct a three-pixel back-projection space ray passing through the center point of the defective pixel. :
[0070]
[0071] In the formula, d is the normalized unit direction vector of the spatial ray in the model coordinate system. , This refers to the geometric center coordinates or sampled pixel coordinates of the two-dimensional disease pixel-level mask in the current image. The three-dimensional spatial position of the camera center obtained through photogrammetry calculation. For distance parameters, This refers to the spatial ray in the aerial triangulation projection.
[0072] In this embodiment, further, when the defect results exist in close-up re-shot images that were not included in the construction, the system directly reads high-precision GPS / RTK latitude and longitude data, gimbal / camera three-axis attitude angles (Yaw, Pitch, Roll), and camera factory focal length parameters from the metadata (Exif / XMP information) of the image.
[0073] First, the system calls the conversion formula from WGS84 to the Earth-centered Earth-fixed (ECEF) coordinate system to convert the WGS84 latitude, longitude, and altitude in the metadata into ECEF spatial coordinates:
[0074]
[0075] In the formula, Latitude Where is longitude, and h is the ellipsoidal elevation measured using high-precision RTK. Let be the semi-major axis of the WGS84 ellipsoid, with a value of . , where e is the first eccentricity, with a value of 0.00669437999. The radius of curvature of the current zonal circle is denoted as . These are the transformed geocentric and geofixed coordinates.
[0076] Subsequently, the inverse of the spatial transformation matrix of the 3D Tiles root node is used. Transform the ECEF coordinate projection to the local engineering coordinate system of the 3D model, and calculate the camera origin coordinates in the local space of the model. :
[0077] In the formula, The local reference origin defined for the model project. Represents a column vector. This represents the first three three-dimensional Cartesian coordinate components of the result of multiplying homogeneous matrices.
[0078] Based on the above camera starting coordinates By combining the camera optical axis attitude direction read from the camera metadata and the sensor intrinsic parameters, a metadata pixel straight ray is constructed. :
[0079]
[0080] In the formula, Let be the unit direction vector of the camera's optical axis. and These are the rightward and downward unit vectors defined in the local model space under the current camera pose, respectively. , The coordinates of the diseased pixels. , , , These are the principal point and focal length intrinsic parameters, respectively.
[0081] Because the supplementary images not used in the construction project have not undergone bundle adjustment correction, their RTK positioning and gimbal attitude both have certain physical errors, making it extremely easy for a single straight ray to miss the model mesh. Therefore, the system utilizes metadata positioning and attitude errors to calculate the ray angle tolerance. :
[0082] In the formula, , , These represent the standard deviation of RTK positioning, the standard deviation of gimbal attitude angle measurement error, and the uncertainty of the camera's factory calibration intrinsic parameters, respectively. D is the distance from the camera to the target unit. to These are the corresponding transformation weight coefficients.
[0083] Based on angle tolerance Construct a straight ray around the metadata pixel Expanded spatial uncertainty cone :
[0084] In the formula, X represents a spatial candidate point within the uncertainty cone. The starting point of the camera in model space. The center direction vector of the straight ray. Calculate the angle between the direction vectors.
[0085] In this embodiment, after constructing the space camera ray (for modeling images) or the uncertainty cone (for re-captured images), the system performs physical intersection calculations with the 3D model. A layered hit-and-back-off mechanism is used during intersection calculations. Real Mesh Hit: Prioritize parsing the real triangular mesh in the model 'b3dm / GLB' file, and perform ray triangulation calculations. Ray With triangular facets The intersection condition is:
[0086] In the formula, To find the spatial intersection point, , These are the centroid coordinates of the point of impact within the triangle. Satisfying the above boundary constraints indicates that the ray has successfully intersected the actual mesh surface.
[0087] Component semantic layer hit: If the intersection of layer 1 is broken or there is no intersection, then backtrack and perform intersection projection calculation with the semantic surface of the component pointed to by the current viewpoint.
[0088] Local bounding box hit: If the semantic face is unavailable, backtrack and perform AABB intersection calculation with the local tile bounding box that contains the region.
[0089] Global bounding box hit: If they still do not intersect, then perform a fast intersection with the global bounding box of the entire 3D infrastructure model.
[0090] The correspondence between the feature attributes of the hierarchical hit fallback mechanism and the output mechanism is shown in Table 2:
[0091] Find the intersection point in the obtained space. Based on this, the system uses a multi-sampling point fusion algorithm to eliminate jitter caused by uneven image feature points or edge noise, and obtains the final three-dimensional location of the lesion after fusion. :
[0092] The restrictions are: In the formula, This is the set of pixels sampled from the boundary and interior of a two-dimensional disease segmentation mask. Let be the pixel coordinates of the l-th sampling point. To obtain the spatial hit point by back-projection intersection of the sampling point, The spatial geometric midpoint of all intersecting points. An outlier removal threshold is set. This algorithm automatically removes outliers that penetrate the broken surface of the model and takes the spatial median as the optimal three-dimensional location of the defect.
[0093] The approximate projected area A of the lesion in the real physical space is estimated by performing physical scale transformation and integration on all sampled pixels within the segmentation mask:
[0094] In the formula, Where is the actual total number of pixels in the 2D segmentation mask for the disease, and D is the measured shooting distance. , The formula, which represents the camera's scaling intrinsic focal length in the corresponding direction, provides a reliable algorithm for rapid quantitative estimation of disease area using the current imaging distance when the model lacks a high-resolution surface unfolding.
[0095] Ultimately, the specific level that was hit during the physical intersection calculation will be determined. Calculate and record the corresponding model hit mass fractions according to the following formula. :
[0096] In the formula, The base score for hit validity is calculated when there are no valid three-dimensional intersections. Otherwise, it is 1. The penalty coefficient () varies with the hit level if and only if the hit level is the real mesh, the component semantic layer, the local bounding box, or the global bounding box. The corresponding values are 1.0, 0.8, 0.6 and 0.3 respectively.
[0097] To avoid statistical bias caused by the same disease being repeatedly attached to multiple adjacent modeling or re-captured images, the system introduces an IoU determination formula for duplicate diseases within the same image / across images when performing attachment:
[0098] In the formula, and These are the bounding boxes for the two identified diseases on the image. and These are the intersection area and the union area of the two bounding boxes, respectively. and The three-dimensional spatial locations of the two defects were calculated. Preset Merging threshold, In this embodiment, the preferred spatial physical merging distance threshold is 0.1m. If the following conditions are met... Then, a merge operation is performed. The system will merge two candidate targets into the same physical defect if and only if the overlap of the two-dimensional bounding boxes and the physical distance in the three-dimensional space simultaneously meet the threshold.
[0099] In this embodiment, further, after completing the spatial mounting, the system initiates a multi-dimensional credibility assessment, calculating a mounting credibility score S for each generated three-dimensional disease mounting result:
[0100] In the formula, S is the overall mounting credibility score (ranging from 0 to 100). To normalize the evaluation sub-scores to the range [0,1], The weights are assigned to each sub-score, and the sum of all weights is 1.
[0101] The normalized evaluation sub-score Specifically, it includes the following five dimensions: Metadata integrity sub-score Scoring is based on the number of RTK positioning satellites, the differential screen lock status of the RTK base station, and the completeness of attitude angle measurement in the metadata of the re-shot image.
[0102] Geometric stability sub-fraction :
[0103] In the formula, D is the measured shooting distance. To determine the maximum permissible mounting and shooting distance, n is the surface unit normal vector at the intersection point, and d is the pixel optical axis projection direction. Let cosine be the value of the incident angle. This is an adjustment coefficient that balances the weights of distance and angle of incidence, and its value ranges from 0 to 1.
[0104] Model hit mass fraction The above calculations are derived from the hierarchical hit rollback. .
[0105] Defect identification quality sub-score :
[0106] In the formula, The recognition confidence score output by the object detection / instance segmentation model. This represents the segmented pixel area of the disease on a two-dimensional image. To support the minimum pixel area threshold for stable determination, These are the weight parameters.
[0107] Consistency Sub-score :
[0108] In the formula, The spatial standard deviation, calculated after intersecting the spatial back projections of multiple sampling points, is the three-dimensional point cloud dispersion, representing the degree of consistency of geometric positions when observed from multiple angles and with multiple pixels.
[0109] In this embodiment, further, based on the calculated mount reliability score S, the system executes a reason code generation mechanism. The reason code is used to provide calculable and interpretable evidence for fault tracing when the mount result is unreliable. Analyze the five sub-scores in the mounting credibility score S. , , , , .
[0110] Extract the evaluation sub-score with the lowest value, which has the most serious effect on lowering the credibility score and has the largest negative contribution.
[0111] Based on the evaluation sub-score with the largest negative contribution, the dominant physical bias source that leads to low positioning confidence is mapped to a specific cause code. And the cause code is associated with the disease record and stored, for example: like The minimum value generates the DATA_LOSS code, which indicates an interruption of the RTK differential signal and GPS signal drift. like The minimum value is ANGLE_LIMIT, which is generated because the shooting angle is greater than the preset upper limit, indicating that the angled shooting angle is too large and the projection geometry is distorted. like The smallest, and only hits the global bounding box, generates a bounding box backtracking code MESH_MISS, representing a broken or missing face in the triangular mesh.
[0112] In this embodiment, after the disease is successfully mounted and its location, confidence level S, and cause code are output, the system performs automatic hierarchical management based on the mounting confidence level S: High credibility ( The system determines that the three-dimensional mounting result is highly reliable, and the disease record is automatically entered into routine manual review or directly pushed to the digital maintenance ledger.
[0113] Medium credibility ( Disease records are automatically highlighted in red and sent to the management personnel's terminal for manual verification and review.
[0114] Low credibility ( ): Mounting and positioning do not directly enter the database; the system automatically calls the feedback decision mapping function. Generate closed-loop feedback action To drive a new round of reshoots or reverse parameter corrections:
[0115] In the formula, This refers to the refined scenario conditions of the current disease, among which For component types (such as main beams, piers). The type of disease (e.g., cracks). For the currently mounted link, The final hit level.
[0116] Low-trust reason code, scenario conditions and feedback actions The closed-loop control mapping is shown in Table 3.
[0117] Table 3. Mapping Table of Low-Confidence Cause Codes, Scenario Conditions, and Feedback Actions in Closed-Loop Control
[0118] Furthermore, when the manual verification terminal confirms or fine-tunes the above-mentioned medium and low confidence mounting results, the system utilizes the adjustment deviation feedback from the manual verification. The internal weight allocation of the scoring model is optimized by using a feedback-based weight adaptive update formula.
[0119] In the formula, The updated weights for each evaluation sub-score. The current weight value, To update the learning step size parameter, This refers to the deviation calculated by the system when the user manually fine-tunes the i-th indicator on the terminal. For example, if the user confirms that there is indeed a positioning offset, the system may determine that the preceding geometric score is too high. Take the negative value. , The preset lower and upper limits for weight safety are... Operators are used to prevent weights from increasing indefinitely or decreasing to zero, ensuring the stability of the scoring mechanism.
[0120] In summary, the method and system provided in this embodiment, through a two-stage flight planning mechanism, embeds the photogrammetric geometric conditions required for subsequent 3D defect localization into the planning target during the pre-flight planning stage. This ensures that subsequent supplementary photographs naturally possess a high geometric quality foundation for metadata direct ray mounting. Simultaneously, through dual-link mounting, uncertainty cone, and layered hit backoff techniques, it successfully solves the industry challenge of automatically and accurately locating close-range detail photographs that did not participate in 3D reconstruction. Building upon this, through multi-dimensional credibility scoring based on interpretable cause codes and data reverse flow control, it establishes a complete lifecycle from flight planning, 3D reconstruction, defect mounting, credibility assessment to manual review and feedback and parameter reverse optimization. This constructs a fully closed-loop, adaptively updated defect mounting and inspection system, significantly improving the efficiency, positioning accuracy, and engineering feasibility of digital infrastructure operation and maintenance management.
[0121] Example 2
[0122] In conjunction with the method provided in Embodiment 1, this embodiment provides a three-dimensional mounting and closed-loop inspection system 2 for infrastructure defects, the system structure diagram of which is shown below. Figure 2 The above includes: The 3D model and state diagram construction module 21 is used to acquire modeling images according to the planned modeling route, and construct a 3D model of the infrastructure and a surface unit state diagram for recording the state of each surface unit based on the modeling images. The supplementary shooting planning module 22 is used to generate a supplementary shooting route based on the surface unit state diagram, and to acquire close-range supplementary shooting images according to the supplementary shooting route; The two-dimensional defect identification module 23 is used to identify defects in the modeling image and the close-up re-shot image to obtain a two-dimensional defect result that includes the defect pixel range and defect identification quality. The dual-link intersection mounting module 24 is used to select either the aerial triangulation projection link or the metadata direct ray link based on whether the image containing the two-dimensional disease results participates in aerial triangulation reconstruction, to perform spatial intersection between the two-dimensional disease results and the three-dimensional model, obtain the three-dimensional disease location, and record the corresponding spatial hit quality. The credibility assessment module 25 is used to perform multi-dimensional credibility assessment based on the defect identification quality, the spatial hit quality, and the camera ray geometric parameters in spatial intersection, and to calculate the mounting credibility score. The reason code generation module 26 is used to extract the evaluation index sub-score that has the greatest negative contribution to the mounting credibility score to determine the dominant deviation source and generate a reason code to characterize the specific cause of the positioning deviation. The closed-loop feedback control module 27 is used to drive the closed-loop feedback action according to the cause code and update the surface unit state diagram.
[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.
Claims
1. A method for three-dimensional mounting and closed-loop inspection of infrastructure defects, characterized in that, include: Modeling images are acquired according to the planned modeling route, and a three-dimensional model of the infrastructure and a surface unit state diagram for recording the state of each surface unit are constructed based on the modeling images. A reshooting route is generated based on the surface unit state diagram, and close-up reshooting images are obtained based on the reshooting route; The modeling image and the close-up re-shot image are used to identify defects, and a two-dimensional defect result including the range of defect pixels and the quality of defect identification is obtained. Depending on whether the image containing the two-dimensional defect results participates in aerial triangulation reconstruction, an aerial triangulation projection link or a metadata direct ray link is selected, and the two-dimensional defect results are spatially intersected with the three-dimensional model to obtain the three-dimensional defect location and record the corresponding spatial hit quality. A multi-dimensional credibility assessment is performed based on the defect identification quality, the spatial hit quality, and the camera ray geometry parameters in spatial intersection, and the mounting credibility score is calculated. Extract the evaluation index sub-score that has the greatest negative contribution to the mounting credibility score to determine the dominant deviation source and generate a cause code to characterize the specific cause of the positioning deviation. The surface unit state diagram is updated based on the cause code that drives the closed-loop feedback action.
2. The method for three-dimensional mounting and closed-loop inspection of infrastructure defects according to claim 1, characterized in that, The surface unit state diagram records the modeling coverage, texture quality, occlusion status, disease risk, existing mount credibility, historical review conclusions, and reshoot history of each surface unit. Based on the surface unit state diagram, the reshoot priority of each surface unit is utilized. Identify the target for reshoots: In the formula, e is a surface element. For model quality score, This represents the disease risk value. The existing mount credibility score, For manual review of abnormalities or rejections, This refers to historical anomaly frequencies or historical risk values. , , , , These are the corresponding weighting factors for each item.
3. The method for three-dimensional mounting and closed-loop inspection of infrastructure defects according to claim 1, characterized in that, When generating the reshoot route, the following multi-objective comprehensive optimization objective function is used. The reshoot route is evaluated and selected from the candidate route set: In the formula, As candidate routes, To complete the model's returns, The quality score is based on the identifiable disease. This is for the subsequent automatic mounting quality prediction value. To cover the benefits of high-risk components, For flight risk values, For flight time or energy consumption, to These are the weights of each corresponding item.
4. The method for three-dimensional mounting and closed-loop inspection of infrastructure defects according to claim 1, characterized in that, Determine whether the surface unit needs to be re-photographed based on the following steps: When the trigger flag of the surface unit meets the following condition, it is determined that it will enter the generation queue of the reshoot route: In the formula, As a trigger flag, To improve the quality of disease identification, To assess mount reliability or predict mount quality, This represents the disease risk value. , , These are the corresponding identification quality threshold, mounting credibility threshold, and disease risk threshold. The logic states that if and only if, This indicates a logical OR.
5. The method for three-dimensional mounting and closed-loop inspection of infrastructure defects according to claim 1, characterized in that, Depending on whether the image containing the two-dimensional defect results is involved in aerial triangulation reconstruction, either an aerial triangulation projection link or a metadata direct ray link is selected, including: In the formula, I represents the two-dimensional image of the disease result. For the selected mount mode, This is a collection of images that have already been used in aerial triangulation reconstruction. In response to The aforementioned spatial triangulation projection link is selected at that time; In response to The metadata direct ray link is selected at that time.
6. The method for three-dimensional mounting and closed-loop inspection of infrastructure defects according to claim 1, characterized in that, Selecting the aerial triangulation projection link, the two-dimensional defect results are spatially intersected with the three-dimensional model to obtain the three-dimensional defect location and spatial hit quality, including: The original camera's calibration intrinsic parameters are scaled proportionally to the actual size of the current inference image, and the scaled camera intrinsic parameters are calculated as follows: In the formula, , These are the original calibrated focal lengths. , These are the original principal point coordinates. , The original calibration image width and height, , To infer the actual width and height of the image; Combined with the camera rotation matrix calculated from the photogrammetric attitude angle Construct the empty three-pixel back-projection ray r(t): In the formula, The normalized unit ray direction. , The coordinates of the diseased pixels. The coordinates of the camera center in the aerial triangulation results. This is the distance parameter.
7. The method for three-dimensional mounting and closed-loop inspection of infrastructure defects according to claim 1, characterized in that, Select the metadata direct ray link, and perform spatial intersection between the two-dimensional disease results and the three-dimensional model to obtain the three-dimensional disease location and spatial hit quality, including: Calculate the tolerance of the ray angle : In the formula, , , These represent the real-time dynamic differential positioning accuracy error, gimbal attitude angle error, and intrinsic parameter uncertainty, respectively. The shooting distance is the distance from the camera to the target surface. , , , The corresponding error term weights; Constructing an uncertainty cone : In the formula, As spatial candidate points, These are the camera's starting coordinates in the model coordinate system. The metadata direct ray direction is constructed from the pixel coordinates of the defect, the camera optical axis direction, and intrinsic parameters. This indicates the angle between the direction of the candidate point and the direction of the straight ray.
8. The method for three-dimensional mounting and closed-loop inspection of infrastructure defects according to claim 1, characterized in that, A layered hit-and-back mechanism is employed to spatially intersect the two-dimensional defect results with the three-dimensional model to obtain the three-dimensional defect location and spatial hit quality, including: During the intersection calculation, the camera ray or the uncertainty cone is intersected with the real triangular mesh of the 3D model; If no match is found, hierarchical hit backoff and intersection are performed sequentially to the component semantic layer, local bounding box, and global bounding box to obtain the final hit level, and the location of the three-dimensional defect is determined based on the final hit level. Based on the final hit level, calculate the model hit quality sub-score in the spatial hit quality. This serves as the quantitative basis for the quality of the space hit: In the formula, The base score for hit validity, For the hit level variable, This is the penalty coefficient that varies with the hit level.
9. The method for three-dimensional mounting and closed-loop inspection of infrastructure defects according to claim 1, characterized in that, Calculate the credibility score using the following formula: In the formula, These are the normalized evaluation sub-scores for each dimension. The weights for the corresponding scoring items; The normalized sub-scores for each dimension include: Metadata integrity sub-score calculated using positioning and attitude parameters recorded in the close-up recaptured images ; Geometric stability sub-fraction calculated based on the angle between the camera's line of sight and the target surface normal. ; The model hit mass fraction determined based on the final hit level. ; The defect identification quality sub-score is calculated based on the confidence level and pixel area output by the defect detection or segmentation algorithm. ; Consistency sub-score calculated based on backprojection discreteness of defect multi-sampling points in 3D space ; The weights corresponding to the above evaluation sub-scores are, and .
10. A three-dimensional mounting and closed-loop inspection system for infrastructure defects, characterized in that, include: The 3D model and state diagram construction module is used to acquire modeling images according to the planned modeling route, and construct a 3D model of the infrastructure and a surface unit state diagram for recording the state of each surface unit based on the modeling images. The reshoot planning module is used to generate a reshoot route based on the surface unit state diagram and to acquire close-up reshoot images according to the reshoot route. The two-dimensional defect identification module is used to identify defects in the modeled image and the close-up re-shot image to obtain two-dimensional defect results that include the defect pixel range and defect identification quality. The dual-link intersection mounting module is used to select either the aerial triangulation projection link or the metadata direct ray link based on whether the image containing the two-dimensional disease results participates in aerial triangulation reconstruction, to perform spatial intersection between the two-dimensional disease results and the three-dimensional model, obtain the three-dimensional disease location, and record the corresponding spatial hit quality. The credibility assessment module is used to perform multi-dimensional credibility assessment based on the defect identification quality, the spatial hit quality, and the camera ray geometric parameters in spatial intersection, and to calculate the mounting credibility score. The cause code generation module is used to extract the evaluation index sub-score that has the greatest negative contribution to the mounting credibility score to determine the dominant deviation source and generate a cause code to characterize the specific cause of the positioning deviation. The closed-loop feedback control module is used to drive the closed-loop feedback action according to the cause code and update the surface unit state diagram.
