Unmanned aerial vehicle building intelligent inspection method and system

Abstract

The application is an unmanned aerial vehicle building intelligent inspection method and system, which relates to the field of image processing and computer vision technology, including autonomous planning of an inspection path based on a BIM model, environmental data and a historical defect heat map. Data is collected by using a multi-source sensor, and a local dense point cloud is constructed through space-time registration. A tight coupling graph SLAM is used to fuse BIM prior constraints to generate a global dense point cloud. A hierarchical Transformer is used to obtain a fine-grained three-dimensional grid, and a multi-scale HRNet network is used to realize defect segmentation and classification. Finally, the defect information is mapped to the BIM model to generate a semantic enhanced BIM, thereby providing precise data support for building operation and maintenance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing and computer vision, in particular to a method and system for intelligent building inspection by unmanned aerial vehicle. BACKGROUND

[0002] The outer walls of urban buildings are mainly divided into decorated tile walls, glass curtain walls and unpainted cement mortar walls. The outer walls of ordinary high-rise residential buildings and other low-rise buildings are mainly decorated with tiles or painted with cement mortar, and super high-rise office buildings are mainly decorated with glass curtain walls. No matter what form the outer wall is, it will gradually age, and the outer wall tiles will fall off due to adhesion materials and construction defects, and the glass curtain walls and cement mortar walls will crack and peel off due to rain, strong winds and earthquakes. Building wall inspection has become an important part of building daily operation, and building wall hollowing, cracking and other hidden dangers are the core risk sources that threaten public safety. In order to ensure the safety of structural performance and prolong the service life of buildings, a large amount of manpower and resources are needed for daily operation every year.

[0003] In order to improve the efficiency and accuracy of inspection, unmanned aerial vehicle technology has been gradually introduced into intelligent building wall inspection. Unmanned aerial vehicle technology significantly improves the efficiency and flexibility of detection, especially in high-altitude and complex environment inspection operations, which can effectively overcome the limitations of manual inspection.

[0004] Existing technical solutions, such as Chinese granted patent application No. CN202410709447.X, propose a method and system for intelligent building wall inspection by unmanned aerial vehicle, which solves the problem of balancing between inspection efficiency and accuracy of recognition results. However, this solution still has some problems, especially in the process of obtaining multi-source heterogeneous data, which is difficult to accurately fuse and cooperatively understand, resulting in insufficient reliability of the final defect recognition result, distorted evaluation result, missed detection, misjudgment, and inability to guarantee the effectiveness and accuracy of the inspection result, thereby failing to provide timely and effective early warning and maintenance decisions, and ultimately exacerbating the public safety hazards of building walls. SUMMARY

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the main purpose of the present application is to provide a method and system for intelligent building inspection by unmanned aerial vehicle, which can provide more reliable and accurate building wall inspection results through accurate synchronization of multi-source data, optimized data fusion technology and more detailed analysis.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solution, a method for intelligent building inspection by unmanned aerial vehicle, comprising the following steps:

[0007] Based on the building BIM model of the building to be inspected, real-time environmental perception data and historical defect distribution heat map, an unmanned aerial vehicle inspection route for the building to be inspected is generated through a reinforcement learning model.

[0008] synchronously collecting building facade data by a multi-source sensor carried by a UAV, spatio-temporally registering the collected building facade data, obtaining synchronous multi-source data, and constructing a local dense point cloud model of a building facade;

[0009] introducing a structure prior constraint of a BIM model fused by an improved tight coupling graph SLAM, registering and fusing the local dense point cloud model of the building facade, and obtaining a global dense point cloud model of a building outer wall;

[0010] based on the global dense point cloud model of the building outer wall, obtaining a fine-grained three-dimensional grid model of a building outer wall through a hierarchical Transformer network;

[0011] constructing a multi-task deep learning model, adopting a multi-scale HRNet network and an edge-aware loss function, performing defect region segmentation and classification on the fine-grained three-dimensional grid model of the building outer wall, and obtaining a defect classification result;

[0012] based on the spatial position of the fine-grained three-dimensional grid model of the building outer wall, mapping the defect classification result to a grid vertex to generate a fine-grained three-dimensional defect model with a semantic label;

[0013] performing semantic alignment and information integration on the fine-grained three-dimensional defect model with a semantic label and the building BIM model, generating a semantic enhanced BIM model containing building outer wall defect detection information as a final UAV building intelligent inspection result.

[0014] the UAV inspection route of the building to be inspected generated by the reinforcement learning model comprises:

[0015] constructing a state space by using semantic information of a building BIM model of the building to be inspected, real-time environment perception data, and a historical defect distribution heat map; the state space comprises a UAV state vector and an environment state vector; the UAV state vector comprises position coordinates, attitude angles, battery power, and a current sensor mode; the environment state vector comprises a current region material type, a wind speed, an illumination intensity, an obstacle distance, and a historical defect density;

[0016] defining a set of actions of the UAV and material-sensitive actions, and obtaining an action space; when a high-reflective material is detected, triggering an action of reducing a flight height and activating a polarization camera; when a low-texture region is detected, triggering an action of increasing a lateral flight path and activating a laser radar;

[0017] designing a multi-target reward function:

[0018] ;

[0019] Wherein, Rcover is the area coverage reward, which is obtained based on the proportion of grid elements covered by the BIM model; Rquality is the defect detection reward, which is generated by weighting the historical defect density H; Rsafe is the safety constraint reward, which punishes the action of obstacle distance or wind speed; Reff is the efficiency reward, which is negatively correlated with the battery consumption rate; a, b, g, d are dynamic weight coefficients, which are adaptively adjusted according to the task stage;

[0020] The strategy network is trained by a deep reinforcement learning algorithm to obtain an optimal action sequence as a UAV inspection route of the building to be inspected.

[0021] The multi-source sensor includes a visible light camera, an infrared thermal imager, and a laser radar, and the synchronous acquisition of building facade data includes:

[0022] The visible light camera acquires high-definition images of the building facade;

[0023] The infrared thermal imager acquires temperature distribution data of the building facade;

[0024] The laser radar acquires three-dimensional point cloud data of the building facade;

[0025] The FPGA module is used to synchronize the timestamps of the high-definition images, temperature distribution data, and three-dimensional point cloud data, and to perform spatial registration, to obtain synchronous multi-source data;

[0026] The acquired building facade data is spatio-temporally registered, including:

[0027] The multi-source sensor data is coarsely registered based on a feature point matching algorithm to obtain coarsely registered multi-source data;

[0028] The coarsely registered multi-source data is finely registered using an iterative closest point algorithm to obtain finely registered data;

[0029] The registered multi-source data is unified to a local coordinate system to generate spatio-temporally aligned synchronous multi-source data.

[0030] The global building facade dense point cloud model is obtained, including:

[0031] According to the local dense point cloud model, the BIM model structure prior constraint, and the obtained initial pose sequence, pose nodes, local point cloud block nodes, point cloud matching edges, BIM constraint edges, and motion constraint edges are obtained;

[0032] The pose nodes and local point cloud block nodes are used as node layers, and the point cloud matching edges, BIM constraint edges, and motion constraint edges are used as edge layers to construct a graph optimization model to obtain a graph structure fused with multi-source constraints;

[0033] The graph optimization model is combined with optimization of pose estimation and geometric constraints through a dynamic weight distribution mechanism, wherein the weights are adaptively adjusted according to point cloud matching errors, and an optimized global pose sequence is obtained;

[0034] The optimized global pose sequence and the local dense point cloud model are subjected to multi-resolution registration based on the optimized pose, and a global dense point cloud is generated through weighted voxel fusion, thereby obtaining a global dense point cloud initial model;

[0035] The global dense point cloud initial model and the topological relationship of the BIM model are used to structurally complete the occluded area and assign geometric component semantic labels, thereby obtaining a global building outer wall dense point cloud model containing geometric and semantic information.

[0036] The pose node represents the unmanned aerial vehicle pose parameters, including position coordinates and attitude quaternions;

[0037] The local point cloud node is referenced to the pose node and is associated with a local point cloud data block;

[0038] The point cloud matching edge calculates the registration error between adjacent point cloud blocks through an ICP algorithm;

[0039] The BIM constraint edge generates geometric constraints based on the plane equation, symmetry rules and component topological relationship extracted from the BIM model;

[0040] The motion constraint edge constructs the relative motion constraint between adjacent poses through IMU pre-integration.

[0041] The obtained defect classification result comprises:

[0042] The multi-scale HRNet network is used as the backbone network, and the multi-scale HRNet network comprises four groups of parallel resolution branches, a cross-resolution feature fusion module and a double-task output layer;

[0043] The building outer wall fine-grained three-dimensional grid model is input into the backbone network, multi-scale geometric and texture features are extracted, the features are aggregated through the cross-resolution feature fusion module, and 256-dimensional fusion features are generated;

[0044] The fusion features are input into the double-task output layer, the double-task output layer comprises a segmentation task of outputting a vertex-level defect area probability map and a classification task of outputting a vertex-level defect area probability map, and the defect types include cracks, peeling, leakage and hollowing;

[0045] An edge-aware loss function is used to obtain the total loss, and the edge-aware loss function is combined by a cross-entropy loss and an edge gradient difference loss according to a weight of 7:3; wherein the edge gradient difference loss obtains the defect boundary gradient difference through a Sobel operator.

[0046] training the model based on the total loss to obtain a type classification result of the vertex-level defect.

[0047] The mapping of the defect classification result to the grid vertex includes:

[0048] According to the patch-level segmentation result, for each patch, all grid vertices in the coverage area of the patch are obtained;

[0049] The defect type classification result and the confidence of each patch are assigned to all grid vertices in the coverage area of the patch as vertex semantic labels; a fine-grained three-dimensional defect model containing vertex coordinates, normal vectors, patch texture mapping coordinates and defect labels is generated.

[0050] The semantic alignment and information integration include:

[0051] Based on the ICP algorithm, the fine-grained three-dimensional defect model is registered with the building BIM model to obtain a spatial transformation matrix;

[0052] The spatial transformation matrix is used to convert the fine-grained three-dimensional defect model to the BIM model coordinate system;

[0053] According to the spatial position of the defect vertex converted to the BIM model coordinate system, the defect vertex is associated with the corresponding building component instance in the BIM model;

[0054] The defect information associated with the building component instance is written into the BIM model in the form of an attribute set to generate a semantic enhanced BIM model.

[0055] An unmanned aerial vehicle building intelligent inspection system, comprising:

[0056] An inspection data preprocessing module is configured to generate an unmanned aerial vehicle inspection route for a building to be inspected based on a building BIM model of the building to be inspected, real-time environmental perception data and a historical defect distribution heat map, through a reinforcement learning model; the building facade data is synchronously collected by a multi-source sensor carried by the unmanned aerial vehicle, the collected building facade data is spatio-temporally registered, synchronous multi-source data is obtained, and a local dense point cloud model of the building facade is constructed;

[0057] The fusion processing module is used for introducing improved tight coupling graph SLAM to fuse the structural prior constraint of the BIM model, registering and fusing the local dense point cloud model of the building facade, and obtaining a global building outer wall dense point cloud model; based on the global building outer wall dense point cloud model, a hierarchical Transformer network is used to obtain a building outer wall fine-grained three-dimensional grid model; a multi-task deep learning model is constructed, a multi-scale HRNet network and an edge perception loss function are used to perform defect region segmentation and classification on the building outer wall fine-grained three-dimensional grid model, and a defect classification result is obtained;

[0058] The inspection data analysis module is used for mapping the defect classification result to a grid vertex based on the spatial position of the building outer wall fine-grained three-dimensional grid model, generating a fine-grained three-dimensional defect model with a semantic label, and performing semantic alignment and information integration on the fine-grained three-dimensional defect model with the building BIM model to generate a semantic enhanced BIM model containing building outer wall defect detection information as the final unmanned aerial vehicle building intelligent inspection result.

[0059] Compared with the prior art, the present application has the following advantages:

[0060] The present application realizes high automation in the whole process from inspection path planning, data acquisition, processing and analysis to result generation, significantly reduces the dependence on manual experience, improves the efficiency and consistency of inspection work, and constructs a precise digital model from macro to micro through multi-source sensor synchronous acquisition, tight coupling SLAM fusion BIM prior knowledge and fine recognition based on deep learning, greatly improves the accuracy and reliability of defect detection. And further break through the data barrier from detection to operation, the finally generated "semantic enhanced BIM model" not only contains geometric information, but also deeply integrates defect semantic information, realizes seamless connection of detection result and design, operation management, provides powerful data foundation and decision support for intelligent and fine operation and maintenance of buildings.

[0061] The application specifically fuses BIM semantics, environmental perception and historical defect data, dynamically generates an optimized inspection path through reinforcement learning, significantly improves coverage efficiency and defect detection rate, and can adaptively adjust flight and sensing strategies according to material characteristics. And through hardware synchronization and improved registration algorithm, accurate spatio-temporal alignment of multi-source heterogeneous data is realized, providing high-quality, consistent synchronous multi-source data basis for subsequent processing. In order to obtain more accurate inspection results, a more accurate three-dimensional model is needed, BIM structure prior constraints are introduced through tight coupling graph SLAM, pose estimation is effectively optimized, cumulative errors are eliminated, and a high-precision global dense point cloud and fine-grained grid model containing rich semantic information are constructed. Further, in order to realize more accurate defect recognition, multi-scale HRNet and edge perception loss function are used to realize automatic, high-precision segmentation and classification of various defects on the three-dimensional grid, with high accuracy. And the information integration closed loop of the application is efficient, which generates a semantic enhanced BIM model, accurately associates the defect detection results with the building components, forms a closed loop digital asset from detection to operation decision, and greatly improves the intelligent level and efficiency of building operation. BRIEF DESCRIPTION OF DRAWINGS

[0062] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and serve to explain the principles of the application, and do not limit the application in any way.

[0063] Figure 1 is a schematic diagram of the overall flow framework of the application.

[0064] Figure 2 is a schematic diagram of the inspection route generation process in the application.

[0065] Figure 3 is a schematic diagram of the multi-source data synchronous acquisition process in the application.

[0066] Figure 4 is a schematic diagram of the global point cloud reconstruction process in the application.

[0067] Figure 5 is a schematic diagram of the defect recognition process in the application.

[0068] Figure 6 is a schematic diagram of the defect result mapping process in the application.

[0069] Figure 7 is a schematic diagram of the semantic BIM integration process in the application.

[0070] Figure 8 is a schematic diagram of the system composition frame in the application. DETAILED DESCRIPTION

[0071] The application of UAV technology in building exterior wall inspection effectively improves the efficiency and range of detection. However, the core bottleneck of existing technical solutions such as Chinese patent application number CN202410709447.X is that it is difficult to perform high-precision fusion and collaborative understanding of the multi-source heterogeneous data obtained during the inspection process, resulting in insufficient reliability of the final defect identification results, and the inability to effectively integrate with the digital operation and maintenance system.

[0072] Specifically, due to the vibration of the UAV platform and the timing difference of each sensor, there are temporal and spatial synchronization errors between visible light, infrared thermal image, and laser radar data, causing distortion in the registration of three-dimensional point cloud and image data. At the same time, the simultaneous localization and mapping (SLAM) process in large-scale scenarios is prone to cumulative errors, and lacks effective use of prior structural information of buildings, making it difficult to generate accurate and reliable global three-dimensional models. Based on this distortion and unreliable model, the accuracy and reliability of defect identification will inevitably decrease, and the identification results are mostly in the form of isolated reports, which cannot be automatically integrated with the building information model (BIM), forming a "data island" from detection to operation and maintenance, greatly limiting the practical application value of intelligent inspection.

[0073] Therefore, how to fundamentally solve the distortion problem of multi-source data fusion and build an accurate, reliable and deeply integrated three-dimensional defect model with BIM has become the most prominent and urgent core technical problem in the field.

[0074] In order to overcome the shortcomings of the above-mentioned prior art, the main purpose of the present application is to provide a UAV building intelligent inspection method and system, which can provide more reliable and accurate building exterior wall inspection results through accurate synchronization of multi-source data, optimized data fusion technology and more detailed analysis.

[0075] To achieve the above-mentioned purpose, the UAV building intelligent inspection method adopts the following technical solutions, referring to Figure 1 , comprising the following steps:

[0076] Based on the building BIM model of the building to be inspected, real-time environmental perception data and historical defect distribution heat map, a UAV inspection route for the building to be inspected is generated through a reinforcement learning model;

[0077] Synchronously collecting building facade data by a multi-source sensor carried by a UAV, performing temporal and spatial registration on the collected building facade data, obtaining synchronous multi-source data, and constructing a local dense point cloud model of the building facade;

[0078] Introducing an improved tight coupling graph SLAM to fuse the structural prior constraints of the BIM model, registering and fusing the local dense point cloud model of the building facade to obtain a global building exterior wall dense point cloud model;

[0079] Based on the global building outer wall dense point cloud model, a hierarchical Transformer network is used to obtain a building outer wall fine-grained three-dimensional grid model.

[0080] A multi-task deep learning model is constructed, a multi-scale HRNet network and an edge perception loss function are used to perform defect region segmentation and classification on the building outer wall fine-grained three-dimensional grid model, and a defect classification result is obtained.

[0081] Based on the spatial position of the building outer wall fine-grained three-dimensional grid model, the defect classification result is mapped to the grid vertex to generate a fine-grained three-dimensional defect model with semantic labels.

[0082] The fine-grained three-dimensional defect model with semantic labels is semantically aligned and information integrated with the building BIM model to generate a semantic enhanced BIM model containing building outer wall defect detection information as the final unmanned aerial vehicle building intelligent inspection result.

[0083] The application will be further described below in conjunction with the drawings and embodiments.

[0084] Embodiment 1: This embodiment is aimed at unmanned aerial vehicle intelligent inspection of the main venue of a large sports center, referring to Figures 1 to 8 . The building is a mixed body of reinforced concrete and steel structure, the outer facade contains large-area glass curtain wall, metal decorative plate and concrete structure, the building height is 45 meters, and the total area of the facade is about 85000 square meters. The building is located in a coastal city, the wind speed during the inspection is 4.5 m / s, the air temperature is 18 ° C, humidity 75%, and light conditions are variable.

[0085] The unmanned aerial vehicle system used for this inspection is equipped with:

[0086] Canon EOS R5 full-frame visible light camera, resolution 8192x5464 pixels, 3 frames per second;

[0087] FLIR T1040 infrared thermal imager, resolution 1024x768 pixels, thermal sensitivity ≤0.03 ° C;

[0088] Velodyne VLS-128 laser radar, emitting 1.2 million points per second, detection distance 200 meters;

[0089] Built-in high-precision integrated navigation system, IMU sampling frequency 500Hz, positioning accuracy ±1cm;

[0090] and Xilinx UltraScale+FPGA synchronous processing unit.

[0091] Based on the Revit BIM model of the building to be inspected, the specific file size is 2.1 GB, including 12 categories of components, 35 types of material properties, real-time environmental data, and a five-year historical defect distribution heat map, recording 327 defect locations and types, and planning the inspection path through the reinforcement learning model.

[0092] The state space dimension is 28, including the position (XYZ) of the UAV, the attitude including roll, pitch, yaw, the remaining power 85%, the sensor working mode, the environmental state including the current facade material type such as glass, metal, concrete, the real-time wind speed, the light intensity (78000 lux), the distance to the nearest obstacle 12.5m, and the historical defect density 0.38 per square meter.

[0093] The action space defines 17 discrete actions, including triggering a 2.5m descent and activating a polarized filter camera when encountering a highly reflective glass curtain wall, and triggering a 3m lateral movement and enhancing the laser radar scanning frequency when encountering a low-texture concrete area. The initial values of the dynamic weight coefficients in the multi-objective reward function are set as a=0.35, b=0.25, g=0.20, d=0.20, and are adjusted adaptively with the task progress. The strategy network is trained through the PPO algorithm, and finally generates an inspection path with a total length of 38.6 kilometers, containing 12450 waypoints, and the estimated flight time is 4.2 hours.

[0094] In the data acquisition phase:

[0095] The visible light camera collects 43620 high-definition images, with an original data volume of 4.7TB;

[0096] The infrared thermal imager collects 10240 frames of temperature distribution data, with a data volume of 168GB;

[0097] The laser radar generates 2.15 billion three-dimensional point cloud data, with a data volume of 3.4TB.

[0098] Through the FPGA hardware synchronization system, the synchronization accuracy of multiple source data timestamps is ±0.2ms. The SuperPoint feature point algorithm is used for coarse registration, and 385,000 matching point pairs are extracted, with an initial registration error of ±6.8cm; then the generalized ICP algorithm is used for fine registration, and the final registration error is reduced to ±0.7cm. All data is converted to a local coordinate system with the northeast corner of the building as the origin, generating a spatiotemporally aligned synchronous multi-source data set.

[0099] The constructed local dense point cloud model of the building facade contains 215 local blocks, with an average point density of 28,000 points per square meter. In the tightly coupled SLAM optimization model:

[0100] Establish 12450 pose nodes, each node contains 3D position and quaternion attitude; 215 local point cloud block nodes; 428 edges are generated by ICP calculation of adjacent point cloud blocks; 86 plane equations, 32 groups of symmetry rules and 572 component topological relations are generated based on BIM constraint edges; 12449 edges are generated by IMU pre-integration motion constraint edges.

[0101] The dynamic weight distribution mechanism is adopted, the initial value of the point cloud matching error weight is 0.6, the BIM constraint weight is 0.3, and the motion constraint weight is 0.1. After graph optimization, the global pose estimation accuracy reaches ±1.2cm, and the attitude error is ±0.3°. Through multi-resolution registration and weighted voxel fusion, an initial global dense point cloud is generated, with a total of 3.24 billion points. Using the topological relationship of the BIM model, 183 occluded areas are structurally completed, with a total of about 42 million points, and finally a global building outer wall dense point cloud model containing geometric and semantic information is obtained.

[0102] A 12-layer hierarchical Transformer network is used, with a feature dimension of 1024 and 16 attention heads, to process point cloud data and generate a building outer wall fine-grained three-dimensional mesh model. The mesh contains 12.3 million triangular facets, with an average facet area of 0.0069 square meters and 6.15 million vertices.

[0103] The multi-task deep learning model uses HRNet-48 as the backbone network, and inputs the grid vertex coordinates, normal vectors and RGB texture information. The network contains four parallel resolution branches with resolutions of 1 / 4, 1 / 8, 1 / 16 and 1 / 32, respectively, and generates 256-dimensional fusion features through cross-resolution feature fusion. The double-task output layer simultaneously outputs:

[0104] A vertex-level defect segmentation probability map with a resolution of 8192x8192;

[0105] The vertex-level defect classification results include four categories: cracks, peeling, leakage and hollowing.

[0106] The training data contains 25400 labeled samples, including 8200 cracks, 7100 peeling, 6300 leakage and 3800 hollowing. An edge-aware loss function is used, with a cross-entropy loss weight of 0.7 and an edge gradient difference loss weight calculated by Sobel operator of 0.3. The final model achieves an accuracy of 94% and a recall rate of 92% on the test set.

[0107] This inspection identified a total of:

[0108] Cracks: 283, maximum length 3.2m, average width 2.1mm, confidence 0.96;

[0109] Spalling: 197 locations, maximum area 0.86 m2, average depth 1.5 cm, confidence 0.93;

[0110] Leakage: 156 locations, maximum area 2.4 m2, confidence 0.91;

[0111] Hollow: 82 locations, maximum area 1.2 m2, confidence 0.89.

[0112] The face-level defect classification results are mapped to the grid vertices, each face covers an average of 8.7 vertices, and a total of 10.7 million vertices are given semantic labels. The generated fine-grained three-dimensional defect model contains vertex coordinates, normal vectors, RGB texture values, defect types, and confidence, etc. The total data volume is 8.3 GB.

[0113] ICP algorithm is used for model registration, and the registration error is ±0.9 cm. After converting the defect model to the BIM coordinate system through the spatial transformation matrix, the defect vertices are associated with the component instances in the BIM model:

[0114] There are 183 associated defects for glass curtain wall components, 297 for metal decorative plates, and 238 for concrete structures. All defect information is written into the BIM model in the form of attribute sets, generating a semantic-enhanced BIM model in IFC format, with a file size of 3.8 GB.

[0115] The total time for this inspection is 3.5 hours, and a total of 718 defects are identified, with a comprehensive detection accuracy of 94%. Compared with traditional manual inspection, which takes about 3 weeks, the efficiency is improved by about 40 times. The generated semantic-enhanced BIM model can provide accurate data support for the subsequent maintenance and reconstruction of the sports center, and is expected to save maintenance costs by 35%.

[0116] Example 2: This example is aimed at the intelligent inspection of a super high-rise building in a financial center by using a UAV, which is described in detail in Figures 1 to 8 . The building is a steel structure outer frame + reinforced concrete core tube structure, with a unit type glass curtain wall and granite facing plate on the facade, a building height of 632 meters, and a total facade area of about 225,000 square meters. The building is located in the core area of the city, and the wind speed during the inspection period is 3.8 m / s, the air temperature is 22 ° C, humidity 68%, and there is slight electromagnetic interference.

[0117] The UAV system used for this inspection is equipped with:

[0118] Phase One iXU-RS1000 multi-spectral camera, resolution 10100x11320 pixels, 5-channel spectral imaging, including RGB, near-infrared, red edge band;

[0119] InfraTec ImageIR 8300 infrared thermal imager, resolution 1280×1024 pixels, frame rate 500 Hz, temperature measurement accuracy ±0.02 ° C;

[0120] Leica BLK2FLY autonomous flight laser radar, emitting 360000 points per second, ranging accuracy ±1mm;

[0121] Septentrio AsteRx-i3 GNSS / IMU integrated navigation system, supporting multi-frequency and multi-constellation, positioning accuracy ±0.5cm;

[0122] and Xilinx Versal ACAP adaptive computing acceleration platform, integrating FPGA and AI engine.

[0123] Based on the Bentley BIM model of the building to be inspected, the file size is 4.8GB, containing 23 categories of components, 8542 component instances, 67 material properties, real-time environmental data, and nearly eight years of historical defect distribution heat maps, recording 1285 defect locations and types, and planning the inspection path of 3 drones through the reinforcement learning model.

[0124] The state space dimension is 42, including the joint state of the drone swarm, including relative position, power distribution, task load, and environmental dynamic parameters, specifically real-time wind field model, electromagnetic interference intensity, and light change gradient.

[0125] Sensor operating mode; environmental state includes current facade material type, such as Low-E glass, granite, aluminum plate, real-time wind speed, light intensity of 125000 lux, distance to the nearest obstacle of 15.8m, and historical defect density of 0.42 per square meter.

[0126] Action space defines 32 collaborative actions, including triggering a 3.2m descent and activating multispectral imaging mode when encountering highly reflective Low-E glass curtain walls; triggering three-dimensional surround flight and increasing laser radar scanning density when encountering complex decorative line areas. The initial values of the dynamic weight coefficients in the multi-objective reward function are set as α=0.38, β=0.28, γ=0.19, δ=0.15, and are adjusted adaptively with the progress of the task. The strategy network is trained through the SAC algorithm, and finally generates an inspection path with a total length of 126.8 kilometers, containing 38450 waypoints, and the estimated flight time is 8.5 hours.

[0127] Data acquisition phase:

[0128] Multispectral camera collects 58620 high-definition images, with raw data volume of 12.8TB;

[0129] The infrared thermal imager collected 24580 frames of temperature distribution data, with a data volume of 2.1TB;

[0130] The laser radar generated 586 million three-dimensional point cloud data, with a data volume of 9.2TB.

[0131] Through the FPGA hardware synchronization system, the synchronization accuracy of multi-source data time stamp is ±0.1ms. LoFTR feature matching algorithm is used for coarse registration, and 1268 million matching point pairs are extracted, with an initial registration error of ±5.2cm; then NDT algorithm is used for fine registration, and the final registration error is reduced to ±0.5cm. All data is converted to a local coordinate system with the building center as the origin, generating a spatio-temporally aligned synchronous multi-source data set.

[0132] The constructed local dense point cloud model of building facade contains 586 local blocks, with an average point density of 36,000 points per square meter. In the tightly coupled SLAM optimization model:

[0133] 38450 pose nodes are established;

[0134] 586 local point cloud block nodes are established;

[0135] 1172 edges are generated by calculating the constraints between adjacent point cloud blocks through GICP;

[0136] BIM constraint edges are generated based on 156 plane equations, 68 symmetry rules and 892 component topology relationships extracted from the Bentley model;

[0137] 38449 motion constraint edges are generated through IMU pre-integration.

[0138] A dynamic weight distribution mechanism is adopted, with the initial value of point cloud matching error weight being 0.65, the BIM constraint weight being 0.25, and the motion constraint weight being 0.10. After graph optimization, the global pose estimation accuracy reaches ±0.8cm, and the attitude error is ±0.2°. Through multi-resolution registration and weighted voxel fusion, an initial global dense point cloud is generated, with a total of 864 million points. Using the topological relationship of the BIM model, 356 occluded areas are structurally completed, with a total of about 120 million points of completed point cloud, and finally a global building facade dense point cloud model containing geometric and semantic information is obtained.

[0139] A 16-layer hierarchical Transformer network is used to process point cloud data, with a feature dimension of 2048 and 24 attention heads, to generate a fine-grained three-dimensional mesh model of building facade. The mesh contains 38.6 million triangular facets, with an average facet area of 0.0058 square meters and 19.3 million vertices.

[0140] The multi-task deep learning model takes HRNet-64 as the backbone network, and inputs the grid vertex coordinates, normal vectors and multi-spectral texture information. The network contains four parallel resolution branches with resolutions of 1 / 4, 1 / 8, 1 / 16 and 1 / 32, respectively, to generate 512-dimensional fusion features through cross-resolution feature fusion. The dual-task output layer simultaneously outputs: a vertex-level defect segmentation probability map with a resolution of 16384x16384; and a vertex-level defect classification result, specifically five categories of cracks, peeling, leakage, hollowing and corrosion.

[0141] The training data contains 68,500 labeled samples, specifically 18,200 cases of cracks, 16,800 cases of peeling, 15,600 cases of leakage, 9,400 cases of hollowing, and 8,500 cases of corrosion. An improved edge-aware loss function is used, with a weighted cross-entropy loss weight of 0.75 and an adaptive edge gradient difference loss weight of 0.25. The final model achieves an accuracy of 96.2% and a recall rate of 94.8% on the test set.

[0142] This inspection identified a total of:

[0143] Cracks: 428, maximum length 4.8m, average width 1.8mm, confidence 0.97, detection accuracy 97.2%;

[0144] Peeling: 286, maximum area 1.26m², average depth 2.1cm, confidence 0.95, detection accuracy 96.5%;

[0145] Leakage: 198, maximum area 3.2m², confidence 0.93, detection accuracy 95.8%;

[0146] Hollowing: 125, maximum area 1.8m², confidence 0.90, detection accuracy 94.3%;

[0147] Corrosion: 87, maximum area 2.4m², confidence 0.88, detection accuracy 93.1%.

[0148] Map the face-level defect classification results to the grid vertices, with each face covering an average of 9.2 vertices, and a total of 35.5 million vertices assigned semantic labels. The generated fine-grained three-dimensional defect model contains vertex coordinates, normal vectors, multi-spectral texture values, defect types and confidence, with a total data size of 25.6GB.

[0149] The improved ICP algorithm is used for model registration, and the registration error is ±0.6 cm. After the defect model is converted to the BIM coordinate system through the space transformation matrix, the defect vertices are associated with the component instances in the BIM model: 362 defects are associated with the glass curtain wall component, 418 defects are associated with the granite facing, and 144 defects are associated with the metal decoration. All defect information is written into the BIM model in the form of an attribute set, generating a semantic enhanced BIM model in IFC format with a file size of 6.4 GB.

[0150] The total inspection time is 7.8 hours, and 1124 defects are identified, with a comprehensive detection accuracy of 96.2%. Compared with the traditional manual inspection which takes about 6 weeks, the efficiency is improved by about 50 times. The generated semantic enhanced BIM model can provide accurate data support for the safe operation and maintenance of the super high-rise building, and is expected to save maintenance cost by 42% and improve maintenance efficiency by 3.5 times.

[0151] This embodiment demonstrates the high-precision and high-efficiency detection capability of the unmanned aerial vehicle intelligent inspection system in complex large public buildings, providing a complete technical solution for intelligent operation and maintenance of important infrastructure.

[0152] It should be noted that in the present application, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment.

[0153] The above embodiments are only illustrative of the present application and do not constitute a limitation on the protection scope of the present application. Any design identical or similar to the present application falls within the protection scope of the present application.

Claims

1. A method for intelligent building inspection using unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: Based on the building BIM model of the building to be inspected, real-time environmental perception data, and historical defect distribution heat map, a drone inspection route for the building to be inspected is generated through a reinforcement learning model. Building facade data is collected synchronously by multi-source sensors carried by a drone. The collected building facade data is spatiotemporally registered to obtain synchronous multi-source data and to construct a local dense point cloud model of the building facade. An improved tightly coupled graph SLAM fusion BIM model is introduced with structural prior constraints. The local dense point cloud model of the building facade is registered and fused to obtain a global dense point cloud model of the building exterior wall. Based on the global dense point cloud model of the building exterior wall, a fine-grained three-dimensional mesh model of the building exterior wall is obtained through a hierarchical Transformer network. A multi-task deep learning model is constructed, using a multi-scale HRNet network and an edge-aware loss function. The multi-scale HRNet network is used as the backbone network. The multi-scale HRNet network includes four sets of parallel resolution branches, a cross-resolution feature fusion module, and a dual-task output layer. Defect regions are segmented and classified on the fine-grained three-dimensional mesh model of the building exterior wall to obtain defect classification results; Based on the spatial location of the fine-grained three-dimensional mesh model of the building exterior wall, the defect classification results are mapped to the mesh vertices to generate a fine-grained three-dimensional defect model with semantic labels. The fine-grained 3D defect model with semantic tags is semantically aligned and information integrated with the building BIM model to generate a semantically enhanced BIM model containing building exterior wall defect detection information, which serves as the final result of UAV intelligent building inspection.

2. The unmanned aerial vehicle (UAV) intelligent building inspection method as described in claim 1, characterized in that, The process of generating drone inspection routes for buildings to be inspected using a reinforcement learning model includes: Using the semantic information of the building BIM model of the building to be inspected, the real-time environmental perception data, and the historical defect distribution heatmap, a state space is constructed. The state space includes a UAV state vector and an environmental state vector. The UAV state vector includes: position coordinates, attitude angle, battery level, and current sensor mode. The environmental state vector includes: current area material type, wind speed, light intensity, obstacle distance, and historical defect density. Define the drone's action set and material-sensitive actions to obtain the action space. When a highly reflective material is detected, the action is to reduce the flight altitude and activate the polarization camera; when a low-texture area is detected, the action is to increase the lateral flight path and activate the lidar. Design a multi-objective reward function, expressed as follows: ； in, Rcover The coverage ratio of grid cells is obtained based on the BIM model as a regional coverage bonus. Rquality Incentives for defect detection are based on historical defect density. H Weighted generation; Rsafe To ensure safety, rewards and penalties are imposed for actions involving obstacle distance or wind speed. Reff The reward is based on efficiency and is negatively correlated with the battery consumption rate; α, β, γ, and δ are dynamic weighting coefficients that are adaptively adjusted according to the task stage. The policy network is trained using deep reinforcement learning algorithms to obtain the optimal action sequence, which serves as the drone inspection route for the building to be inspected.

3. The unmanned aerial vehicle (UAV) intelligent building inspection method as described in claim 1, characterized in that, The multi-source sensors include a visible light camera, an infrared thermal imager, and a lidar; the synchronous acquisition of building facade data includes: High-resolution images of the building facade are acquired using the visible light camera; The infrared thermal imager is used to collect temperature distribution data of the building facade; The laser radar is used to collect three-dimensional point cloud data of the building facade; The FPGA module is used to perform time-stamp synchronization and spatial registration of the high-definition images, temperature distribution data and 3D point cloud data to obtain synchronized multi-source data. Spatiotemporal registration of the collected building facade data includes: The synchronous multi-source data is coarsely registered based on the feature point matching algorithm to obtain coarsely registered multi-source data. The iterative nearest point algorithm is used to perform fine registration on the coarse registration multi-source data to obtain fine registration data; The registered multi-source data are unified into a local coordinate system to generate spatiotemporally aligned synchronous multi-source data.

4. The unmanned aerial vehicle (UAV) intelligent building inspection method as described in claim 1, characterized in that, The process of obtaining a global dense point cloud model of building exterior walls includes: Based on the local dense point cloud model, the prior constraints of the BIM model structure, and the obtained initial pose sequence, pose nodes, local point cloud block nodes, point cloud matching edges, BIM constraint edges, and motion constraint edges are obtained. By using pose nodes and local point cloud block nodes as node layers, and point cloud matching edges, BIM constraint edges, and motion constraint edges as edge layers, a graph optimization model is constructed to obtain a graph structure that integrates multi-source constraints. The graph optimization model is used to jointly optimize pose estimation and geometric constraints through a dynamic weight allocation mechanism, wherein the weights are adaptively adjusted according to the point cloud matching error to obtain the optimized global pose sequence. The optimized global pose sequence and the local dense point cloud model are used to perform multi-resolution registration of the local point cloud based on the optimized pose, and the global dense point cloud is generated by weighted voxel fusion to obtain the initial model of the global dense point cloud. The topological relationship between the initial global dense point cloud model and the BIM model is used to structurally complete the occluded areas using the BIM topological relationship, and semantic labels are assigned to geometric components to obtain a global building exterior wall dense point cloud model containing geometric and semantic information.

5. The unmanned aerial vehicle (UAV) intelligent building inspection method as described in claim 4, characterized in that, The pose node represents the pose parameters of the UAV, including position coordinates and attitude quaternions; The local point cloud block node is associated with the local point cloud data block with reference to the pose node; The point cloud matching edge is calculated using the ICP algorithm to determine the registration error between adjacent point cloud blocks. The BIM constraint edges are generated based on the plane equations, symmetry rules, and component topological relationships extracted from the BIM model to generate geometric constraints. The motion constraint edges are constructed by IMU pre-integration to establish relative motion constraints between adjacent poses.

6. The unmanned aerial vehicle (UAV) intelligent building inspection method as described in claim 5, characterized in that, The method for obtaining defect classification results includes: The fine-grained 3D mesh model of the building exterior wall is input into the backbone network, multi-scale geometric and texture features are extracted, and the features are aggregated through the cross-resolution feature fusion module to generate 256-dimensional fused features; The fused features are input into a dual-task output layer, which includes a segmentation task that outputs a vertex-level defect region probability map and a classification task that outputs vertex-level defect type labels. The defect types include cracks, peeling, leakage, and hollowness. The model is trained by obtaining the total loss using an edge-aware loss function, which is a combination of cross-entropy loss and edge gradient difference loss with a weight of 7:3; wherein, the edge gradient difference loss is obtained by using the Sobel operator to obtain the gradient difference of the defect boundary. Based on the total loss training model, the type classification results of vertex-level defects are obtained.

7. The unmanned aerial vehicle (UAV) intelligent building inspection method as described in claim 1, characterized in that, The semantic alignment and information integration include: Based on the ICP algorithm, the fine-grained 3D defect model and the building BIM model are spatially registered to obtain the spatial transformation matrix. The fine-grained three-dimensional defect model is transformed to the BIM model coordinate system using the spatial transformation matrix. Based on the spatial location of the defect vertex in the BIM model coordinate system, the defect vertex is associated with the corresponding building component instance in the BIM model; Defect information associated with building component instances is written into the BIM model as an attribute set to generate a semantically enhanced BIM model.

8. A system applying the UAV intelligent building inspection method according to any one of claims 1-7, characterized in that, include: The inspection data preprocessing module is used to generate the UAV inspection route for the building to be inspected based on the building BIM model, real-time environmental perception data and historical defect distribution heat map of the building to be inspected through a reinforcement learning model; and to collect building facade data synchronously through multi-source sensors carried by the UAV, perform spatiotemporal registration on the collected building facade data to obtain synchronous multi-source data, and construct a local dense point cloud model of the building facade. The fusion processing module is used to introduce structural prior constraints into the improved tightly coupled graph SLAM fusion BIM model, register and fuse the local dense point cloud model of the building facade to obtain a global dense point cloud model of the building exterior wall; based on the global dense point cloud model of the building exterior wall, a fine-grained 3D mesh model of the building exterior wall is obtained through a hierarchical Transformer network; a multi-task deep learning model is constructed, using a multi-scale HRNet network and an edge-aware loss function, with the multi-scale HRNet network as the backbone network. The multi-scale HRNet network includes four sets of parallel resolution branches, a cross-resolution feature fusion module, and a dual-task output layer. Defect regions are segmented and classified on the fine-grained three-dimensional mesh model of the building exterior wall to obtain defect classification results; The inspection data analysis module is used to map the defect classification results to the grid vertices based on the spatial location of the fine-grained three-dimensional mesh model of the building exterior wall, generating a fine-grained three-dimensional defect model with semantic labels; the fine-grained three-dimensional defect model with semantic labels is semantically aligned and information integrated with the building BIM model to generate a semantically enhanced BIM model containing building exterior wall defect detection information, which serves as the final result of the UAV intelligent building inspection.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 7.

Images