Onboard infrared-visible double light combined building outer wall hollowing identification and positioning method

Abstract

The invention discloses an airborne infrared-visible double light combined building outer wall hollowing identification and positioning method. The method comprises the following steps: designing flight parameters of an unmanned aerial vehicle; designing a calibration checkerboard; shooting an infrared thermal image and a visible light image of the checkerboard calibration target, recording the images as dual-light images for calibration, and calibrating and calculating internal and external parameters and a distortion coefficient of a camera; collecting images of all external facades of the target building; carrying out distortion removal processing on the dual-light image to form a dual-light image database for detection; the registration between the dual-light images synchronously shot by the unmanned aerial vehicle is realized; realizing registration between the building facade view and the visible light image; determining a correct hollowing area identification result, and marking the infrared thermal image corresponding to the correct hollowing area identification result as a hollowing infrared thermal image; calculating a pixel coordinate range of the hollowing area; and accurate positioning of the hollowing area is realized. According to the method, the visible light image serves as a medium, registration of the infrared thermal image and the macroscopic 3D model is achieved, the problem that calibration and registration are difficult due to the fact that the imaging quality of the infrared thermal image is low is solved, and high readability and user friendliness of an output result are guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of building exterior wall disease prevention and control, and in particular to an airborne infrared-visible dual-light combined building exterior wall hollowing identification and positioning method. Background Art

[0002] Hollowing refers to the separation, peeling, or loss of effective adhesion between the exterior wall finish and its substrate due to insufficient adhesion, material aging, or poor construction quality. This not only affects the building's aesthetics but can also develop into cracking, peeling, and other later stages, posing safety hazards. Therefore, detecting hollow areas on building exterior walls is crucial. Infrared thermal imaging is currently one of the most efficient and accurate nondestructive testing methods for building exterior walls. However, traditional infrared image acquisition methods rely primarily on handheld or mounted cameras, making it difficult to implement on tall buildings, and image quality is affected by the shooting angle. The maturation of drone remote sensing technology, thanks to its fully automated flight control and superior maneuverability, has overcome the challenges of hollowing detection in tall, wide, and extensive building exteriors. Traditional infrared image data interpretation typically relies on manual labor, which is time-consuming and labor-intensive. Results are often influenced by the inspector's experience, subject to significant subjectivity, and lack of guaranteed accuracy. Furthermore, due to significant distortion, low resolution, and the difficulty in aligning the field of view with visible light cameras during simultaneous capture, hollowing detection results are difficult to pinpoint and visualize in macroscopic models. Therefore, the hollowing detection of building exterior walls based on UAV remote sensing still needs an efficient adaptive recognition method and an accurate positioning method. Summary of the Invention

[0003] The present invention aims to overcome the aforementioned problems of the prior art by providing an airborne infrared-visible dual-light combined method for identifying and locating hollowing in building exterior walls. This method achieves adaptive identification and precise location of hollowing areas, while providing highly readable detection results. The core of the present invention is to align infrared thermal images with macroscopic 3D models using visible light images as a medium. This method addresses the difficulty in calibrating and registering infrared thermal images due to their low imaging quality, ensuring highly readable and user-friendly output results.

[0004] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0005] An airborne infrared-visible dual-light combination method for identifying and locating hollowing in building exterior walls comprises the following steps:

[0006] (1) According to the target building and its surrounding environment, and taking into account parameters such as detection distance and ground sampling distance, a suitable drone is selected and equipped with appropriate infrared and visible light cameras, and flight parameters such as the drone route, altitude, and speed are customized.

[0007] (2) Design a calibration checkerboard with suitable size, use 3D printing technology, select two materials with significant contrast to print each unit separately, and form a checkerboard calibration target through precise splicing.

[0008] (3) Fix the calibration checkerboard at a suitable location on the building's exterior wall. Based on the detection distance determined in step (1), capture infrared thermal images and visible light images of the checkerboard calibration target from multiple angles. These images are recorded as calibration bi-optical images. The Zhang Zhengyou calibration method is used to calculate the camera's intrinsic and extrinsic parameters and distortion coefficients.

[0009] (4) Place the drone in an open area, start the drone and fly it along the predetermined route. Simultaneously collect infrared thermal images and visible light images at each waypoint until the image acquisition task of the entire facade of the target building is completed.

[0010] (5) Dedistort the bi-optical images obtained in steps (3) and (4) according to the camera calibration parameters. Establish the correspondence between the synchronously collected bi-optical images, and classify them step by step according to the building number and facade to form a bi-optical image database for detection.

[0011] (6) For the calibration bi-optical image obtained in step (5), a sub-pixel precision corner detection algorithm is used to detect the corner points of the calibration target as control points, and the pixel coordinates of the control points are located. Based on the known geometric relationship of the calibration target, the correspondence between the bi-optical image control points is established, and the homography matrix is ​​calculated. The homography matrix is ​​used to achieve registration between the bi-optical images taken synchronously by the UAV.

[0012] (7) Using the visible light image obtained in step (5), a point cloud model of the target building is generated using 3D reconstruction technology, and an orthophoto of the target building facade is extracted from the point cloud model. The building facade view is drawn using Building Information Modeling (BIM) software. A method based on Generalized Hough Transform (GHT) is used to achieve registration between the building facade view and the visible light image.

[0013] (8) For the infrared thermal image obtained in step (5), the K-means++ clustering algorithm is used to cluster the number of clusters from 2 to 12. The three clustering effectiveness indicators of silhouette coefficient, Davis-Boulding index and variance ratio are comprehensively used for quantitative evaluation to determine the optimal number of clusters and the corresponding clustering results. Furthermore, the convex hull algorithm is used to screen the preliminary recognition results of the hollow area, determine the correct hollow area recognition results, and mark the corresponding infrared thermal image as the hollow infrared thermal image.

[0014] (9) For the hollowing infrared thermal image obtained by step (8), the homography matrix obtained in step (6) is used to align the hollowing area identification result corresponding to the infrared thermal image with the visible light image taken simultaneously with the image to obtain the hollowing visible light image. According to the pixel coordinate system, the pixel coordinate range of the hollowing area is calculated.

[0015] (10) For the hollow visible light image obtained by processing in step (9), according to the registration relationship between the visible light image obtained in step (7) and the facade view of the building to which it belongs, the positioning of the hollow area in the building facade view is achieved, and the position of the hollow area is accurately drawn in the facade view to achieve accurate positioning of the hollow area.

[0016] The embodiments of the present invention have the following advantages:

[0017] (1) By combining UAV technology with infrared thermal imaging technology, the advantages of UAV flexibility and maneuverability and the non-destructive detection characteristics of infrared thermal imaging are fully utilized, which not only improves the efficiency and range of image acquisition, but also enhances the stability and reliability of the acquired images, providing a practical and cost-effective data acquisition method for hollowing detection of building exterior walls.

[0018] (2) For a single-frame infrared thermal image of a building exterior wall, a clustering algorithm is used for multi-threshold segmentation, and clustering evaluation indicators are used to evaluate the clustering effect. This achieves adaptive segmentation of the hollow area, reduces the interference of human factors, improves the robustness of the algorithm, and greatly improves the recognition efficiency while ensuring recognition accuracy.

[0019] (3) For infrared thermal images of building exterior walls with hollow areas, visible light images are used as a bridge to achieve accurate registration between infrared thermal images and building facade views in BIM. By accurately mapping the hollow areas in infrared thermal images to building facade views, an accurate hollow area positioning method and a highly user-friendly detection result presentation form are formed, achieving a leap from qualitative identification to quantitative analysis, and providing a solid data foundation for the overall planning of subsequent assessments, repairs, and other work. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Flowchart of the main implementation steps of this embodiment. DETAILED DESCRIPTION

[0021] The specific implementation of the present invention will be further described below with reference to specific examples and in conjunction with the accompanying drawings.

[0022] An airborne infrared-visible dual-light combination method for identifying and locating hollowing in building exterior walls comprises the following steps:

[0023] (1) Based on the target building and its surrounding environment, and in combination with the detection distance required by the project (usually the optimal flight distance is determined based on 1.2-1.5 times the building height) and the ground sampling distance GSD requirements (generally set to 1 / 1000-1 / 2000 of the building facade feature size), select a suitable drone and equip it with appropriate infrared and visible light cameras, and customize the drone's flight parameters such as route, altitude and speed to ensure the efficiency of the detection process and the accuracy of the detection results. The relevant calculation formulas for the flight parameters are as follows:

[0024] Based on the BIM model of the target building, the Delaunay triangulation algorithm is used to construct a 3D mesh model of the building surface. The improved ant colony algorithm is used for path optimization. The objective function of the drone route is:

[0025]

[0026] Where, L i is the length of the flight segment, v is the speed, E i is the energy consumption coefficient, E i It is the task time window constraint;

[0027] Establishing the altitude-speed coupling control model:

[0028] H=K·(GSD·f) / (s·N)

[0029] v max =H·f·FR / (GSD·n)

[0030] Where FR is the frame rate and n is the heading overlap ratio. A PID controller is used to adjust flight parameters in real time to ensure that the heading overlap ratio is ≥80% and the side overlap ratio is ≥70%, and that the sampling theorem requirement of λ / 4H ≤ 0.25 is met.

[0031] (2) In computer software, a 7×7 checkerboard calibration target model is constructed and divided into cells of equal size, each cell being 100 mm×100 mm. Using 3D printing technology, two materials with significant contrast are selected to print each cell separately, and the checkerboard calibration target is formed by precise splicing. Specifically, polylactic acid (PLA) and matte aluminum are selected as printing materials. These two materials have significant contrast characteristics in thermal imaging. With this material combination, the checkerboard calibration target produced can not only meet the high-precision calibration requirements, but also provide clear visual effects in thermal imaging applications.

[0032] Develop a standard unit module with a mortise and tenon positioning structure, use laser cutting to produce ±5μm positioning holes, and use high-precision pneumatic clamps to achieve rapid assembly. Design a two-degree-of-freedom adjustment mechanism to compensate for the dimensional deviation ΔL=α·L·ΔT (α=1.2×10 -5 / ℃).

[0033] (3) Fix the calibration chessboard at a suitable position on the building's exterior wall. The selected position must meet the following conditions: ensure that the plane of the calibration chessboard is stable and vertical, and avoid chessboard flatness errors caused by uneven walls or improper installation; avoid interference from surrounding environmental factors (such as obstructions, strong light source reflections, etc.) on the image acquisition process to ensure the accuracy and stability of image acquisition; take into account the convenience of subsequent multi-angle shooting, so that the camera can focus smoothly at different angles and capture clear and complete chessboard images.

[0034] Based on the detection distance determined in step (1), infrared thermal images and visible light images of the checkerboard calibration target are captured from multiple angles and recorded as calibration bi-optical images. These captured images are uniformly labeled and stored to form a complete calibration bi-optical image dataset.

[0035] The Zhang Zhengyou calibration method is used to calculate the camera's intrinsic parameters, extrinsic parameters, and distortion parameters. The Zhang Zhengyou calibration method's related formulas and parameter solution procedures are as follows:

[0036] For visible light images, a sub-pixel Harris-Laplace corner detection algorithm is used with a positioning error of σ≤0.1pixel. For infrared images, a thermal gradient feature enhancement function is constructed. Combined with template matching to achieve corner point recognition, the matching success rate is ≥98%.

[0037] Establish dual-spectrum joint calibration objective function:

[0038]

[0039] Where, is the internal parameter matrix, and D = [k1, k2, p1, p2, k3] is the distortion coefficient vector. Introduce the cross constraint term (σ is the sensor noise variance);

[0040] (4) Select an open area within a radius of 500m around the construction site as the take-off and landing point for the UAV, ensuring that the take-off and landing site is away from electromagnetic interference sources. During the flight preparation phase, the ground station software completes a comprehensive self-test of the UAV and payload system, environmental adaptability testing, and route logic verification.

[0041] After the drone is launched, the flight controller manually hovers and aligns according to the pre-set route planning parameters, ensuring that the initial attitude deviation of the aircraft is controlled within ±1° and the altitude deviation is ≤0.5m. Automatic flight mode is activated, and the flight control system flies along the preset route based on RTK centimeter-level differential positioning (horizontal accuracy ≤0.05m, vertical accuracy ≤0.1m). At the same time, the visual inertial odometry (VIO) corrects the track deviation in real time, ensuring that the heading angle error is controlled within ±2° and the roll angle error is ≤1° throughout the flight.

[0042] The image acquisition system uses a dual-light camera triggering mechanism based on time synchronization: 1 second before reaching each waypoint, the flight control system sends a pre-trigger signal to the camera via the CAN bus, starting the image acquisition timer; when hovering at the waypoint, the infrared camera captures thermal images at a frame rate of 30Hz, and the visible light camera simultaneously captures RGB images at a frame rate of 120Hz. An embedded algorithm is used to achieve real-time sub-pixel registration of the dual-light images.

[0043] During the entire flight mission, the ground station receives real-time flight status data (update frequency ≥ 10Hz) and image histogram statistics through a 4G backup link, and verifies flight parameters every 30 seconds to ensure that the deviation between the actual flight parameters and the preset values ​​is within the allowable range.

[0044] (5) For the bifocal images obtained in steps (3) and (4), first perform geometric correction on the image based on the camera intrinsic parameter matrix, extrinsic parameters, and distortion coefficients accurately calculated by the Zhang Zhengyou calibration method in step (3) to verify the reprojection error of the correction feature points. When the error is lower than the set threshold (e.g., 0.5 pixels), the calibration result can be considered valid. The relevant formula is as follows:

[0045]

[0046] Where, e is the reprojection error, p i is the actual image coordinate, is the projection coordinate calculated based on the internal and external parameters, N is the total number of corner points of the calibration target;

[0047] The bi-optical image used for inspection is then dedistorted based on camera parameters to improve image quality. The dedistorted image is then evaluated using an image quality check algorithm, which assesses parameters such as clarity, contrast, and brightness. Blurred or overexposed images are automatically identified and removed to ensure the accuracy of subsequent analysis.

[0048] After completing the image dedistortion processing, the precise correspondence between the synchronously collected bi-optical images is established. Since the UAV triggers the bi-optical camera to collect data synchronously at each waypoint, there is theoretically a one-to-one corresponding image pair. However, considering the slight time deviation during the actual flight, a spatiotemporal synchronous registration method is used for double verification. First, a time synchronization layer is constructed: the PTP protocol is used to align the GPS timestamp, the synchronization error is ≤2ms, and a time compensation model Δt=α·t+β(α=1.0002,β=-0.12ms) is established; then, a spatial registration layer is constructed: the affine transformation matrix is ​​calculated based on the calibration parameters for initial registration, and the improved SURF algorithm is used to extract edge feature points for fine registration. The mismatched points are eliminated through RANSAC, and the target registration error is ≤1.5px; finally, a radiation consistency layer is constructed to establish a grayscale-temperature mapping relationship: T(x,y)=a·I vis (x,y)+b·I ir (x,y)+c, solve the coefficients (a,b,c) by the least squares method, and determine the coefficient R 2 ≥0.92.

[0049] Bi-optical images are classified based on building numbers, facades, and simultaneous capture relationships to construct a structured bi-optical image database for detection. The specific classification steps are as follows: First, the image data is classified at the first level according to the target building number (e.g., Building A, Building B, etc.), with each building number corresponding to a separate database folder. Second, within each building folder, a second level of classification is performed based on the building facade (four main facades: East E, South S, West W, and North N, as well as special facades such as roofs and local details). Each facade folder stores bi-optical images of that facade captured from different angles. Finally, within the facade folder, a third level of classification is performed based on the drone waypoint number during image acquisition (e.g., Waypoint 1, Waypoint 2, etc.). Each subfolder contains a complete set of corresponding infrared thermal images (IR) and visible light images (RGB). The image files are named according to the unified naming convention of "building number_facade_waypoint number_image type (IR or RGB)", for example, "A_E_3_IR.jpg". To facilitate subsequent data query and analysis, an image information management system based on a MySQL database was developed. This system stores metadata for each image in a relational database, enabling fast and accurate image retrieval and classification statistics through SQL queries. This database system supports data filtering based on any combination of criteria and automatically generates reports containing image paths, key parameters, and statistical information, providing an efficient and convenient data support platform for building thermal performance assessment and structural analysis.

[0050] (6) For the calibration bi-optical image obtained by dedistortion in step (5), a sub-pixel precision corner detection algorithm is used to detect the calibration target grid points as control points, and the pixel coordinates of the control points are located. Specifically, first, for the visible light image, the improved Foerstner operator is used to achieve sub-pixel positioning:

[0051]

[0052] In the formula The iteration termination condition is set as the coordinate change Δ≤0.01 pixels;

[0053] Then, for infrared image processing, a thermal gradient enhancement operator is designed:

[0054]

[0055] Where Z is the normalization coefficient, k = 3, σ = 1.5, which improves the corner point recognition rate in low-contrast areas to 92%;

[0056] Finally, the bispectral corner confidence evaluation function is established:

[0057]

[0058] Where a×b is the physical size of the checkerboard unit, and when C ≥ 0.85, it is confirmed as a valid control point;

[0059] Based on the known geometric relationships of the checkerboard calibration target, the ideal coordinate position of each corner point is pre-set in the calibration target's 3D world coordinate system. A one-to-one mapping relationship is then established between the pixel coordinates of the control points extracted from the bifocal image and the ideal coordinates of the calibration target. The pixel coordinates of the control points in the bifocal image are then converted to normalized image plane coordinates based on the desired intrinsic parameter matrix K for each bifocal camera. The coordinate conversion formula is:

[0060]

[0061] Where, P n is the normalized pixel plane coordinate, K is the camera intrinsic parameter matrix, (u, v) is the two-dimensional image coordinate;

[0062] Then, the rotation matrix and translation vector are solved by establishing the transformation relationship of the normalized coordinates of the bifocal cameras. The relative position relationship between the bifocal cameras is as follows:

[0063] P R =R·P I +t

[0064] Where, P R is the normalized coordinate matrix of the visible light camera, P Iis the normalized coordinate matrix of the infrared thermal camera, R and t are the rotation matrix and translation vector respectively;

[0065] Then, based on the above parameters and the projection plane normal vector and the distance from the projection plane to the camera optical center determined in advance by shooting the calibration target, the homography matrix describing the transformation relationship between the bi-optical image planes can be solved. The relevant formula of the homography matrix is ​​as follows:

[0066] P R =H·P I

[0067]

[0068] Where H is the homography matrix, K R is the visible light camera internal parameter matrix, K I is the intrinsic parameter matrix of the infrared thermal camera, R and t are the rotation matrix and translation vector, T is the coordinate matrix of the optical center of the infrared thermal camera relative to the visible light camera in the world coordinates, n is the normal vector of the calibration target projection plane, and d is the distance from the calibration target projection plane to the optical center of the infrared camera;

[0069] Finally, after obtaining the homography matrix H by calibrating the bi-optical image, the matrix can be used to realize the registration of infrared thermal images to visible light images.

[0070] (7) For the dedistorted visible light image obtained in step (5), a high-precision point cloud model of the target building is generated using structured light 3D reconstruction technology for 3D reconstruction. The specific operations are as follows:

[0071] Start professional 3D reconstruction software and import the classified and dedistorted visible light image, as well as the image's own EXIF ​​information (such as the image's shooting date, location, and camera parameters). The software automatically extracts feature points, performs image matching, and generates a 3D point cloud for the imported visible light image. Then, use the PMVS / CMVS software package to densely reconstruct the sparse point cloud to generate a dense point cloud model containing millions of points. The point cloud density is controlled at no less than 500 points per square meter to ensure the detail and accuracy of the model. During the point cloud optimization stage, voxel filtering and statistical filtering are used to remove noise points while retaining the building's edges and texture details. The final point cloud model has an accuracy of 2-3mm.

[0072] An orthophoto of the target building facade is extracted from the point cloud model. Using orthographic projection technology and an appropriate projection resolution, photogrammetry software (such as Photoscan or ContextCapture) is used to project the 3D point cloud data onto the 2D plane of the building facade to generate an orthophoto. During the orthophoto generation process, radiometric correction and color balancing are performed to ensure color consistency and texture clarity, while compensating for projection distortion. This ensures that the orthophoto accurately reflects the true geometry and texture characteristics of the building facade.

[0073] Building information modeling (BIM) software, Revit, was used to draw building elevation views. First, based on the building's design drawings and on-site measurement data, a 3D model of the building was created in Revit, including the main structure, exterior facade components, and decorative details. Standard front, side, and back elevation views were then extracted from this 3D model to ensure that the elevation views' dimensional accuracy and component details met construction industry standards. During the drawing process, layers, line types, and annotation styles were set strictly in accordance with building regulations, ensuring that the elevation views provided both technical guidance and facilitated subsequent image registration and analysis.

[0074] In order to achieve accurate alignment between building facade views and visible light images, this study uses a method based on Generalized Hough Transform (GHT) for image registration. The detailed process of the method is as follows:

[0075] First, feature descriptors were extracted from the BIM-generated facade view and the visible light image used for detection. These descriptors included geometric features (using the building outline Fourier descriptor with order n = 12), texture features (using the LBP-TOP operator with radius R = 3 and neighborhood P = 24), and semantic features (using the component spatial distribution histogram with bin number B = 16). Then, parts with clear geometric features, such as door and window frames, were selected from the building facade view as shape templates and described using parameters such as center point, width, height, and rotation angle. The feature parameter space was then defined as follows:

[0076]

[0077] Where x and y represent the center coordinates of the template, θ represents the rotation angle of the template, s represents the scale factor, and φ represents the perspective deformation parameter;

[0078] Then, Hough voting is used to map edge points in the visible light image to the template parameter space for accumulation. Peaks in the parameter space are detected to precisely locate the optimal position and shape of the template instance, achieving efficient feature matching. Finally, the homography matrix from the visible light image to the building facade view is calculated using the matched feature points, achieving registration between the visible light image and the building facade view.

[0079] (8) The dedistorted infrared thermal image obtained in step (5) is clustered using the K-means++ clustering algorithm. Specifically, the number of clusters k is set from 2 to 12 for clustering, and each k value is independently run 10 times, and the best result is taken to reduce the error caused by random initialization. In order to determine the optimal number of clusters, based on the clustering results corresponding to each k value, the three clustering effectiveness indicators of silhouette coefficient, Davis-Boulding index and variance ratio are comprehensively used for quantitative evaluation.

[0080] The silhouette coefficient (SC) is calculated as follows:

[0081]

[0082] Where N is the total number of samples in the data set, a(i) is the average distance within a cluster, and b(i) is the average distance between clusters of the nearest cluster;

[0083] The Davis-Boulding Index (DBI) is calculated as follows:

[0084]

[0085]

[0086] Where s i and s j Represents cluster C i and cluster C j The average distance between all sample points and their cluster centers, d i,j Representative cluster C i and cluster C j The cluster center distance, K represents the number of clusters;

[0087] The calculation formula of variance ratio (CHI) is as follows:

[0088]

[0089] In the formula, |C i | is the number of data points in cluster i, c i is the centroid of cluster i, c is the mean vector of the entire data set, and x is any data point in cluster i;

[0090] By comparing and analyzing the values ​​of these indicators, the optimal number of clusters is selected, and then the hollow areas in the image are preliminarily identified. To further improve the accuracy of hollow area identification, the convex hull algorithm is introduced to screen and optimize the preliminary identification results. The specific algorithm process is as follows:

[0091] Calculate the number of pixels in the target area of ​​the image and record it as the area of ​​the connected domain. Traverse each point on the contour of the area, and find the two endpoints A and B with the farthest distance as the poles based on the coordinate judgment, and connect them to form a convex hull segmentation line, dividing the entire convex hull into two upper and lower convex hulls. Then, find the points C and D farthest from the straight line AB in the upper and lower convex hulls as new poles, connect the four points A, B, C, and D, and repeat this operation on the newly divided convex hull until no new poles can be found. Connect all the poles to obtain the minimum circumscribed convex polygon of the area, that is, the convex hull of the area. Calculate the number of pixels in it and record it as the convex hull area. The convex hull area ratio refers to the ratio of the area of ​​the connected domain to its minimum circumscribed convex polygon, and its calculation formula is as follows:

[0092]

[0093] Where R represents the convex hull area ratio, S t represents the area of ​​the connected domain, S c Represents the minimum circumscribed convex polygon area of ​​the connected domain;

[0094] The convex hull area ratio can be used to measure the compactness of the connected domain shape. Its value range is (0, 1]. The closer the area ratio is to 1, the more compact and regular the connected domain shape is. Conversely, the connected domain shape is more scattered and irregular. Finally, the correct hollow area identification result is determined, and the corresponding infrared thermal image is marked as a hollow infrared thermal image. Create a folder named "Hollow", add "_D" to the end of the image file name containing the hollow area, and store it in the folder, for example, "A_E_3_IR_D.jpg".

[0095] (9) For the hollowing infrared thermal image obtained in step (8), the homography matrix H calculated in step (6) is used to achieve accurate registration of the infrared thermal image and the visible light image. The specific operation is as follows:

[0096] First, a dual-spectral spatial mapping model is established:

[0097]

[0098] Where H is the homography matrix, ∈~N(0,∑H) is the covariance error;

[0099] Then, through matrix operations, the coordinates of each pixel in the infrared thermal image are converted to the corresponding coordinate position in the visible light image. Bilinear interpolation is used to resample the pixel values ​​to ensure the geometric consistency of the converted image. The resulting hollow visible light image is precisely aligned with the visible light image, achieving pixel-level fusion of the infrared thermal image and the visible light image.

[0100] Finally, based on the pixel coordinate system (PCS), the pixel coordinate range of the hollow area in the visible light image is calculated to provide a basis for subsequent spatial positioning. The specific operations are as follows:

[0101] Extract the pixel set of the hollow area in the hollow infrared thermal image after filtering by the convex hull algorithm Use the homography matrix to transform the coordinates of these pixel points to obtain the corresponding pixel coordinates in the visible light image Then, the minimum bounding rectangle of these coordinate points in the visible light image is calculated to determine the boundary pixel coordinates of the hollow area. Maximum-minimum coordinate positioning:

[0102]

[0103] Generate a minimum bounding rectangle:

[0104] ROI rect =([u min :u max , v min :v max ]

[0105] By calculating the width and height of the rectangle (in pixels), the pixel coordinate range of the hollow area is obtained. This range can be used for subsequent quantitative analysis and evaluation of the hollow area of ​​the building exterior wall.

[0106] (10) For the hollow visible light image obtained in step (9), the hollow area is accurately positioned in the building facade view based on the registration relationship between the visible light image and the building facade view established in step (7). Finally, the position of the hollow area is accurately drawn in the building facade view, completing the three-dimensional spatial positioning of the hollow area, providing intuitive visualization results for building structure inspection and maintenance.

[0107] The specific description of the present invention in the above embodiments is only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Technical engineers in this field may make some non-essential improvements and adjustments to the present invention based on the contents of the above invention, which fall within the scope of protection of the present invention.

Claims

1. An airborne infrared-visible dual-light combined method for identifying and locating hollowing in building exterior walls, comprising the following steps: (1) Based on the target building and its surrounding environment, and taking into account parameters such as detection distance and ground sampling distance, select a suitable drone, equip it with appropriate infrared and visible light cameras, and customize the drone's flight parameters such as route, altitude, and speed; (2) Design a calibration checkerboard with suitable size, use 3D printing technology, select two materials with significant contrast to print each unit separately, and form a checkerboard calibration target by precise splicing; (3) Fix the calibration checkerboard at a suitable position on the building's exterior wall. According to the detection distance determined in step (1), take infrared thermal images and visible light images of the checkerboard calibration target from multiple angles, which are recorded as calibration bi-optical images. Use Zhang Zhengyou's calibration method to calculate the camera's intrinsic parameters, extrinsic parameters, and distortion coefficients. (4) Place the drone in an open area, start the drone and fly it along the predetermined route, and simultaneously collect infrared thermal images and visible light images at each waypoint until the image collection task of the entire facade of the target building is completed; (5) Dedistorting the bi-optical images obtained in steps (3) and (4) according to the camera calibration parameters; establishing a correspondence between the synchronously collected bi-optical images, and classifying them step by step according to the building number and the facade to form a bi-optical image database for detection; (6) For the calibration bi-optical image obtained in step (5), a sub-pixel precision corner detection algorithm is used to detect the corner points of the calibration target as control points, and the pixel coordinates of the control points are located; based on the known geometric relationship of the calibration target, the correspondence between the bi-optical image control points is established, and the homography matrix is ​​calculated; using the homography matrix, the registration between the bi-optical images taken synchronously by the UAV is achieved; (7) generating a point cloud model of the target building using a three-dimensional reconstruction technique for the visible light image for detection obtained in step (5), and extracting an orthophoto of the target building facade from the point cloud model; Use Building Information Modeling (BIM) software to draw building facade views; use the generalized Hough transform (GHT) method to achieve registration between building facade views and visible light images; (8) For the infrared thermal image obtained in step (5), the K-means++ clustering algorithm is used to cluster the number of clusters from 2 to 12, and the three clustering effectiveness indicators of silhouette coefficient, Davis-Boulding index and variance ratio are comprehensively used for quantitative evaluation to determine the optimal number of clusters and the corresponding clustering results; further, the convex hull algorithm is used to screen the preliminary recognition results of the hollow area, determine the correct hollow area recognition results, and mark the corresponding infrared thermal image as the hollow infrared thermal image; (9) For the hollowing infrared thermal image obtained by processing in step (8), the hollowing area recognition result corresponding to the infrared thermal image is registered with the visible light image taken synchronously with the hollowing infrared thermal image using the homography matrix obtained in step (6) to obtain the hollowing visible light image; and the pixel coordinate range of the hollowing area is calculated according to the pixel coordinate system; (10) For the hollow visible light image obtained by processing in step (9), according to the registration relationship between the visible light image obtained in step (7) and the facade view of the building to which it belongs, the positioning of the hollow area in the building facade view is achieved, and the position of the hollow area is accurately drawn in the facade view to achieve accurate positioning of the hollow area.

2. The method for identifying and locating hollowing in building exterior walls using an airborne infrared-visible dual-light combination as claimed in claim 1, wherein: The following steps are involved: The relevant calculation formulas for the flight parameters described in step (1) are as follows: Based on the BIM model of the target building, the Delaunay triangulation algorithm is used to construct a three-dimensional mesh model of the building surface. The improved ant colony algorithm is used for path optimization. The objective function of the UAV route is: Where, L i is the length of the segment, v is the speed, E i is the energy consumption coefficient, E i It is the task time window constraint; Establishing the altitude-speed coupling control model: H=K·(GSD·f) / (s·N) v max =H·f·FR / (GSD·n) Where FR is the frame rate and n is the heading overlap ratio. The flight parameters are adjusted in real time by the PID controller to ensure that the heading overlap ratio is ≥80% and the side overlap ratio is ≥70%, and the sampling theorem requirement of λ / 4H ≤ 0.25 is met.

3. The method for identifying and locating hollowing in building exterior walls using an airborne infrared-visible dual-light combination as claimed in claim 1, wherein: Step (2) specifically includes: selecting polylactic acid (PLA) and matte aluminum as printing materials. These two materials have significant contrast characteristics in thermal imaging. Through this material combination, the produced checkerboard calibration target meets the high-precision calibration requirements and provides clear visual effects in thermal imaging applications; Develop a standard unit module with a mortise and tenon positioning structure, use laser cutting to produce ±5μm positioning holes, and use high-precision pneumatic clamps to achieve rapid assembly; design a two-degree-of-freedom adjustment mechanism to compensate for the dimensional deviation ΔL=α·L·ΔT (α=1.2×10 -5 / ℃).

4. The method for identifying and locating hollowing in building exterior walls using an airborne infrared-visible dual-light combination as claimed in claim 1, wherein: Step (3) specifically includes: Fix the calibration checkerboard at a suitable location on the building's exterior wall. The selected location must meet the following conditions: Ensure the plane of the calibration checkerboard is stable and vertical to avoid checkerboard flatness errors caused by uneven walls or improper installation; avoid interference from surrounding environmental factors during the image acquisition process to ensure image acquisition accuracy and stability; and consider the convenience of subsequent multi-angle shooting, so that the camera can smoothly focus and capture clear, complete checkerboard images at different angles. Based on the detection distance determined in step (1), infrared thermal images and visible light images of the checkerboard calibration target are captured from multiple angles and recorded as calibration bi-optical images. These captured images are uniformly labeled and stored to form a complete calibration bi-optical image dataset. The Zhang Zhengyou calibration method is used to calculate the camera's intrinsic parameters, extrinsic parameters, and distortion parameters. The Zhang Zhengyou calibration method's related formulas and parameter solution procedures are as follows: For visible light images, a sub-pixel Harris-Laplace corner detection algorithm is used with a positioning error of σ≤0.1pixel; for infrared images, a thermal gradient feature enhancement function is constructed. Combined with template matching to achieve corner point recognition, the matching success rate is ≥98%; Establish dual-spectrum joint calibration objective function: Where, is the internal parameter matrix, D = [k1, k2, p1, p2, k3] is the distortion coefficient vector, and the cross constraint term is introduced σ is the sensor noise variance.

5. The method for identifying and locating hollowing in building exterior walls using an airborne infrared-visible dual-light combination as claimed in claim 1, wherein: Step (5) specifically includes: For the bifocal images obtained in steps (3) and (4), firstly, the image is geometrically corrected based on the camera intrinsic parameter matrix, extrinsic parameters, and distortion coefficients accurately calculated by the Zhang Zhengyou calibration method in step (3) to verify the reprojection error of the corrected feature points. When the error is lower than the set threshold, the calibration result can be considered valid. The relevant formula is as follows: Where, e is the reprojection error, p i is the actual image coordinate, is the projection coordinate calculated based on the internal and external parameters, N is the total number of corner points of the calibration target; Then, based on the camera parameters, the bi-optical image used for inspection is dedistorted to improve image quality. The dedistorted image is then evaluated using an image quality inspection algorithm, which assesses parameters such as clarity, contrast, and brightness. Blurred or overexposed images are automatically identified and removed to ensure the accuracy of subsequent analysis. After image dedistortion, an accurate correspondence between the synchronously collected bi-optical images is established. Since the UAV triggers the bi-optical camera to collect data synchronously at each waypoint, there is theoretically a one-to-one corresponding image pair. However, considering the slight time deviation during the actual flight, a spatiotemporal synchronous registration method is used for double verification. First, a time synchronization layer is constructed: the PTP protocol is used to align the GPS timestamps, with a synchronization error of ≤2ms, and a time compensation model Δt=α·t+β is established, α=1.0002, β=-0.12ms; then, a spatial registration layer is constructed: the affine transformation matrix is ​​calculated based on the calibration parameters for initial registration, and the improved SURF algorithm is used to extract edge feature points for fine registration. RANSAC is used to eliminate mismatched points, and the target registration error is ≤1.5px; finally, a radiation consistency layer is constructed to establish a grayscale-temperature mapping relationship: T(x,y)=a·I vis (x,y)+b·I ir (x,y)+c, solve the coefficients (a,b,c) by the least squares method, and determine the coefficient R 2 ≥0.92; Bi-optical images are classified according to building number, facade, and simultaneous shooting relationship to construct a structured bi-optical image database for inspection. The specific classification steps are as follows: First, the image data is classified at the first level according to the target building number, with each building number corresponding to a separate database folder. Second, within each building folder, the image data is classified at the second level according to the building facade, and each facade folder stores bi-optical images of that facade acquired from different angles. Finally, within the facade folder, the image data is classified at the third level according to the waypoint number used during image acquisition. Each subfolder contains a complete set of corresponding infrared thermal images (IR) and visible light images (RGB). The image files are named according to the unified naming convention of "building number_facade_waypoint number_image type". At the same time, to facilitate subsequent data query and analysis, an image information management system based on a MySQL database was developed. The metadata of each image is stored in a relational database, and SQL query statements are used to achieve fast and accurate image retrieval and classification statistics. This database system supports data filtering according to any combination of conditions and automatically generates data reports containing image paths, key parameters, and statistical information, providing an efficient and convenient data support platform for building thermal performance assessment and structural analysis.

6. The method for identifying and locating hollowing in building exterior walls using an airborne infrared-visible dual-light combination as claimed in claim 1, wherein: Step (6) specifically includes: First, for visible light images, the improved Foerstner operator is used to achieve sub-pixel positioning: In the formula The iteration termination condition is set as the coordinate change Δ≤0.01 pixels; Then, for infrared image processing, a thermal gradient enhancement operator is designed: Where Z is the normalization coefficient, k = 3, σ = 1.5, which improves the corner point recognition rate in low-contrast areas to 92%; Finally, the bispectral corner confidence evaluation function is established: Where a×b is the physical size of the checkerboard unit, and when C ≥ 0.85, it is confirmed as a valid control point; Based on the known geometric relationship of the checkerboard calibration target, the ideal coordinate position of each corner point is pre-set in the calibration target's three-dimensional world coordinate system. Then, a one-to-one mapping relationship is established between the pixel coordinates of the control points extracted from the bi-optical image and the ideal coordinates of the calibration target. The pixel coordinates of the control points in the bi-optical image are converted to normalized image plane coordinates based on the required intrinsic parameter matrix K of each bi-optical camera. The coordinate conversion formula is: Where, P n is the normalized pixel plane coordinate, K is the camera intrinsic parameter matrix, and (u, v) is the two-dimensional image coordinate; Then, the rotation matrix and translation vector are solved by establishing the transformation relationship of the normalized coordinates of the bifocal cameras. The relative position relationship between the bifocal cameras is as follows: P R =R·P I +t Where, P R is the normalized coordinate matrix of the visible light camera, P I is the normalized coordinate matrix of the infrared thermal camera, R and t are the rotation matrix and translation vector respectively; Then, based on the above parameters and the projection plane normal vector and the distance from the projection plane to the camera optical center determined in advance by shooting the calibration target, the homography matrix describing the transformation relationship between the bi-optical image planes can be solved. The relevant formula of the homography matrix is ​​as follows: P R =H·P I Where H is the homography matrix, K R is the visible light camera internal parameter matrix, K I is the intrinsic parameter matrix of the infrared thermal camera, R and t are the rotation matrix and translation vector, T is the coordinate matrix of the optical center of the infrared thermal camera relative to the visible light camera in the world coordinates, n is the normal vector of the calibration target projection plane, and d is the distance from the calibration target projection plane to the optical center of the infrared camera; Finally, after obtaining the homography matrix H by calibrating the bi-optical image, the matrix can be used to realize the registration of infrared thermal images to visible light images.

7. The method for identifying and locating hollowing in building exterior walls using an airborne infrared-visible dual-light combination as claimed in claim 1, wherein: Step (7) specifically includes: Launch professional 3D reconstruction software and import the classified and dedistorted visible light image, along with the image's own EXIF ​​information. The software automatically extracts feature points from the imported visible light image, performs image matching, and generates a 3D point cloud. Then, using the PMVS / CMVS software package, densely reconstructs the sparse point cloud, generating a dense point cloud model containing millions of points. The point cloud density is controlled to no less than 500 points per square meter to ensure model detail and accuracy. During the point cloud optimization phase, voxel filtering and statistical filtering are used to remove noise points while preserving building edges and texture details. The resulting point cloud model achieves an accuracy of 2-3mm. An orthophoto of the target building facade is extracted from the point cloud model. Using orthographic projection technology and an appropriate projection resolution, photogrammetry software is used to project the 3D point cloud data onto the 2D plane of the building facade to generate an orthophoto. During the orthophoto generation process, radiometric correction and color balancing are performed to ensure color consistency and texture clarity, while also compensating for projection distortion, ensuring that the orthophoto accurately reflects the true geometry and texture characteristics of the building facade. Building information modeling (BIM) software, Revit, is used to draw building elevation views. First, based on the building's design drawings and on-site measurement data, a 3D model of the building is created in Revit, including the building's main structure, exterior facade components, and decorative details. Then, standard front, side, and back elevation views are extracted from this 3D model to ensure that the elevation views' dimensional accuracy and component details meet construction industry standards. During the drawing process, layers, line types, and annotation styles are set strictly in accordance with building specifications, ensuring that the elevation views provide both technical guidance and facilitate subsequent image registration and analysis. In order to achieve accurate alignment between the building facade view and the visible light image, this study uses a method based on the generalized Hough transform (GHT) for image registration. The detailed process of the method is as follows: First, feature descriptors are extracted from the BIM-generated facade view and the visible light image used for detection, including geometric features, texture features, and semantic features. The geometric features use the building outline Fourier descriptor with order n = 12, the texture features use the LBP-TOP operator with radius R = 3 and neighborhood P = 24, and the semantic features use the component space distribution histogram with bin number B = 16. Then, parts with clear geometric features, such as door and window frames, are selected from the building facade view as shape templates and described using parameters such as center point, width, height, and rotation angle. The feature parameter space is defined as follows: Where x and y represent the center coordinates of the template, θ represents the rotation angle of the template, s represents the scale factor, and φ represents the perspective deformation parameter; Then, the edge points in the visible light image are mapped to the template parameter space for accumulation through Hough voting, and the optimal position and shape of the template instance are accurately located by detecting the peak in the parameter space, thereby achieving efficient feature matching. Finally, the homography matrix from the visible light image to the building facade view is calculated through the matched feature points to achieve alignment between the visible light image and the building facade view.

8. The method for identifying and locating hollowing in building exterior walls using an airborne infrared-visible dual-light combination as claimed in claim 1, wherein: Step (8) specifically includes: Clustering was performed with the number of clusters k ranging from 2 to 12. Each k value was independently run 10 times, and the best result was selected to reduce the error caused by random initialization. To determine the optimal number of clusters, the clustering results corresponding to each k value were quantitatively evaluated using three clustering effectiveness indicators: silhouette coefficient, Davis-Boulding index, and variance ratio. The calculation formula of the silhouette coefficient SC is as follows: Where N is the total number of samples in the data set, a(i) is the average distance within a cluster, and b(i) is the average distance between clusters of the nearest cluster; The Davis-Boulding Index (DBI) is calculated as follows: Where s i and s j Represents cluster C i and cluster C j The average distance between all sample points and their cluster centers, d i,j Representative cluster C i and cluster C j The cluster center distance, K represents the number of clusters; The calculation formula of variance ratio (CHI) is as follows: In the formula, |C i | is the number of data points in cluster i, c i is the centroid of cluster i, c is the mean vector of the entire data set, and x is any data point in cluster i; By comparing and analyzing the values ​​of these indicators, the optimal number of clusters is selected, and then the hollow areas in the image are preliminarily identified. To further improve the accuracy of hollow area identification, the convex hull algorithm is introduced to screen and optimize the preliminary identification results. The specific algorithm process is as follows: Calculate the number of pixels in the target area of ​​the image and record it as the connected domain area. Traverse each point on the contour of the area and find the two endpoints A and B with the greatest distance from each other as the extreme points based on the coordinates. Connect them to form a convex hull segmentation line, dividing the entire convex hull into upper and lower convex hulls. Then, find the points C and D farthest from the line AB in the upper and lower convex hulls as new extreme points. Connect points A, B, C, and D, and repeat this operation on the newly divided convex hulls until no new extreme points can be found. Connect all the extreme points to obtain the minimum circumscribed convex polygon of the area, that is, the convex hull of the area. Calculate the number of pixels in it and record it as the convex hull area. The convex hull area ratio refers to the ratio of the area of ​​the connected domain to its minimum circumscribed convex polygon. The calculation formula is as follows: Where R represents the convex hull area ratio, S t represents the area of ​​the connected domain, S c Represents the minimum circumscribed convex polygon area of ​​a connected domain; The convex hull area ratio can be used to measure the compactness of the connected domain shape. Its value range is (0,1]. The closer the area ratio is to 1, the more compact and regular the connected domain shape is. Conversely, the connected domain shape is more scattered and irregular. Finally, the correct hollow area identification result is determined, and the corresponding infrared thermal image is marked as a hollow infrared thermal image. Create a folder named "Hollow", add "_D" to the end of the image file name containing the hollow area, and store it in this folder.

9. The method for identifying and locating hollowing in building exterior walls using an airborne infrared-visible dual-light combination as claimed in claim 1, wherein: Step (9) specifically includes: First, a dual-spectral spatial mapping model is established: Where H is the homography matrix, ∈~N(0,∑H) is the covariance error; Then, through matrix operations, the coordinates of each pixel in the infrared thermal image are converted to the corresponding coordinate position in the visible light image, and the pixel values ​​are resampled using bilinear interpolation to ensure the consistency of the geometric position of the converted image. The hollow visible light image is precisely aligned with the visible light image, achieving pixel-level fusion of the infrared thermal image and the visible light image. Finally, based on the pixel coordinate system PCS, the pixel coordinate range of the hollow area in the visible light image is calculated to provide a basis for subsequent spatial positioning. The specific operations are as follows: Extract the pixel set of the hollow area in the hollow infrared thermal image after filtering by the convex hull algorithm Use the homography matrix to transform the coordinates of these pixel points to obtain the corresponding pixel coordinates in the visible light image Then, the minimum bounding rectangle of these coordinate points in the visible light image is calculated to determine the boundary pixel coordinates of the hollow area. Maximum-minimum coordinate positioning: Generate a minimum bounding rectangle: ROD rect =[u min :u max ,v min :v max ] By calculating the width and height of the rectangle, the pixel coordinate range of the hollow area is obtained, which is used for subsequent quantitative analysis and evaluation of the hollow area of ​​the building exterior wall.

Images