A building facade defect intelligent detection method and related equipment
By using multimodal image fusion and 3D localization technology, the problems of poor single-modal detection and lack of 3D information in existing building facade inspection are solved, achieving efficient and accurate defect identification and localization, generating structured reports suitable for building safety management.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SOUTH CHINA UNIV OF TECH
- Filing Date
- 2026-01-21
- Publication Date
- 2026-05-29
AI Technical Summary
Existing building facade inspection technologies suffer from several drawbacks: single-modal image detection is easily affected by lighting and weather, lacks effective cross-modal image registration and fusion, cannot simultaneously and comprehensively detect surface and internal defects, and lacks three-dimensional spatial information, resulting in low inspection efficiency, insufficient accuracy, and high safety risks.
A drone equipped with a visible light camera and an infrared thermal imager was used to acquire multimodal images. A three-dimensional point cloud model was constructed by combining structured beam recovery algorithm and oblique photogrammetry. Image registration and fusion were performed by a two-stage adversarial network and a Swing Transformer image fusion network. Defects were identified using the YOLOv11 target detection model, and the two-dimensional detection results were mapped to three-dimensional space.
It achieves high-precision, automated inspection of building facades, can simultaneously identify surface and internal defects, and provides accurate three-dimensional spatial positioning, improving inspection efficiency and accuracy, generating structured reports suitable for building safety management.
Smart Images
Figure CN122116122A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of building structural health monitoring and intelligent detection technology, and in particular to an intelligent detection method and related equipment for building facade defects. Background Technology
[0002] With the deepening of urbanization, many existing buildings, especially older ones, are prone to defects such as cracks, hollow areas, and peeling of the finishing layer on their facades due to material aging and environmental impacts. If these defects are not detected and addressed in a timely manner, they will seriously affect the aesthetics of the buildings and may even endanger public safety. Currently, traditional defect detection methods mainly rely on manual climbing or using detection equipment for close-range inspections, which suffers from problems such as low efficiency, limited detection coverage, strong subjectivity, high safety risks, and difficulty in quantifying, recording, and locating defects.
[0003] In recent years, the development of drone inspection technology and computer vision technology has provided new solutions for building facade inspection. Drones equipped with visible light cameras can acquire high-resolution images to detect visually obvious defects such as surface cracks; those equipped with infrared thermal imagers can utilize differences in material thermal conductivity to detect hidden defects such as internal voids and leaks. However, existing technologies still have significant shortcomings: 1) Limitations of single-mode imaging: Using only visible light or infrared single-mode images cannot simultaneously and comprehensively and effectively capture surface and internal defects, and the detection effect is easily affected by factors such as lighting, weather, and wall material.
[0004] 2) Poor fusion quality: The lack of effective cross-modal image registration and fusion methods leads to mixed features in the fused image, which fails to highlight the comprehensive features of the defective area and affects the accuracy of subsequent recognition.
[0005] 3) Lack of three-dimensional spatial information: Existing methods mostly rely on two-dimensional image-based defect identification, which cannot provide the precise three-dimensional spatial location of defects on building facades, making it difficult to directly guide subsequent repair construction and quantitative management. Summary of the Invention
[0006] The main objective of this application is to propose an intelligent detection method, electronic device, storage medium, and program product for building facade defects based on multimodal image fusion and three-dimensional positioning. By deeply integrating UAV multimodal data acquisition, advanced image registration and fusion, deep learning defect recognition, and three-dimensional geometric positioning technology, it achieves automated, high-precision, and quantifiable intelligent detection and positioning of facade defects.
[0007] To achieve the above objectives, one aspect of this application proposes an intelligent detection method for defects in building facades, the method comprising: S1. Using a drone equipped with a high-resolution visible light camera and an infrared thermal imager, the building facade is photographed simultaneously to obtain spatially consistent RGB and infrared images. S2. Using the RGB image, construct a three-dimensional point cloud model of the building facade based on the Structure from Motion (SfM) algorithm and oblique photogrammetry. S3. The RGB image and the infrared image are preprocessed by using a two-stage adversarial network (TAN) based on mode transformation for registration. S4. Input the registered RGB image and infrared image into the image fusion network based on Swin Transformer to generate a fused image that combines visible light and thermal features; S5. Input the fused image into the trained YOLOv11 target detection model to achieve automatic identification of three types of building facade defects: cracks, hollow areas, and peeling, and obtain two-dimensional defect target location information. S6. Map the two-dimensional defect target location onto the three-dimensional point cloud model, and calculate the coordinates of the defect target center point in three-dimensional space based on the image registration parameters. .
[0008] S7. Display different defect types in the form of visual annotations in the 3D point cloud model, and automatically generate a structured report containing defect number, type, 3D coordinates and GPS information.
[0009] In some embodiments, step S2 includes: S21. Obtain multi-view RGB images covering all areas of the facade by using drones to conduct oblique photography of the building facade from multiple angles. S22. Use the SfM algorithm to extract key feature points in the image and match the same physical point across images to restore the three-dimensional spatial structure outside the camera and generate a sparse three-dimensional point cloud model. S23. A multi-view stereo matching algorithm is used to generate a dense three-dimensional point cloud model based on sparse point cloud and image information. S24. Filter, register and scale normalize the three-dimensional point cloud model to ensure that the spatial accuracy meets the defect location requirements.
[0010] In some embodiments, step S22 includes: Extract key point features from RGB images and perform cross-image matching to determine the correspondence between images; Based on the matching relationship, calculate the essential matrix E:
[0011] in, , These are the normalized pixel coordinates; The essential matrix is decomposed to obtain the relative rotation matrix R and translation vector t between the cameras; Linear triangulation is performed on the matching feature points to recover their three-dimensional coordinates. Generate a sparse 3D point cloud model: ,
[0012] in, The camera projection matrix is shown in two viewpoints. As a scale factor, These are three-dimensional spatial coordinates.
[0013] In some embodiments, step S23 includes: Based on the camera intrinsic and extrinsic parameter information obtained by the SfM algorithm, a disparity correlation is established between image pairs. For each pixel in the reference image, the best matching point is searched along the epipolar direction in the corresponding auxiliary image, and the disparity of that point under different viewpoints is calculated. Based on parallax and camera parameters, the depth information of each pixel is estimated using an inverse triangulation method, and its three-dimensional coordinates satisfy: , ,
[0014] in, Image pixel coordinates, Main point location, X represents the camera focal length, and X, Y, Z represent the coordinates of a point in three-dimensional space. The depth maps generated from multi-view images are fused, error points are removed, and a dense 3D point cloud is generated.
[0015] In some embodiments, step S3 includes: RGB and infrared images taken at the same location were selected as registration pairs. The first stage uses a generative adversarial network to learn the mapping relationship between RGB images and infrared images, and generates an approximate infrared image based on the RGB image, represented as:
[0016] Where G is the generator network. It is an RGB image. It generates an image that approximates the infrared spectrum; The second stage achieves fine image registration through a Spatial Transformer Network (STN). The STN consists of three parts: a local network, a mesh generator, and a sampler. Local network: Extracts key registration parameters from input image pairs and predicts transform parameters; Mesh generator: Generates a grid of sampling point coordinates on the target image based on the transformation parameters output by the local network; Sampler: Based on the coordinate grid, resample pixel values from the source image and output an aligned and registered image.
[0017] In some embodiments, step S4 includes: Convolution operations are performed on the registered RGB image and infrared image respectively to obtain the initial feature map; The initial feature map is segmented into fixed-size image patches, which are then linearly embedded and input into a multi-layer Swin Transformer module. The Swin Transformer module uses a fixed-window-based multi-head self-attention mechanism (W-MSA) to extract local region features, and then uses a window-moving mechanism (SW-MSA) to realize feature interaction and context modeling between different windows; By using a multi-level feature encoding structure and the Patch Merging module for downsampling, multi-scale feature representations of the image are extracted. In each stage, the Patch Merging module concatenates, linearly maps, and downsamples adjacent image patches to reduce the spatial dimension of the image and increase the number of channels for feature representation. A channel attention mechanism is used to perform weighted fusion of corresponding feature maps of RGB and infrared images to enhance the responsiveness of thermal features to defect areas; The image is restored to its original resolution using an upsampling module, resulting in a high-quality fused image.
[0018] In some embodiments, step S5 includes: The system acquires RGB and infrared images of the building facade to be inspected, performs image registration and fusion processing, and generates a fused image containing multimodal information. The fused image is input into the trained YOLOv11 target detection model, which outputs the category, confidence level, and two-dimensional defect location information of each defect target in the image.
[0019] In some embodiments, training a YOLOv11 object detection model includes: Based on the fused images, a defect dataset was constructed. Three types of building facade defects, namely cracks, hollows, and peeling, were manually labeled in the images, and the dataset was divided into a training set and a validation set. During training, data augmentation strategies such as random scaling, rotation, flipping, and color perturbation are employed. The YOLOv11 model is trained using the training set as input. The loss function uses CIoU loss for bounding box regression and Focal Loss to handle class imbalance problems in classification tasks. After the model is trained, its performance is evaluated using a validation set. Evaluation metrics include mean accuracy (mAP), precision, and recall.
[0020] In some embodiments, step S6 includes: Extract the center pixel coordinates (u, v) of the bounding box of the defect target output by the YOLOv11 model in the fused image. Based on the intrinsic parameter matrix of the corresponding image and the extrinsic parameters of the camera pose, the pixel coordinates (u, v) are back-projected into the corresponding camera coordinate system to generate ray directions, including: , , ,
[0021] Where: K is the camera intrinsic parameter matrix. Let be the direction vector of the ray in the camera coordinate system, and R and t be the camera's rotation matrix and translation vector, respectively. The position of the camera center in the world coordinate system. It is the unit vector along the ray direction. As a scale factor, Let be the spatial coordinates of any three-dimensional point on the ray, i.e., the possible location of the defect center in three-dimensional space. The ray Spatial cross-calculation is performed with the 3D point cloud model to obtain the estimated position of the defect target in 3D space; The estimated location is scaled and transformed to output the three-dimensional spatial coordinates (x, y, z) of the defect center point.
[0022] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.
[0023] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.
[0024] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the method described above.
[0025] Compared with the prior art, the beneficial effects of this application are as follows: 1) High comprehensiveness of detection: By integrating visible light and infrared thermal imaging information, it can simultaneously and effectively detect surface cracks and internal defects such as hollowness and peeling on the building facade, overcoming the limitations of single-modal detection.
[0026] 2) High fusion and recognition accuracy: An advanced two-stage adversarial network is used for cross-modal image registration, combined with a Transformer-based fusion network, which significantly improves the quality of the fused image and the saliency of defect features; the latest target detection model is used for recognition, ensuring the accuracy of defect classification and localization.
[0027] 3) Achieve precise 3D positioning: Innovatively combining 2D image detection results with high-precision 3D point cloud models, through geometric back projection and cross calculation, the precise 3D positioning of defects in the real-world coordinate system is achieved, providing directly usable spatial information for subsequent maintenance.
[0028] 4) High automation and practicality: The entire process, from data acquisition to report generation, is highly automated. The generated structured report includes information such as defect type, 3D coordinates, and GPS, which greatly improves detection efficiency and the convenience of engineering applications. It is suitable for large-scale and intelligent building safety management. Attached Figure Description
[0029] Figure 1 This is a flowchart of an intelligent detection method for building facade defects in an embodiment of this application.
[0030] Figure 2 This is an overall flowchart of the intelligent detection method for building facade defects based on multimodal image fusion and three-dimensional positioning in the embodiments of this application.
[0031] Figure 3 This is a schematic diagram of the hardware structure of the electronic device in the embodiments of this application. Detailed Implementation
[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.
[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0034] With the advancement of urbanization, a large number of existing old buildings are gradually showing varying degrees of structural aging and surface defects, such as cracks, hollow areas, and surface peeling. If these defects are not detected and addressed in a timely manner, they will pose a serious threat to the safety of the buildings. Currently, traditional methods for detecting facade defects usually rely on manual on-site inspections, which suffers from problems such as low efficiency, strong subjectivity, insufficient accuracy, and safety hazards.
[0035] In recent years, the development of UAV imaging technology, infrared thermal imaging technology and computer vision algorithms has provided new possibilities for the intelligent detection and precise location of building defects. For example, UAVs can easily acquire large-scale high-resolution image data of buildings, while infrared thermal imaging can identify thermal features such as hollowness and detachment inside buildings. However, existing technologies have the following shortcomings: (1) Single-modal images cannot simultaneously take into account the effective detection of surface cracks and internal defects, and the detection accuracy of single-modal data is greatly affected by environmental conditions; (2) There is a lack of effective methods for multimodal image fusion and registration, resulting in low image feature fusion quality and difficulty in obtaining high-quality fused images that simultaneously possess structural and thermal features; (3) Currently used two-dimensional image detection methods cannot accurately locate defects in three-dimensional space, resulting in low accuracy and efficiency of subsequent repair work.
[0036] Therefore, there is an urgent need for an intelligent detection method that can simultaneously identify defects on the exterior and interior surfaces of a building and accurately achieve three-dimensional spatial positioning to meet practical application requirements.
[0037] In view of this, this application provides an intelligent detection method, electronic device, storage medium, and program product for building facade defects based on multimodal image fusion and 3D localization. This solution fully leverages the advantages of visible light images and infrared thermal imaging through multimodal image fusion technology to achieve comprehensive and high-precision identification of building facade defects. The image fusion method based on the Swing Transformer network significantly improves the quality of the fused images and enhances the ability to identify defect features. By combining image registration technology with a 3D point cloud model, precise 3D spatial localization of defects is achieved, greatly improving the efficiency and accuracy of subsequent repairs. This method has a high degree of automation and reliable detection accuracy, making it suitable for large-scale and intelligent safety management of urban building facades and possessing broad practical application prospects.
[0038] The intelligent detection method for building facade defects provided in this application relates to the field of building structural health monitoring and intelligent detection technology. The intelligent detection method for building facade defects provided in this application can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application that implements the intelligent detection method for building facade defects, but is not limited to the above forms.
[0039] Figure 1 This is an optional flowchart of an intelligent detection method for building facade defects provided in an embodiment of this application. Figure 1 The method may include, but is not limited to, steps S101 to S107.
[0040] Step S101: Simultaneously acquire RGB and infrared images of the building facade using a drone equipped with a visible light camera and an infrared thermal imager. Step S102: Using the RGB image, construct a three-dimensional point cloud model of the building facade based on the structure-in-motion algorithm and oblique photogrammetry. Step S103: A two-stage adversarial network based on modality conversion is used to perform registration preprocessing on the RGB image and the infrared image to obtain spatially aligned multimodal image pairs; Step S104: Input the registered RGB image and the infrared image into an image fusion network based on the Transformer architecture to generate a fused image that integrates visible light features and thermal features; Step S105: Input the fused image into the pre-trained target detection model to identify defects in the building facade and obtain their two-dimensional location information; Step S106: Map the two-dimensional location information onto the three-dimensional point cloud model, and determine the coordinates of the defect in three-dimensional space through back projection and spatial cross calculation; Step S107: Generate a visual inspection report based on the three-dimensional coordinates and type information of the defect.
[0041] The solutions of the embodiments of this application will be described in detail and explained below with reference to specific application examples.
[0042] like Figure 2 As shown, this embodiment discloses an intelligent detection method for building facade defects based on multimodal image fusion and three-dimensional localization, including the following steps: Step 1: Multimodal data acquisition and preprocessing.
[0043] In one embodiment, step 1 specifically includes steps 1.1-1.2: Step 1.1: Simultaneously capture images of the building facade using a drone equipped with a high-resolution visible light camera and an infrared thermal imager, obtaining paired RGB and infrared images. Specifically: 1) Before data acquisition, thorough experimental preparation is necessary to improve the effectiveness of infrared detection. This includes the selection of facade type, clear definition of facing brick material and defect characteristics, appropriate selection of experimental instruments, and the design and prefabrication of the experimental wall. By pre-setting hollow defects of different areas and depths, the influence of factors such as facade color, material, thickness, detection distance, angle, and detection time on the infrared detection effect of hollow defects is studied. The characteristics of hollow defects in infrared images and the optimal detection conditions are clarified to ensure the accuracy and reliability of subsequent actual data acquisition.
[0044] 2) During data collection, all collection work was carried out under the optimal detection conditions established in the above embodiments to ensure that the collected hollow infrared data has high accuracy and reliability. The collection object was the exterior facade of old campus buildings. At the same time, hollow defects on the building facade were detected by tapping with a hollow hammer in advance, and the areas with hollow defects were marked. Then, a DJI MAVIC2 (Industry Advanced Edition) drone equipped with both visible light camera and infrared camera was used to simultaneously shoot the building facade, and a large number of pairs of RGB images and infrared images were collected.
[0045] Step 1.2: For the acquired RGB image and infrared image pairs, a two-stage adversarial network (TAN) based on mode transformation is used for preprocessing and registration. The specific implementation is as follows: In the first stage, a mode conversion from RGB to infrared images is achieved using a Generative Adversarial Network (GAN) to improve the accuracy and robustness of subsequent spatial registration. The generator network G takes an RGB image as input. Generate approximate infrared images :
[0046] This stage uses an adversarial loss function for network training, specifically:
[0047] Where D is the discriminant network, used to discriminate the generated image. Compared with real infrared images The authenticity of the generated images is improved through GAN training and optimization. The feature distribution of a real infrared image is gradually approximated; G is the generator network. For real infrared images, For true RGB images, For true infrared images Data distribution For true RGB images Data distribution.
[0048] The second stage involves fine-grained spatial registration of RGB and infrared image pairs using a Spatial Transformer Network (STN). The STN comprises three modules: a local network, a mesh generator, and a sampler. (1) Localization Network: Generates approximate infrared images from the input through a convolutional network. and real infrared images Extracting affine transformation parameters required for spatial transformation .
[0049] (2) Grid Generator: Based on the transformation parameters output by the local network Generate the corresponding coordinate sampling grid points on the target image:
[0050] in,( ) represents the coordinates of the source image (to generate approximate infrared image coordinates), ( ) represents the target image coordinates (the coordinates of the real infrared image to be aligned). Let be the affine transformation matrix. This represents the parameters in the affine transformation matrix.
[0051] (3) Sampler: Based on the coordinate grid output by the grid generator, the sampler uses bilinear interpolation to sample pixel values from the source image to generate precisely aligned RGB and infrared image pairs.
[0052] in, These are the pixel values of the source image (approximately an infrared image). The pixel values of the output registered image.
[0053] Step 2: Using the registered RGB images, generate a 3D point cloud model of the building facade through the Structure from Motion (SfM) algorithm and oblique photogrammetry.
[0054] In one embodiment, step 2 specifically includes steps 2.1-2.5: Step 2.1: Multi-view image acquisition, including: Using a drone equipped with a visible light camera, the building facade is photographed. RGB images are captured from multiple different perspectives (with shooting angles covering at least the tilt range of 0° to 45°) around the building using oblique photography. Each area is covered by more than 3 images from different perspectives to ensure the reliability of subsequent spatial feature point matching and the accuracy of spatial positioning.
[0055] Step 2.2: Key feature point extraction and matching, including: The Structure from Motion (SfM) algorithm is used to extract stable and easily matched feature points from multi-view images. Feature descriptors are then used for cross-image matching, and mismatched points are removed. Matched feature point pairs are denoted as... These correspond to the normalized image coordinates from two different perspectives.
[0056] Step 2.3: Camera pose and sparse point cloud computing to obtain a preliminary sparse 3D point cloud model, including: Using matching point pairs Calculate the essential matrix E:
[0057] Here, E is the essential matrix, containing the relative motion information between the two viewpoint cameras. The relative rotation matrix R and translation vector t between the cameras are obtained by performing singular value decomposition (SVD) on the essential matrix. The specific steps are as follows:
[0058] Where U and V are derived from the singular value decomposition of matrix E, and W is a predefined matrix:
[0059] Then, triangulation was performed on the matching points to calculate the corresponding three-dimensional coordinates X: ,
[0060] in, The camera projection matrix is shown in two viewpoints. As a scale factor, These are three-dimensional spatial coordinates.
[0061] Step 2.4: Multi-view stereo matching and dense point cloud generation, including: Based on the camera intrinsic and extrinsic parameter information obtained by the SfM algorithm, a high-density dense 3D point cloud is generated using the Multi-View Stereo (MVS) algorithm, including: Using a reference image from a certain viewpoint, several auxiliary viewpoint images are selected. For each pixel in the reference image, the best matching point is searched on the auxiliary images along its corresponding epipolar line. By calculating the disparity between the best matching point and different viewpoints, the depth value of that pixel is further calculated. The formula for calculating the 3D coordinates is: , ,
[0062] in, Image pixel coordinates, Main point location, X represents the camera focal length, and X, Y, Z represent the coordinates of a point in three-dimensional space. By performing the above operations on all pixels, a dense 3D point cloud from a reference viewpoint is obtained. The point clouds from multiple reference viewpoints are then fused to generate a comprehensive dense 3D point cloud model.
[0063] Step 2.5: Point cloud model filtering, registration, and scale normalization, including: Dense 3D point clouds are filtered to remove noise caused by disparity calculation errors. The Iterative Closest Point (ICP) algorithm is used to achieve fine-grained registration of different local point cloud fragments. Then, the point cloud is scaled using a scale normalization method to conform to the actual spatial dimensions. The specific scale transformation formula is as follows:
[0064] in, The original point cloud coordinates, s represents the point cloud coordinates after scale normalization, s is the scale factor, and t is the translation vector.
[0065] Step 3: Multimodal image fusion.
[0066] In one embodiment, step 3 specifically includes steps 3.1-3.6: Step 3.1: Convolutional feature extraction, including: Initial feature extraction is performed on the registered RGB and infrared images from step 1. Specifically, convolutional layers are used to perform convolution operations to capture the primary features of the images:
[0067] In the formula, The pixel values of the input image. Here, b represents the convolution kernel parameters, and b represents the bias term. It is a non-linear activation function. This is a convolutional feature map.
[0068] Step 3.2: Image patch segmentation and linear embedding, including: The RGB and infrared feature maps obtained from convolution are divided into 4×4 pixel image patches. Linear embedding is performed on each patch to convert the two-dimensional image patch into a one-dimensional feature vector.
[0069] in, For the i-th image patch, This means flattening the image patch into a one-dimensional vector. and These are the weight matrix and bias vector for linear embedding, respectively. The feature vectors are linearly embedded.
[0070] Step 3.3: Feature extraction from the Swin Transformer module, including: The linearly embedded feature vector is input into the Swin Transformer module for window-based multi-head self-attention feature extraction. The Swin Transformer module employs a window-based multi-head self-attention (W-MSA) mechanism and a shifted window-based multi-head self-attention (SW-MSA) mechanism to achieve local feature extraction and cross-window information interaction. Specifically, for the feature vector set The formula for calculating the multi-head self-attention mechanism is as follows:
[0071] in These are projection matrices for the query, key, and value, respectively. This is the corresponding weight matrix. Z represents the dimension of a single attention head, and Z is the input feature matrix.
[0072] Step 3.4: Patch Merging module performs multi-scale feature encoding and downsampling, including: After feature extraction by the Swin Transformer, the Patch Merging module is used to stitch together image patches, perform linear mapping, and downsampling to obtain multi-scale feature representations.
[0073] in, The features of the i-th adjacent image patch within the i-th layer, Indicates the concatenation of features. Represents a linear mapping. The downsampled features are obtained after concatenation and linear mapping. This is the bias vector.
[0074] Step 3.5: Feature fusion of channel attention mechanism, including: The feature representations of RGB and infrared images are weighted and fused using a channel attention mechanism to enhance feature representation capabilities. Specifically, channel attention (CA) is employed.
[0075] Where F is the input feature map, , These represent average pooling and max pooling operations, respectively. MLP stands for Multilayer Perceptron. This is the Sigmoid activation function. The process of fusing RGB and infrared features is as follows:
[0076] Step 3.6: Upsampling to generate the final fused image, including: The fused feature map is then upsampled to the original resolution of the input image using an upsampling module. This upsampling is performed using bilinear interpolation.
[0077] In the formula, Let (x, y) represent the features in the fused feature map, where (x, y) are the coordinates of the upsampled image, and (m, n) are the coordinates of the original fused feature map. This represents the upsampling ratio. The final result is a high-quality fused image that simultaneously possesses RGB and infrared features.
[0078] Step 4: Defect identification based on fused images.
[0079] In some embodiments, step 4 specifically includes steps 4.1-4.4: Step 4.1: Construct a fused image defect dataset. Using the fused images obtained in Step S3, construct a defect dataset and manually label the three typical facade defects—cracks, hollow areas, and detachments—in the fused images. The labeling uses a common label format in the target detection field, generating a label information file containing the defect category and bounding box coordinates. Specifically, each defect target is labeled with a category number and the pixel coordinates of the upper left and lower right corners of the bounding box.
[0080] Step 4.2: Adopt data augmentation strategies. To improve the generalization ability and detection performance of the YOLOv11 model, random data augmentation is performed on the defect image dataset during training. This includes operations such as random scaling, rotation, horizontal flipping, vertical flipping, color perturbation, and brightness changes. (1) Random scaling and rotation operations:
[0081] in, The coordinates of the original image are... Let be the coordinates of the transformed image, and s be a random scaling factor. The rotation angle is random.
[0082] (2) Random flipping mirrors the image by using the horizontal or vertical axis.
[0083] (3) Random color perturbation and brightness change are implemented by randomly scaling and translating the image channels to simulate different shooting conditions.
[0084] Step 4.3: Train the YOLOv11 object detection model. Using the constructed augmented dataset as input, train the YOLOv11 object detection model for defect detection. During training, considering the size and distribution characteristics of defects on the building facade, the CIoU (Complete Intersection over Union) loss function is selected for bounding box regression optimization.
[0085] in, For intersection-union comparison, b is the center of the predicted bounding box. Centered at the true bounding box Let be the Euclidean distance, and c be the diagonal distance of the smallest bounding box containing both the predicted and ground truth boxes. This is a term indicating consistency in aspect ratio.
[0086] To alleviate the defect class imbalance problem and improve the detection performance of small target defects, Focal Loss is selected as the classification loss function:
[0087] In the formula, , For hyperparameters, This represents the probability that the model predicts the true class.
[0088] After model training is complete, the model performance is evaluated using a validation set. Evaluation metrics include mean AP, precision, and recall. Precision:
[0089] Recall:
[0090] Average accuracy (mAP):
[0091] Where TP represents the number of true positives, FP represents the number of false positives, and FN represents the number of false negatives. Let N be the average accuracy of the i-th type of defect target, and N be the number of defect categories.
[0092] Step 4.4: Defect Detection and Result Output. The fused image of the building facade to be inspected is input into the trained YOLOv11 model. The model automatically identifies and outputs the categories, detection confidence scores, and corresponding bounding box coordinates of three types of defects in the fused image: cracks, hollow areas, and detachment. The model output can be represented as follows:
[0093] in, Let k be the defect category of the k-th target. The confidence level for the defect detection. and These are the top-left and bottom-right coordinates of the target bounding box, respectively.
[0094] In this embodiment, the model achieved a high accuracy rate in the overall identification of various facade defects, with an overall accuracy rate exceeding 70%, demonstrating good model performance. Step 5: Map the two-dimensional defect to the three-dimensional model.
[0095] In one embodiment, step 5 specifically includes steps 5.1-5.6: Step 5.1: Extraction of 2D Defect Center Coordinates. Using the defect detection results output by the YOLOv11 target detection model obtained in step S4, extract the center pixel coordinates of the bounding box of each defect target in the fused image. :
[0096] in, and These represent the coordinates of the top left and bottom right corners of the defect bounding box detected by the YOLOv11 model, respectively.
[0097] Step 5.2: Determining Camera Intrinsic and Extrinsic Parameters. Using the camera intrinsic and extrinsic parameters obtained during the construction of the 3D point cloud model in step S2, a two-dimensional coordinate back projection of the defect center is performed. The camera intrinsic parameter matrix K is represented as:
[0098] In the formula, , These are the camera's focal lengths. These are the coordinates of the camera's principal point.
[0099] The camera's extrinsic parameters, including the rotation matrix R and the translation vector t, can be expressed as:
[0100] Step 5.3: Backproject the two-dimensional coordinates of the defect center onto the three-dimensional ray, based on the two-dimensional defect center coordinates. Using the camera intrinsic parameter matrix K and camera extrinsic parameters R and t, calculate the equation of the back-projected ray of the defect center in the camera coordinate system:
[0101] Wherein, the camera center C and the direction vector d are calculated as follows:
[0102] As a scale factor, Indicates the possible three-dimensional spatial location of the defect.
[0103] Step 5.4: Cross-solving of 3D ray and point cloud models. The above ray equations are cross-solved with the dense 3D point cloud model constructed in step S2 to determine the precise 3D location of the defect target. The specific implementation method is as follows: First, define the formula for calculating the distance between the 3D ray and the point cloud model:
[0104] in, Let be the coordinates of any point on the ray. Let P be the coordinates of a point in the point cloud model, and let P be the set of three-dimensional point clouds. It is a Euclidean distance.
[0105] The scale factor is solved using the least squares optimization method. Determine the point cloud location in three-dimensional space that is closest to the ray:
[0106] in, The point in the point cloud model that is closest to the ray. To optimize the scale factor after the solution.
[0107] Therefore, the final position coordinates of the defect target in three-dimensional space are determined by calculation. :
[0108] Step 5.5: 3D coordinate scale normalization and coordinate transformation, applying the calculated 3D spatial coordinates of the defect. Scale normalization and coordinate system transformation are performed to ensure that the accuracy of the position coordinates matches the actual scene size:
[0109] in, The original three-dimensional coordinates, Here are the normalized true spatial coordinates, and s is the scale transformation factor. The translation calibration vector has parameter values determined by a known three-dimensional reference target or measurement scale.
[0110] Step 5.6: Output of 3D localization results for the defect target, including: The final output includes the location coordinates of the defect target in three-dimensional space and related information, including the defect type and the spatial coordinates of the defect center. Information such as defect detection confidence level is represented as follows:
[0111] Step 6: Generate a structured inspection report.
[0112] For example, a visual annotation method is used in a 3D point cloud model to intuitively display the location of different defect types and automatically generate a structured inspection report containing defect number, type, 3D coordinates and GPS information.
[0113] In summary, this embodiment provides a complete intelligent detection method from image acquisition, image fusion, defect recognition to three-dimensional spatial positioning, which effectively improves the automation level, detection accuracy and efficiency of building facade defect detection, and has significant practical value and broad application prospects.
[0114] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.
[0115] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0116] Please see Figure 3 , Figure 3 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 301 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 302 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 302 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 302 and is called and executed by the processor 301 using the methods described in the embodiments of this application. Input / output interface 303 is used to implement information input and output; The communication interface 304 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 305 transmits information between various components of the device (e.g., processor 301, memory 302, input / output interface 303, and communication interface 304); The processor 301, memory 302, input / output interface 303, and communication interface 304 are connected to each other within the device via bus 305.
[0117] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.
[0118] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0119] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0120] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0121] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented in the embodiments of this program product are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments. The executable computer program code or "code" used to perform the various embodiments can be written in high-level programming languages such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.
[0122] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0123] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0124] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0125] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0126] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0127] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0128] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0129] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0130] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0131] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0132] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. A method for intelligent detection of defects in building facades, characterized in that, The method includes the following steps: A drone equipped with a visible light camera and an infrared thermal imager was used to simultaneously collect RGB and infrared images of the building's exterior facade. Using the RGB image, a three-dimensional point cloud model of the building facade is constructed based on the structure-by-motion algorithm and oblique photogrammetry. A two-stage adversarial network based on modality transformation is used to perform registration preprocessing on the RGB image and the infrared image to obtain spatially aligned multimodal image pairs; The registered RGB image and the infrared image are input into an image fusion network based on the Transformer architecture to generate a fused image that combines visible light features and thermal features; The fused image is input into a pre-trained target detection model to identify defects in the building facade and obtain their two-dimensional location information; The two-dimensional location information is mapped onto the three-dimensional point cloud model, and the coordinates of the defect in three-dimensional space are determined by back projection and spatial cross calculation. Based on the three-dimensional coordinates and type information of the defect, a visual inspection report is generated.
2. The method according to claim 1, characterized in that, The process of constructing a 3D point cloud model of the building facade using the RGB image, based on a structure-on-motion algorithm and oblique photogrammetry, includes: Feature points are extracted and matched from RGB images acquired from multiple perspectives, and the camera pose is recovered based on the feature points to generate a sparse 3D point cloud. Based on the camera pose, a dense 3D point cloud is generated using a multi-view stereo matching algorithm. The dense 3D point cloud is filtered, registered, and scaled to obtain a high-precision 3D point cloud model.
3. The method according to claim 2, characterized in that, The step of recovering the camera pose based on the feature points and generating a sparse 3D point cloud includes: The sparse 3D point cloud is generated by calculating the essential matrix to recover the relative rotation matrix R and translation vector t between cameras, and by performing triangulation on the matched feature points to recover their 3D coordinates.
4. The method according to claim 2, characterized in that, The generation of dense 3D point clouds through a multi-view stereo matching algorithm includes: For each pixel in the reference image, matching points are searched along the epipolar line in the auxiliary image and the disparity is calculated. The depth value of the pixel is estimated by inverse triangulation based on the camera parameters, and then its three-dimensional coordinates are calculated. The dense three-dimensional point cloud is generated by fusing the multi-view depth maps.
5. The method according to claim 1, characterized in that, The two-stage adversarial network based on mode transition includes: In the first stage, a generative adversarial network is used to learn the mapping relationship from RGB images to infrared images and generate an approximate infrared image. In the second stage, the generated approximate infrared image and the real infrared image are finely spatially registered using a spatial transformation network. The spatial transformation network includes a local network for predicting transformation parameters, a grid generator for generating sampling grids, and a sampler for pixel resampling.
6. The method according to claim 1, characterized in that, The image fusion network based on the Transformer architecture performs the following operations: Convolutional feature extraction was performed on the registered RGB and infrared images respectively; The feature map is divided into image patches and then linearly embedded. Feature encoding is performed using a Transformer module that includes a multi-head self-attention mechanism and a moving window mechanism; RGB and infrared features are weighted and fused using a channel attention mechanism; The fused features are upsampled to restore the original image resolution, and the fused image is output.
7. The method according to claim 1, characterized in that, The methods for training the object detection model include: Construct a fused image dataset containing annotations for three types of defects: cracks, hollow areas, and detachments; Data augmentation strategies including random scaling, rotation, flipping, and color perturbation are employed. Model training is performed using a loss function that combines CIoU loss and Focal Loss; Use average precision, precision, and recall to evaluate model performance.
8. The method according to claim 1, characterized in that, The step of mapping the two-dimensional location information onto the three-dimensional point cloud model and determining the coordinates of the defect in three-dimensional space through back projection and spatial cross calculation includes: Extract the center pixel coordinates of the defect bounding box output by the target detection model. ; Based on the corresponding camera's intrinsic parameter matrix K, rotation matrix R, and translation vector t, the pixel coordinates are back-projected into three-dimensional space to form a ray. ,in Centered on the camera is the ray direction vector, and is the scale factor; Calculate the shortest distance between the ray and the three-dimensional point cloud model to determine the estimated location of the defect in three-dimensional space; The estimated location is scaled and transformed to output the final three-dimensional coordinates of the defect center point. .
9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 8.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.