A building facade defect detection method and system based on a dual-mode image

Abstract

The application belongs to the technical field of intelligent detection and computer vision, and specifically discloses a building facade defect detection method and system based on a dual-mode image, which comprises the following steps: based on the visible light image and the infrared image of the building facade, performing alignment processing to obtain the aligned visible light image and the aligned infrared image; inputting the aligned visible light image and the aligned infrared image into a dual-mode image defect detection model to obtain the defect detection result of the building facade output by the dual-mode image defect detection model; wherein the dual-mode image defect detection model is constructed through a cascaded dual-flow backbone network, a cross-modal attention fusion unit, a semantic mask application unit and a detection head. Through the application, the building facade defect can be efficiently and accurately detected.

Description

Technical Field

[0001] This application belongs to the field of interdisciplinary technology of intelligent detection and computer vision, and more specifically, relates to a method and system for detecting defects in building facades based on bimodal images. Background Technology

[0002] Current building quality inspection methods primarily rely on manual tapping and visual inspection, which suffer from low efficiency (inspecting a single building can take several days), high subjectivity (high rate of missed detections), and high risks associated with high-altitude operations, making it difficult to meet the needs of large-scale, high-precision inspections. Some existing technologies use drones equipped with single-modal sensors (visible light or infrared) for inspection, but these have limitations: while visible light images can reveal structural details, they struggle to identify hidden defects such as hollow areas and leaks; infrared images can capture temperature anomalies, but cannot accurately identify morphological defects such as cracks. Therefore, existing technologies suffer from low efficiency in manual inspection, limitations in single-modal data recognition, and low accuracy of single-modal target detection in complex environments. How to efficiently and accurately detect defects on building facades is a pressing technical problem that needs to be solved in this field. Summary of the Invention

[0003] In view of the shortcomings of the existing technology, the purpose of this application is to achieve efficient and accurate detection of defects in building facades.

[0004] To achieve the above objectives, in a first aspect, this application provides a method for detecting defects in building facades based on dual-modal images, comprising:

[0005] Based on the visible light and infrared images of the building facade, alignment processing is performed to obtain aligned visible light and infrared images;

[0006] Input the aligned visible light image and the aligned infrared image into the dual-modal image defect detection model, and obtain the defect detection results of the building facade output by the dual-modal image defect detection model;

[0007] The dual-modal image defect detection model is constructed by a cascaded dual-stream backbone network, a cross-modal attention fusion unit, a semantic mask application unit, and a detection head.

[0008] The dual-stream backbone network consists of a visible light stream backbone network and an infrared stream backbone network. The visible light stream backbone network is used to extract visible light feature maps based on aligned visible light images. The infrared stream backbone network is used to extract infrared feature maps based on aligned infrared images.

[0009] The cross-modal attention fusion unit is used to fuse visible light feature maps and infrared feature maps through a cross-modal attention mechanism to obtain a fused feature map;

[0010] The semantic mask application unit is used to apply the semantic segmentation mask to the fused feature map to obtain the optimized feature map. The semantic segmentation mask is used to identify non-wall areas in the image through the mask.

[0011] The detection head is used to perform defect detection based on the optimized feature map and obtain the defect detection results.

[0012] In one possible implementation, the cross-modal attention fusion unit is constructed through a cascaded feature concatenation layer, a convolutional dimensionality reduction and nonlinear transformation layer, a channel and spatial attention mechanism layer, and a weighted fusion layer;

[0013] The feature stitching layer is used to stitch together the visible light feature map and the infrared feature map to obtain the stitched features;

[0014] Convolutional dimensionality reduction and nonlinear transformation layers are used to perform convolutional dimensionality reduction and nonlinear transformation operations on the concatenated features to obtain the nonlinearly transformed features.

[0015] The channel and spatial attention mechanism layer is used to process the features after nonlinear transformation through channel attention mechanism and spatial attention mechanism to obtain attention weight map;

[0016] The weighted fusion layer is used to perform weighted fusion of visible light feature maps and infrared feature maps based on the attention weight map to obtain the fused feature map.

[0017] In one possible implementation, the semantic mask application unit is specifically used to: add the semantic segmentation mask to the fused feature map through element-wise multiplication to obtain the optimized feature map.

[0018] In one possible implementation, the detection head uses the decoupling head of YOLOv8.

[0019] In one possible implementation, the semantic segmentation mask is obtained through the following steps:

[0020] The aligned visible light image is input into the semantic segmentation model. The semantic segmentation model detects non-wall regions in the image and obtains the semantic segmentation mask output by the semantic segmentation model.

[0021] In one possible implementation, the defect detection results include: defect location, defect category, and defect confidence level;

[0022] The dual-modal image defect detection model is obtained through supervised training based on a comprehensive loss function;

[0023] The comprehensive loss function is constructed based on the localization loss function, the category loss function, and the mask-assisted loss function;

[0024] The localization loss function is used to calculate the localization loss based on the defect location label and the defect location in the defect detection results;

[0025] The category loss function is used to calculate category loss based on the defect category label and the defect category in the defect detection results;

[0026] The mask-assisted loss function is used to calculate the mask-assisted loss based on the semantic mask label value and the defect confidence in the defect detection result. The semantic mask label value is used to indicate whether the corresponding area in the image is a wall area or a non-wall area, and the mask-assisted loss is used to indicate the situation where the dual-modal image defect detection model misjudges a non-wall area.

[0027] In one possible implementation, the mask-assisted loss function is as follows:

[0028] ;

[0029] in, Indicates mask-assisted loss, The weighting coefficients represent the mask-assisted loss. This indicates the total number of defects in the defect detection results. The number indicating the defect. Indicates the first The semantic mask label value corresponding to the defect location. A semantic mask label value of 1 indicates that the corresponding area in the image is a wall area, and a semantic mask label value of 0 indicates that the corresponding area in the image is a non-wall area. Indicates the first Defect confidence level The target label representing the mask loss is fixed at 0. This represents the binary cross-entropy loss function.

[0030] In one possible implementation, visible light and infrared images of the building facade are acquired through the following steps:

[0031] Control the drone to fly near the building's exterior facade;

[0032] The drone is equipped with a visible light camera and an infrared thermal imager to capture images of the building's exterior facade, obtaining visible light and infrared images.

[0033] Secondly, this application provides a building facade defect detection system based on dual-modal images, comprising:

[0034] The image alignment module is used to align visible light and infrared images based on the building facade to obtain aligned visible light and infrared images.

[0035] The defect detection module is used to input aligned visible light images and aligned infrared images into the dual-modal image defect detection model, and obtain the defect detection results of the building facade output by the dual-modal image defect detection model.

[0036] The dual-modal image defect detection model is constructed by a cascaded dual-stream backbone network, a cross-modal attention fusion unit, a semantic mask application unit, and a detection head.

[0037] The dual-stream backbone network consists of a visible light stream backbone network and an infrared stream backbone network. The visible light stream backbone network is used to extract visible light feature maps based on aligned visible light images. The infrared stream backbone network is used to extract infrared feature maps based on aligned infrared images.

[0038] The cross-modal attention fusion unit is used to fuse visible light feature maps and infrared feature maps through a cross-modal attention mechanism to obtain a fused feature map;

[0039] The semantic mask application unit is used to apply the semantic segmentation mask to the fused feature map to obtain the optimized feature map. The semantic segmentation mask is used to identify non-wall areas in the image through the mask.

[0040] The detection head is used to perform defect detection based on the optimized feature map and obtain the defect detection results.

[0041] Thirdly, this application provides an electronic device, including: a memory and one or more processors; the memory is coupled to one or more processors, the memory is used to store computer program code, the computer program code including computer instructions; one or more processors invoke the computer instructions to cause the electronic device to perform the actions described in the first aspect or any possible implementation of the first aspect.

[0042] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art:

[0043] By precisely aligning visible light and infrared images, spatial consistency is ensured. Visible light images reflect details such as surface texture and color, while infrared images reveal temperature anomalies and can expose hidden defects such as hollowness and leakage. The dual-modal images are then input into a dual-stream backbone network, which extracts visible light and infrared features respectively. Visible light features characterize visible defects such as surface cracks and stains, while infrared features characterize hidden defects in areas of temperature anomalies. A cross-modal attention fusion unit dynamically focuses complementary information from the two modalities using an attention mechanism. For example, cracks in visible light are correlated with temperature anomalies in corresponding infrared regions, achieving adaptive feature fusion and generating a fused feature map rich in defect information. Furthermore, semantic segmentation masks are introduced to identify non-wall areas (such as windows and air conditioner units), combining them with the fused feature map. The masks suppress feature interference from non-wall areas, allowing the model to focus on defect features in wall areas. Finally, the detection head performs precise localization and classification based on the optimized feature map. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating the building facade defect detection method based on dual-modal images provided in this application embodiment;

[0045] Figure 2 This is a schematic diagram of the structure of the dual-modal image defect detection model provided in the embodiments of this application;

[0046] Figure 3 This is a schematic diagram of a visible light image with defect detection results provided in an embodiment of this application;

[0047] Figure 4 This is a schematic diagram of an infrared image with defect detection results provided in an embodiment of this application;

[0048] Figure 5 This is a schematic diagram of the cross-modal attention fusion unit provided in an embodiment of this application;

[0049] Figure 6 This is a schematic diagram of the structure of the building facade defect detection system based on dual-modal images provided in the embodiments of this application;

[0050] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0053] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0054] The embodiments of this application are described below with reference to the accompanying drawings.

[0055] Figure 1 This is a flowchart illustrating the building facade defect detection method based on dual-modal images provided in this application embodiment, as shown below. Figure 1 As shown, the method includes the following steps S101 and S102.

[0056] Step S101: Based on the visible light image and infrared image of the building facade, perform alignment processing to obtain the aligned visible light image and the aligned infrared image;

[0057] Step S102: Input the aligned visible light image and the aligned infrared image into the dual-modal image defect detection model, and obtain the defect detection results of the building facade output by the dual-modal image defect detection model.

[0058] Figure 2 This is a schematic diagram of the structure of the dual-modal image defect detection model provided in the embodiments of this application, as shown below. Figure 2 As shown, the dual-modal image defect detection model is constructed by a cascaded dual-stream backbone network, a cross-modal attention fusion unit, a semantic mask application unit, and a detection head.

[0059] like Figure 2 As shown, the dual-stream backbone network includes a visible light stream backbone network and an infrared stream backbone network; the visible light stream backbone network is used to extract visible light feature maps based on the aligned visible light image; the infrared stream backbone network is used to extract infrared feature maps based on the aligned infrared image.

[0060] Optionally, the visible light flow backbone network and the infrared flow backbone network can specifically adopt the CSPDarknet network structure, employing multi-scale fusion. The visible light flow backbone network and the infrared flow backbone network do not share network weights.

[0061] like Figure 2As shown, the cross-modal attention fusion unit is used to fuse visible light feature maps and infrared feature maps through a cross-modal attention mechanism to obtain a fused feature map.

[0062] like Figure 2 As shown, the semantic mask application unit is used to apply the semantic segmentation mask to the fused feature map to obtain the optimized feature map (so that the optimized feature map can identify non-wall areas in the image through the mask, effectively suppressing interference from non-wall areas). The semantic segmentation mask is used to identify non-wall areas (windows, air conditioner outdoor units, etc.) in the image through the mask.

[0063] like Figure 2 As shown, the detection head is used to perform defect detection based on the optimized feature map and obtain the defect detection results.

[0064] It should be noted that the wall area of ​​a building facade refers to the continuous solid wall portion that constitutes the main structure of the building facade and is used for load-bearing and enclosure. The non-wall area of ​​a building facade refers to auxiliary components attached to or interspersed on the wall that do not participate in the main load-bearing and enclosure functions (such as windows, air conditioning outdoor units, etc.).

[0065] Specifically, spatial consistency is ensured by precisely aligning visible light and infrared images. Visible light images reflect details such as surface texture and color, while infrared images reveal temperature anomalies and can expose hidden defects such as hollowness and leakage. The dual-modal images are then input into a dual-stream backbone network, which extracts visible light and infrared features respectively. Visible light features characterize visible defects such as surface cracks and stains, while infrared features characterize hidden defects in areas of temperature anomalies. A cross-modal attention fusion unit dynamically focuses complementary information from the two modal features using an attention mechanism. For example, cracks in visible light are correlated with temperature anomalies in corresponding areas in infrared, achieving adaptive feature fusion and generating a fused feature map containing rich defect information. Furthermore, semantic segmentation masks are introduced to identify non-wall areas (such as windows and air conditioner units), combining them with the fused feature map. The masks suppress feature interference from non-wall areas, allowing the model to focus on defect features in wall areas. Finally, the detection head performs precise localization and classification based on the optimized feature map.

[0066] It should be noted that this application combines a dual-stream backbone network with a cross-modal attention fusion unit. The dual-stream backbone network is used to achieve modality adaptation, and the cross-modal attention fusion unit is used to achieve feature fusion. This not only preserves the uniqueness of the two modalities, but also adaptively fuses complementary information, which is beneficial for the model to adapt to the differences in physical characteristics between the two modalities (visible light depends on light reflection, and infrared depends on thermal radiation).

[0067] Figure 3This is a schematic diagram of a visible light image with defect detection results provided in an embodiment of this application, such as... Figure 3 As shown, defect detection results can be attached to a visible light image to form a visible light image with defect detection results. "Defect" represents a defect.

[0068] Figure 4 This is a schematic diagram of an infrared image with defect detection results provided in an embodiment of this application, such as... Figure 4 As shown, defect detection results can be attached to an infrared image to form an infrared image with defect detection results.

[0069] It is evident that automating the processing of dual-modal images of building facades using a dual-modal image defect detection model can effectively improve the efficiency of building facade defect detection. Furthermore, employing multiple techniques such as dual-modal feature extraction, dual-modal feature fusion, and interference suppression in non-wall areas can effectively enhance the accuracy of building facade defect detection. Therefore, the building facade defect detection method based on dual-modal images provided in this application can achieve efficient and accurate detection of building facade defects (such as hollow areas, cracks, and leaks).

[0070] In one possible implementation, such as Figure 5 As shown, the cross-modal attention fusion unit is constructed through a cascaded feature splicing layer, a convolutional dimensionality reduction and nonlinear transformation layer, a channel and spatial attention mechanism layer, and a weighted fusion layer.

[0071] like Figure 5 As shown, the feature stitching layer is used to stitch together the visible light feature map and the infrared feature map to obtain the stitched features.

[0072] like Figure 5 As shown, the convolutional dimensionality reduction and nonlinear transformation layers are used to perform convolutional dimensionality reduction and nonlinear transformation operations on the concatenated features to obtain the nonlinearly transformed features.

[0073] like Figure 5 As shown, the channel and spatial attention mechanism layer is used to process the nonlinearly transformed features through channel attention mechanism and spatial attention mechanism to obtain attention weight map (to dynamically allocate the weight of dual-modal features, such as strengthening infrared features in the leakage area and strengthening visible light features in the crack area).

[0074] like Figure 5 As shown, the weighted fusion layer is used to perform weighted fusion of the visible light feature map and the infrared feature map based on the attention weight map to obtain the fused feature map.

[0075] Specifically, the feature stitching layer stitches visible light and infrared feature maps along the channel dimension to form a joint feature containing original information from both modalities. Subsequently, the convolutional dimensionality reduction and nonlinear transformation layer reduces the feature dimension through convolution operations and introduces a nonlinear activation function to enhance feature expressiveness while reducing computational redundancy. Next, the channel and spatial attention mechanism layers generate dynamic attention weight maps from the channel dimension (e.g., strengthening the infrared channel sensitive to leakage or the visible light channel sensitive to cracks) and the spatial dimension (e.g., focusing on temperature anomaly areas or surface texture fracture areas), enabling the model to adaptively adjust the weights of the dual-modal features according to the defect type (e.g., highlighting infrared features in leakage areas and strengthening visible light features in crack areas). Finally, the weighted fusion layer performs weighted fusion of the original dual-modal features based on the attention weight map to generate a fused feature map. This achieves complementary enhancement of dual-modal features through dynamic weight allocation, enabling the model to better adapt to various types of defect features and effectively improve defect detection accuracy.

[0076] For example, the channel and spatial attention mechanism layer uses channel attention and spatial attention to generate an attention weight map. Specifically, a channel attention weight map is generated through channel attention, a spatial attention weight map is generated through spatial attention, and then the channel attention weight map and the spatial attention weight map are merged to obtain a comprehensive attention weight map.

[0077] This section provides an illustrative example of the workflow of the weighted fusion layer. The visible light feature map and the infrared feature map maintain the same scale, and the generated attention weight map also maintains the same scale. Under this premise, based on the visible light feature map and the infrared feature map of the same scale, weighted fusion is performed using an attention weight map with the same size as the feature map to obtain the fused feature map.

[0078] In one possible implementation, the semantic mask application unit is specifically used to: add the semantic segmentation mask to the fused feature map through element-wise multiplication to obtain the optimized feature map.

[0079] Specifically, through element-wise multiplication, the generated semantic segmentation mask (where non-wall areas such as windows and air conditioner units have a value of 0, and wall areas have a value of 1) is multiplied pixel-by-pixel with the feature map fused by cross-modal attention. This directly suppresses the feature response of non-wall areas at the feature level (making its value approach 0), while completely preserving the defect features of wall areas. This operation is equivalent to spatial filtering of the fused feature map, effectively eliminating the interference of non-wall structures (such as window reflections and air conditioner unit heat radiation) on defect detection. This allows the subsequent detection head to focus only on the feature information of the wall areas, thereby significantly improving the model's accuracy in locating and classifying real defects and reducing the false detection rate.

[0080] In one possible implementation, the detection head uses the YOLOv8 decoupled head.

[0081] YOLOv8's decoupling head separates the classification task and the regression task (bounding box prediction) by building independent branch networks for classification and bounding box regression respectively.

[0082] In one possible implementation, the semantic segmentation mask is obtained through the following steps:

[0083] The aligned visible light image is input to a semantic segmentation model (e.g., U-Net network). The semantic segmentation model detects non-wall regions in the image and obtains the semantic segmentation mask output by the semantic segmentation model.

[0084] For example, the training and inference process of the semantic segmentation model is as follows: (1) Data annotation: Collect a large number of visible light images containing various wall backgrounds and annotate them. (2) Model selection and training: Select the U-Net semantic segmentation model architecture and train the model with the annotated data until it can accurately distinguish between wall and non-wall regions in the image. (3) Model inference: Input the visible light image (e.g., RGB image) into the trained semantic segmentation model. The semantic segmentation model performs forward propagation (inference) on the image and calculates the probability of each pixel belonging to "wall" to obtain a probability map. By setting a threshold (e.g., 0.5), the probability map is converted into a binary semantic segmentation mask. Among them, the pixel (x,y) with Mask(x,y)=1 (white) is judged as a wall. The pixel (x,y) with Mask(x,y)=0 (black) is judged as a non-wall.

[0085] The purpose of generating this semantic segmentation mask is to act as a "filter" after feature fusion, effectively suppressing the feature responses of non-wall areas through point-by-point multiplication, so that the detection head only focuses on the wall area.

[0086] In one possible implementation, the defect detection results include: defect location, defect category, and defect confidence level;

[0087] The dual-modal image defect detection model is obtained through supervised training based on a comprehensive loss function;

[0088] The comprehensive loss function is constructed based on the localization loss function, the category loss function, and the mask-assisted loss function;

[0089] The localization loss function is used to calculate the localization loss based on the defect location label and the defect location in the defect detection results;

[0090] The category loss function is used to calculate category loss based on the defect category label and the defect category in the defect detection results;

[0091] The mask-assisted loss function is used to calculate the mask-assisted loss based on the semantic mask label value and the defect confidence in the defect detection result. The semantic mask label value is used to indicate whether the corresponding area in the image is a wall area or a non-wall area, and the mask-assisted loss is used to indicate the situation where the dual-modal image defect detection model misjudges a non-wall area.

[0092] For example, the comprehensive loss function = localization loss function + category loss function + mask auxiliary loss function.

[0093] Here is an example of how the localization loss function is implemented. When there is a deviation between the model's predicted defect bounding box coordinates and the actual location marked by the label, the localization loss function will calculate the difference in the intersection-over-union ratio (IoU) or the coordinate regression error between the two to quantify the localization accuracy and guide the model to adjust the bounding box parameters.

[0094] Here is an example of how the category loss function is implemented. If the defect detection result given by the model results in a misclassification of the defect category (for example, misclassifying the "crack" defect as the "stain" category), the category loss function will calculate the difference between the predicted category probability and the true category label based on cross-entropy or focus loss, in order to penalize the classification error and optimize the category discrimination ability.

[0095] Understandably, by introducing semantic mask labels (which explicitly identify wall areas and non-wall areas in an image), the mask-assisted loss function supervises the defect confidence of the model output during training. When the model produces a high-confidence defect prediction in non-wall areas (such as windows, air conditioner outdoor units, etc.), the loss function calculates a penalty term based on the mask label value, directly increasing the model's training loss. This mechanism forces the model to suppress the feature responses of non-wall areas during the learning process, avoiding misjudging texture, edge, or temperature anomalies in non-wall areas as defects, thereby guiding the model to focus its attention on defect features in wall areas.

[0096] Therefore, by co-optimizing the mask-assisted loss function with the localization loss and category loss, while ensuring the accuracy of defect location and category, the model significantly reduces invalid predictions in the training and inference stages by explicitly constraining the prediction behavior of non-wall areas, thereby improving the model's accuracy and robustness in identifying real defects.

[0097] In one possible implementation, the mask-assisted loss function is as follows:

[0098] ;

[0099] in, Indicates mask-assisted loss, The weighting coefficients represent the mask-assisted loss. This indicates the total number of defects in the defect detection results. The number indicating the defect. Indicates the first The semantic mask label value corresponding to the defect location. A semantic mask label value of 1 indicates that the corresponding area in the image is a wall area, and a semantic mask label value of 0 indicates that the corresponding area in the image is a non-wall area. Indicates the first Defect confidence level The target label representing the mask loss is fixed at 0. This represents the binary cross-entropy loss function.

[0100] In one possible implementation, visible light and infrared images of the building facade are acquired through the following steps:

[0101] Control the drone to fly near the building's exterior facade;

[0102] The drone is equipped with a visible light camera and an infrared thermal imager to capture images of the building's exterior facade, obtaining visible light and infrared images.

[0103] The following comprehensive example illustrates the building facade defect detection method based on bimodal images provided in this application.

[0104] In this comprehensive example, the building facade defect detection method based on dual-modal images provided in this application includes: (1) accurate acquisition of multimodal data; (2) collaborative preprocessing of dual-modal data; (3) multimodal fusion; and (4) target detection and output.

[0105] (1) Accurate acquisition of multimodal data;

[0106] By using a drone equipped with a visible light camera and an infrared thermal imager, a serpentine flight path is planned based on preset parameters such as building size, equipment zoom level, and image overlap rate to achieve collaborative data acquisition both horizontally row-by-row and vertically layered. Visible light images (capturing morphological details such as texture and edges) and infrared temperature images (recording thermal radiation differences; areas with hollowness or leakage will show temperature anomalies, providing a basis for subsequent model identification of defects such as hollowness and leakage) are acquired simultaneously, ensuring that the dual-modal data corresponds to the same detection area in time and space, providing raw data support for feature fusion.

[0107] (2) Collaborative preprocessing of dual-modal data;

[0108] The acquired dual-modal images were registered at the pixel level (by matching infrared and visible light images with SIFT / ORB feature points and affine transformation), and defective areas (hollows, cracks, etc.) and non-wall interference areas (windows, air conditioner outdoor units, etc.) were marked. Data augmentation strategies such as Mosaic stitching, visible light brightness perturbation, and infrared temperature and noise simulation were used to expand sample diversity and improve data quality and model robustness.

[0109] For example, the ORB (Oriented FAST and Rotated BRIEF) algorithm is used to extract key feature points of dual-modal images and complete the matching. Based on the matching results, the affine transformation matrix is ​​calculated to achieve pixel-level spatial alignment between infrared and visible light images.

[0110] Mosaic stitching is a technique that seamlessly integrates multiple images or video clips into a single, coherent image.

[0111] (3) Multimodal fusion;

[0112] A dual-stream backbone network is constructed to extract visible light texture features and infrared temperature features respectively. A cross-modal attention mechanism is introduced into the fusion layer to dynamically allocate the weights of the dual-modal features (e.g., infrared features are enhanced in the leakage area, and visible light features are enhanced in the crack area). Semantic masks are embedded to suppress interference from non-wall areas, and the loss function is optimized to reduce invalid predictions in non-wall areas.

[0113] Specifically, in terms of model structure, optimizations are made in modal adaptation, feature fusion, and suppression of interference in non-wall areas.

[0114] A dual-stream backbone network (without weights shared between the two backbone networks) is constructed to extract visible light texture features and infrared temperature features respectively, thereby achieving modal adaptation.

[0115] Cross-modal attention fusion mechanism: Based on infrared and visible light feature maps of the same scale, an attention weight map with the same size as the feature map is generated through a cross-modal attention fusion unit, which adaptively allocates the contribution of the two modal features (e.g., strengthening visible light features in crack regions and strengthening infrared features in hollow regions).

[0116] Semantic masking application: The generated semantic mask is combined with the feature map fused with cross-modal attention through element-wise multiplication (point-by-point multiplication). This improves defect detection accuracy by precisely suppressing interference from non-wall regions.

[0117] In addition, optimization is performed on the loss function during model training.

[0118] The loss function is optimized to distinguish between defective areas (hollows, cracks, etc.) and non-wall interference areas (windows, outdoor units, etc.). The loss function is a combination of "localization loss + category loss + mask-assisted loss." Localization loss ensures the accuracy of locating targets like hollows and penetrations, category loss ensures classification accuracy, and mask-assisted loss provides additional penalties to enable the model to learn to exclude interference from non-wall areas, further reducing the false detection rate. "Localization loss + category loss" penalizes incorrect detections in wall areas, while mask-assisted loss penalizes invalid predictions in non-wall areas.

[0119] Mask-assisted loss formula:

[0120] ;

[0121] in, Masking auxiliary loss value, The weight coefficients for mask-assisted loss are denoted by S, which is the total number of defects predicted by the model. That is, the number of defects output by the model when performing defect detection on the optimized feature map. For a defect, the model will provide the corresponding defect location (usually marked by a bounding box), defect category, and confidence level. The semantic mask label value corresponding to the k-th defect (the semantic mask label value is extracted from the corresponding region of the semantic segmentation mask label based on the defect location), for non-wall areas ( =0), (1- ) = 1 activation mask loss; for the wall area ( =1), (1- If ) = 0, the mask loss term is directly 0, and there will be no penalty for "defect prediction" in the wall area. It represents the confidence level (a probability value between 0 and 1) corresponding to the k-th defect, in the mask-assisted loss. Its function is to find non-wall areas A misjudgment of approximately 1 is penalized. The target label for the mask loss is fixed at 0, and BCE(•) is the binary cross-entropy loss function. When the model misclassifies non-wall regions... Approximately equal to 1, and The largest difference occurs when BCE outputs a significantly high value, imposing a strong penalty on the model and supervising the model to reduce the size of non-wall areas during subsequent training. The prediction rate is maintained until it approaches zero (i.e., the defect is correctly determined), effectively suppressing invalid predictions in non-wall areas.

[0122] (4) Target detection and output;

[0123] The fused multi-scale feature maps are input into the YOLOv8 detection head to complete target localization and classification. The detection head predicts bounding box coordinates (x, y, w, h), defect category, and defect confidence score through convolutional layers; it uses non-maximum suppression (NMS) to filter redundant bounding boxes and outputs the final detection results (including defect location (i.e., the bounding box corresponding to the defect), defect category, and defect confidence score). Based on the detection results, a defect detection report can be automatically generated using a large language model (referred to as the large model).

[0124] In the above comprehensive example, the building facade defect detection method based on dual-modal images provided in this application has the following beneficial effects.

[0125] Efficiency Improvement: Automated data collection by drones replaces manual high-altitude operations and visual observation, significantly shortening the inspection cycle for a single building; the automatic report generation function driven by large models replaces the manual compilation process, achieving efficient connection of the entire process from data collection to report output.

[0126] Accurate identification: By fusing visible light and infrared multimodal data, it overcomes the limitations of single-modal identification in hidden defects (hollows, leaks) or morphological defects (cracks); combined with cross-modal attention mechanism and semantic mask optimization, it reduces interference from non-wall areas and improves the accuracy and reliability of defect identification.

[0127] Enhanced scene adaptability: It breaks through the spatial limitations of manual inspection and achieves full coverage inspection of high-altitude or hidden areas such as building facades and roofs; the dual-modal data complementarity reduces the impact of complex environments such as uneven lighting and occlusion on the inspection results, making it suitable for various old building scenarios.

[0128] Cost optimization: Reduce reliance on manual high-altitude operations to lower labor costs and safety risks; lower the implementation threshold for large-scale testing through technical standardization and automated processes to promote the large-scale application of testing services.

[0129] The following describes the building facade defect detection system based on bimodal images provided in this application. The building facade defect detection system based on bimodal images described below can be referred to in correspondence with the building facade defect detection method based on bimodal images described above.

[0130] Figure 6 This is a schematic diagram of the building facade defect detection system based on dual-modal images provided in an embodiment of this application, as shown below. Figure 6 As shown, the system includes: an image alignment module 10 and a defect detection module 20. Wherein:

[0131] The image alignment module 10 is used to perform alignment processing on the visible light image and infrared image based on the building facade, and to obtain the aligned visible light image and the aligned infrared image.

[0132] The defect detection module 20 is used to input aligned visible light images and aligned infrared images into the dual-modal image defect detection model, and obtain the defect detection results of the building facade output by the dual-modal image defect detection model.

[0133] The dual-modal image defect detection model is constructed by a cascaded dual-stream backbone network, a cross-modal attention fusion unit, a semantic mask application unit, and a detection head.

[0134] The dual-stream backbone network consists of a visible light stream backbone network and an infrared stream backbone network. The visible light stream backbone network is used to extract visible light feature maps based on aligned visible light images. The infrared stream backbone network is used to extract infrared feature maps based on aligned infrared images.

[0135] The cross-modal attention fusion unit is used to fuse visible light feature maps and infrared feature maps through a cross-modal attention mechanism to obtain a fused feature map;

[0136] The semantic mask application unit is used to apply the semantic segmentation mask to the fused feature map to obtain the optimized feature map. The semantic segmentation mask is used to identify non-wall areas in the image through the mask.

[0137] The detection head is used to perform defect detection based on the optimized feature map and obtain the defect detection results.

[0138] It is understood that the detailed functional implementation of each of the above units / modules can be found in the description in the aforementioned method embodiments, and will not be repeated here.

[0139] It should be understood that the above system is used to execute the methods in the above embodiments. The corresponding program modules in the system are similar in implementation principle and technical effect to those described in the above methods. The working process of the system can be referred to the corresponding process in the above methods, and will not be repeated here.

[0140] Based on the methods in the above embodiments, this application provides an electronic device. Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 7As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the methods in the above embodiments.

[0141] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0142] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0143] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0144] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0145] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0146] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0147] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0148] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for building facade defect detection based on bimodal images, characterized in that, The method comprises the following steps: Based on the visible light image and the infrared image of the building facade, alignment processing is performed to obtain an aligned visible light image and an aligned infrared image; Input the aligned visible light image and the aligned infrared image into a dual-modal image defect detection model to obtain a defect detection result of the building facade output by the dual-modal image defect detection model; The dual-modal image defect detection model is constructed by cascading a dual-stream backbone network, a cross-modal attention fusion unit, a semantic mask application unit, and a detection head; The dual-stream backbone network includes a visible light stream backbone network and an infrared stream backbone network; the visible light stream backbone network is used to extract a visible light feature map based on the aligned visible light image; the infrared stream backbone network is used to extract an infrared feature map based on the aligned infrared image; The cross-modal attention fusion unit is used to fuse the visible light feature map and the infrared feature map through a cross-modal attention mechanism to obtain a fused feature map; The semantic mask application unit is used to apply a semantic segmentation mask to the fused feature map to obtain an optimized feature map, and the semantic segmentation mask is used to identify non-wall regions in the image through the mask; The detection head is used to perform defect detection based on the optimized feature map to obtain a defect detection result; The cross-modal attention fusion unit is constructed by cascading a feature concatenation layer, a convolution dimension reduction and nonlinear transformation layer, a channel and spatial attention mechanism layer, and a weighted fusion layer; The feature concatenation layer is used to concatenate the visible light feature map and the infrared feature map to obtain a concatenated feature; The convolution dimension reduction and nonlinear transformation layer is used to perform convolution dimension reduction and nonlinear transformation operations on the concatenated feature to obtain a nonlinear transformed feature; The channel and spatial attention mechanism layer is used to process the nonlinear transformed feature through channel attention mechanism and spatial attention mechanism to obtain an attention weight map; The weighted fusion layer is used to perform weighted fusion on the visible light feature map and the infrared feature map based on the attention weight map to obtain a fused feature map.

2. The method for building facade defect detection based on dual-mode images according to claim 1, characterized in that, The semantic mask application unit is specifically used to add the semantic segmentation mask to the fused feature map through element-level multiplication operation to obtain an optimized feature map. 3.The building facade defect detection method based on dual-mode images according to claim 1, characterized in that, The detection head adopts the decoupled head of YOLOv8. 4.The building facade defect detection method based on dual-mode images of claim 1, wherein, The semantic segmentation mask is obtained through the following steps: Input the aligned visible light image into a semantic segmentation model to detect non-wall regions in the image through the semantic segmentation model to obtain a semantic segmentation mask output by the semantic segmentation model. 5.The building facade defect detection method based on dual-mode images of claim 1, wherein, The defect detection result includes defect location, defect category, and defect confidence; The dual-modal image defect detection model is obtained through supervised training based on a comprehensive loss function; The comprehensive loss function is constructed based on a positioning loss function, a category loss function, and a mask auxiliary loss function; The positioning loss function is used to calculate a positioning loss based on a defect location label and a defect location in the defect detection result; The category loss function is used to calculate a category loss based on a defect category label and a defect category in the defect detection result; The mask auxiliary loss function is used to calculate a mask auxiliary loss based on a semantic mask label value and a defect confidence in a defect detection result, the semantic mask label value is used to indicate whether a corresponding region in an image is a wall region or a non-wall region, and the mask auxiliary loss is used to indicate a misjudgment of the bimodal image defect detection model for a non-wall region. 6.The building facade defect detection method based on dual-mode images according to claim 5, characterized in that, The mask auxiliary loss function is as follows: ； wherein, denotes a mask-assisted loss, denotes a weight coefficient of the mask-assisted loss, denotes a total number of defects in the defect detection result, denotes a number of the defect, denotes a first a semantic mask label value corresponding to a defect position of the defect, the semantic mask label value taking a value of 1 indicates that a corresponding region in the image is a wall region, and the semantic mask label value taking a value of 0 indicates that the corresponding region in the image is a non-wall region, denotes a second a defect confidence of the defect, denotes a target label of the mask loss, and is fixed as 0, denotes a binary cross-entropy loss function. 7.The building facade defect detection method based on dual-mode images of claim 1, wherein, The visible light image and the infrared image of the building facade are obtained through the following steps: Controlling the unmanned aerial vehicle to fly to the vicinity of the building facade; Through the visible light camera and the infrared thermal imager carried by the unmanned aerial vehicle, the image of the building facade is collected to obtain the visible light image and the infrared image.

8. A system for building facade defect detection based on bimodal images, characterized in that, Comprise: An image alignment module is configured to perform alignment processing based on the visible light image and the infrared image of the building facade to obtain an aligned visible light image and an aligned infrared image; A defect detection module is configured to input the aligned visible light image and the aligned infrared image into a bimodal image defect detection model to obtain a defect detection result of the building facade output by the bimodal image defect detection model; The bimodal image defect detection model is constructed by cascading a double-flow backbone network, a cross-modal attention fusion unit, a semantic mask application unit, and a detection head; The double-flow backbone network comprises a visible light flow backbone network and an infrared flow backbone network; the visible light flow backbone network is configured to extract a visible light feature map based on the aligned visible light image; and the infrared flow backbone network is configured to extract an infrared feature map based on the aligned infrared image; The cross-modal attention fusion unit is configured to fuse the visible light feature map and the infrared feature map through a cross-modal attention mechanism to obtain a fused feature map; The semantic mask application unit is configured to apply a semantic segmentation mask to the fused feature map to obtain an optimized feature map, and the semantic segmentation mask is configured to identify a non-wall region in the image through a mask; The detection head is configured to perform defect detection based on the optimized feature map to obtain a defect detection result; The cross-modal attention fusion unit is constructed by cascading a feature concatenation layer, a convolution dimension reduction and nonlinear transformation layer, a channel and spatial attention mechanism layer, and a weighted fusion layer; The feature concatenation layer is configured to concatenate the visible light feature map and the infrared feature map to obtain a concatenated feature; The convolution dimension reduction and nonlinear transformation layer is configured to perform convolution dimension reduction and nonlinear transformation operations on the concatenated feature to obtain a nonlinear transformed feature; The channel and spatial attention mechanism layer is configured to process the nonlinear transformed feature through a channel attention mechanism and a spatial attention mechanism to obtain an attention weight map; The weighted fusion layer is configured to perform weighted fusion on the visible light feature map and the infrared feature map based on the attention weight map to obtain a fused feature map.

9. An electronic device, comprising: Comprise: A memory and one or more processors; The memory is coupled to the one or more processors, and the memory is configured to store computer program code, the computer program code comprising computer instructions; The one or more processors invoke the computer instructions to cause the electronic device to perform the method of any one of claims 1-7.

Images