Visible light and infrared image fused building outer wall defect detection method

Abstract

The invention discloses a visible light and infrared image fused building outer wall surface defect detection method. The method comprises the following steps: acquiring a visible light picture and an infrared picture of an outer wall surface; matching the visible light picture and the infrared picture; calibrating an interferent in the visible light picture and calibrating an invisible interferent in the visible light picture in the infrared picture; carrying out secondary calibration on the calibrated interferent in the visible light picture in the infrared picture; performing interference removal processing on the infrared picture; calibrating diseases in the infrared picture and the visible light picture; respectively inputting the processed infrared picture, the processed visible light picture and the disease label data corresponding to the infrared picture and the visible light picture into a model for training and identification; and carrying out multi-modal data fusion on the identification result to obtain a final disease identification result. According to the method, interference is removed through combination of the infrared image and the visible light image, mask adding processing is carried out, then the image is substituted into the model for prediction, and the prediction accuracy is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of building exterior wall detection, and in particular relates to a building exterior wall defect detection method that fuses visible light and infrared images. Background Art

[0002] With the acceleration of urbanization, the stock of existing buildings continues to grow, and building facade defects (such as cracking, hollowing, and falling) are becoming more frequent, seriously threatening people's lives and property. Regular inspection and maintenance of buildings is currently a common method to reduce the risk of building facade falling. However, traditional inspection methods mainly rely on manual climbing visual inspection or simple instrument assistance, which has problems such as low efficiency, high risk, and insufficient quantification. Especially in high-rise buildings and large-scale urban renewal projects, it is difficult to meet the needs of fast, accurate, and safe inspection. At present, in order to improve the detection efficiency of building facade defects, infrared images or visible light images are often used to inspect building facades. Infrared images are generated by detecting infrared rays released by objects. They can provide better visual effects without being affected by weather and lighting. For example, they can still highlight the target well in scenes such as haze, at night, and with obstructions. However, they are insufficient in terms of resolution and texture details. Visible light images are formed by capturing visible light reflected or emitted by objects, but they are easily affected by weather or lighting factors and lose target information. Summary of the Invention

[0003] The purpose of the present invention is to overcome the defects of the existing technology and provide a building exterior wall defect detection method that fuses visible light and infrared images. The method uses an unmanned aerial vehicle equipped with an infrared thermal imager and a visible light imager to collect data on existing building facade defects and artificially constructed building facade defects. The method combines the respective characteristics and advantages of infrared images and visible light images, and presents the information contained in the two modal source images in the same image, thereby obtaining a fused image that has both important thermal target information and rich texture details. By removing interference from the infrared image and the visible light image, and adding a mask process, the method is finally introduced into the model for prediction, which can improve the accuracy of the prediction and achieve rapid positioning of potential risk areas of high-rise building facade facing bricks.

[0004] The object of the present invention is achieved through the following technical solutions: A method for detecting building exterior wall defects by fusing visible light and infrared images comprises the following steps: Step S1: Obtain a visible light image and an infrared image at the same position on the building exterior wall; Step S2: matching feature points of the visible light image and the infrared image; Step S3: calibrating the interference objects in the visible light image and calibrating the invisible interference objects in the visible light image in the infrared image; Step S4: performing secondary positioning calibration on the interference objects calibrated in the visible light image in the infrared image; Step S5: performing interference removal processing on the infrared image; Step S6: Mark the disease in the infrared image and the visible light image respectively; Step S7: input the calibrated infrared image and visible light image and their corresponding disease label data into the deep learning model for training and recognition; Step S8: performing multimodal data fusion on the recognition results of visible light and infrared images to obtain the final disease recognition result; Through this embodiment, the respective characteristics and advantages of infrared images and visible light images are combined, and the information contained in the two modal source images is presented in the same image, thereby obtaining a fused image that has both important thermal target information and rich texture details, which can more accurately eliminate interference factors.

[0005] In one embodiment, in step S1 , visible light and infrared lenses of a drone are used to capture visible light and infrared images of the building's exterior wall, respectively.

[0006] In one embodiment, in step S2, coordinate alignment of the visible light image and the infrared image is established by cross-modal feature point matching to match the visible light image and the infrared image taken at the same location; Through this embodiment, two images at the same position are quickly matched together through cross-modal feature point matching.

[0007] In one embodiment, step S5 further includes: Step S501: Using a deep learning network to remove interference objects marked in steps S3 and S4 from the infrared image; Step S502: Filter the infrared image after step S501 using a morphological filtering algorithm.

[0008] In one embodiment, in step S502, the method further includes: After completing step S501, the infrared image is converted into the LAB color space to enhance the color contrast. Contrast-limited adaptive histogram equalization is applied for contrast enhancement. The enhanced image is then converted into a grayscale image, smoothed by Gaussian filtering, and divided into foreground and background using adaptive threshold segmentation. The morphological filtering algorithm is then used to further remove noise points.

[0009] In one embodiment, in step S5, removing the interferents includes: Draw a mask at the interference area and cover it on the image for filtering; Through this implementation, the interference is removed by masking, which makes it easier for the subsequent model to reduce misjudgment and further improve the accuracy of prediction; In one embodiment, in step S501 , interference objects marked on the infrared image are removed using a U-net network model.

[0010] In one embodiment, step S6 includes: For infrared images and visible light images, use labeling software to classify and label them according to the characteristics and defects of the disease, and save the corresponding label files.

[0011] In one embodiment, step S8 includes: The defect recognition results based on appearance features in visible light images are fused with the defect recognition results based on thermal features in infrared images to obtain the true situation of the disease.

[0012] The beneficial effects of the present invention are: Drones equipped with infrared thermal imagers and visible light imagers are used to collect images of existing building facade defects and artificially constructed building facade defects. The characteristics and advantages of infrared images and visible light images are combined, and the information contained in the two modal source images is presented in the same image. This obtains a fused image that has both important thermal target information and rich texture details. It can more accurately eliminate interference factors and apply masking to the interference areas to facilitate subsequent models to reduce misjudgment, thereby improving the accuracy of predictions and achieving rapid positioning of potential risk areas of high-rise building facade facing tiles. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The present invention will be described in more detail below based on embodiments and with reference to the accompanying drawings, wherein: Figure 1 Shows a schematic diagram of the detection process of the present invention; Figure 2 A schematic diagram showing the original image of the present invention being labeled; Figure 3 A schematic diagram showing the final disease identification result of the present invention is shown; In the drawings, like reference numerals are used for like parts, but the drawings are not necessarily true to scale. DETAILED DESCRIPTION

[0014] The present invention will be further described below with reference to the accompanying drawings.

[0015] The present invention provides a method for detecting building exterior wall defects by fusing visible light and infrared images. Figure 1 As shown, the following steps are included: Step S1: Obtain a visible light image and an infrared image at the same position on the building exterior wall; Use a drone equipped with a sensor to fly to the walls of a residential area that meets the requirements to shoot. The drone shooting is divided into two parts. One part uses the visible light high-definition lens of the gimbal to shoot, and the other part uses the gimbal's infrared lens to shoot. When shooting, ensure that the shooting distance of the drone from different positions to the wall is the same, and shoot from the top floor to the bottom floor at the same interval. Step S2: matching feature points of the visible light image and the infrared image; Step S3: calibrating the interference objects in the visible light image and calibrating the invisible interference objects in the visible light image in the infrared image; Step S4: performing secondary positioning calibration on the interference objects calibrated in the visible light image in the infrared image; Step S5: performing interference removal processing on the infrared image; Step S501: Using a deep learning network to remove interference objects marked in steps S3 and S4 from the infrared image; Step S502: The infrared image after step S501 is filtered using a morphological filtering algorithm. For example, if the channel steel and the filled slot are not filtered cleanly, the infrared image is converted to the LAB color space to enhance the color contrast. Contrast-limited adaptive histogram equalization is applied for contrast enhancement. The enhanced image is then converted to a grayscale image, smoothed using a Gaussian filter, and the image is divided into foreground and background using adaptive threshold segmentation. The morphological filtering algorithm is then used to further remove noise points. Contours are searched on the pre-processed image, and each contour is screened based on four aspects: shape, size, aspect ratio, and area. In step S501 and step S502, a mask is drawn at the interference location and covered on the image for filtering; Step S6: Mark the defects in the infrared image and the visible light image respectively, that is, use labeling software to classify and label the defects in the infrared image and the visible light image according to their characteristic defects, and save the corresponding label files; Step S7: input the calibrated infrared image and visible light image and their corresponding disease label data into the deep learning model for training and recognition; Step S8: performing multimodal data fusion on the recognition results of visible light and infrared images to obtain the final disease recognition result; It should be noted that this embodiment provides a method for detecting defects on building exterior walls by fusing visible light and infrared images, and also includes training of a defect detection model. That is, the defect detection model is first trained. After its accuracy is higher than a preset value, the visible light image and the infrared image of the building facade can be input into the defect detection model to automatically complete the defect detection of the building facade. By combining the respective characteristics and advantages of infrared images and visible light images, the information contained in the two modal source images is presented in the same image, thereby obtaining a fused image with both important thermal target information and rich texture details. Interference factors can be eliminated more accurately, and masking can be applied to the interference areas to facilitate the subsequent model to reduce misjudgment, thereby improving the accuracy of the prediction and realizing the rapid positioning of potential risk areas of facing bricks on the facades of high-rise buildings. Furthermore, if Figure 2 As shown, (a), (b), (c) and (d) show four labeled result images on the original pictures (the known defects are artificially marked on the original pictures as a verification set), and Figure 3 The middle one is the detection result diagram of the corresponding original picture after the deep learning model and multimodal data fusion. It can be seen that it is different from the real label. Figure 2 By comparison, it can be clearly seen that most of its defects are detected accurately, proving that the present invention can effectively and accurately identify defects on building facades; Specifically, in step S2, representative and stable feature points can be extracted from the visible light image and the infrared image. Even under complex lighting conditions, different shooting angles, and a certain degree of noise interference, the two different types of images at the same location can still be accurately matched, such as the position, size, and interference of the images. In one embodiment, in step S3, interference objects are calibrated in the visible light image, such as obvious interference objects such as windows, sky, holes and wall cracks; invisible interference objects in the visible light image are calibrated in the infrared image, such as channel steel and filling walls, which are not obvious in the visible light image. In step S4, a matching correspondence is performed based on the labels calibrated in step S3, and the information of the obvious interference objects calibrated in the visible light image is mapped to the infrared image. The interference objects at the corresponding positions in the infrared image are accurately positioned and calibrated for a second time, so as to realize the redetermination and calibration of the calibrated interference objects in the visible light image in the infrared image, that is, firstly, the interference objects at the corresponding positions in the infrared image are accurately positioned and calibrated for a second time, that is, the windows, sky, holes and wall cracks calibrated in the visible light image are re-confirmed and calibrated in the infrared image, and then the channel steel and filling walls that have been calibrated in the infrared image can be referenced to confirm that the calibration is correct, and finally a correctly calibrated infrared image and label are obtained; In one embodiment, in step S5, the infrared image is subjected to interference removal processing by a deep learning network, such as a convolutional neural network (CNN), a U-net network, a Mask R-CNN, etc. In this embodiment, the U-net network is used for interference removal processing. The symmetrical encoder-decoder structure of the U-net network can effectively capture the contextual information and detailed features of the image, and is suitable for removing complex interference objects in infrared images. At the same time, by learning a large amount of labeled data, the U-net can accurately identify and segment the interference area, thereby achieving efficient interference removal of infrared images. The U-net network used uses 4 upsampling modules and 1 additional upsampling module for further upsampling and feature fusion. The backbone network uses ResNet50, which is mainly responsible for extracting hierarchical features. During the model training process, mixed precision training technology is used to accelerate model convergence. The pixel classification accuracy and regional consistency are simultaneously improved through the joint optimization function of cross entropy loss and Dice loss. A dynamic learning rate adjustment mechanism is also introduced to automatically reduce the learning rate based on the segmentation accuracy of the validation set, and retain the optimal weight in combination with a periodic model preservation strategy, and finally output a deep learning model with pixel-level semantic segmentation capabilities; Furthermore, after removing the calibrated interference objects, the inventors discovered that interference objects still existed in the infrared image that had not been completely removed, and that the remaining interference objects were highly regular. Therefore, step S502 was performed, where morphological filtering was used for fine filtering. This effectively removed interference objects with regular shapes and further optimized the quality of the infrared image. Specifically, targeted morphological filtering was used to gradually determine whether relatively regular and small-area interference objects were interference objects based on four aspects: shape, size, aspect ratio, and area. After confirmation, a mask was added for filtering. It should be noted that in step S5, whether it is the removal of the calibration interference or the removal of the remaining interference, the filtering process is performed by drawing a mask covering the image, which is convenient for the subsequent defect detection model to reduce misjudgment and thus improve the accuracy of the prediction. Figure 2 and Figure 3 In the figure, the green part is the effect of the mask covering the image and does not participate in the identification of defects.

[0016] In one embodiment, in step S7, in response to the dual requirements of real-time and accuracy for drone data collection, the YOLOv11 model is preferentially used for prediction and recognition. Through an improved backbone network and feature fusion mechanism, it can accurately locate defective areas such as cracks and hollows on the facade while maintaining a high inference speed. The YOLOv11 model is used to identify defects in infrared images and visible light images, and the two identification results are deeply fused at the data level to accurately obtain the final true situation of the defects. This method is highly scalable. Researchers can independently select adaptation models from different deep learning architectures such as ResNet, LSTM, and GAT according to specific task requirements, and supports multi-model integration strategies. It should be noted that in step S8, the defects identified jointly in the visible light image and the infrared image as well as the defects identified separately are fused to obtain the final defect identification result, such as Figure 3 As shown, Figure 2 Compared with the validation set, it can be clearly seen that most of its defect detections are accurate, proving that the deep learning model at this time can already perform efficient and accurate defect recognition on visible light images and infrared images of building facades taken at the same location.

[0017] In the description of the present invention, it should be understood that the terms "upper", "lower", "bottom", "top", "front", "back", "inside", "outside", "left", "right", etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention.

[0018] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the invention. It should be understood that many modifications may be made to the illustrative embodiments, and that other arrangements may be devised, without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in ways other than those described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be employed in conjunction with other described embodiments.

Claims

1. A method for detecting building exterior wall defects by fusing visible light and infrared images, characterized in that: The steps include: Step S1: Obtain a visible light image and an infrared image at the same position on the building exterior wall; Step S2: matching feature points of the visible light image and the infrared image; Step S3: calibrating the interference objects in the visible light image and calibrating the invisible interference objects in the visible light image in the infrared image; Step S4: performing secondary positioning calibration on the interference objects calibrated in the visible light image in the infrared image; Step S5: performing interference removal processing on the infrared image; Step S6: Mark the disease in the infrared image and the visible light image respectively; Step S7: input the calibrated infrared image and visible light image and their corresponding disease label data into the deep learning model for training and recognition; Step S8: Perform multimodal data fusion on the recognition results of visible light and infrared images to obtain the final disease recognition result.

2. The method for detecting building exterior wall defects by fusing visible light and infrared images according to claim 1, characterized in that: In step S1, a visible light lens and an infrared lens of a drone are used to respectively capture a visible light image and an infrared image of the building exterior wall.

3. The method for detecting building exterior wall defects by fusing visible light and infrared images according to claim 1, characterized in that: In step S2 , coordinate alignment of the visible light image and the infrared image is established through cross-modal feature point matching to match the visible light image and the infrared image taken at the same location.

4. The method for detecting building exterior wall defects by fusing visible light and infrared images according to claim 1, characterized in that: Step S5 further includes: Step S501: Using a deep learning network to remove interference objects marked in steps S3 and S4 from the infrared image; Step S502: Filter the infrared image after step S501 using a morphological filtering algorithm.

5. The method for detecting building exterior wall defects by fusing visible light and infrared images according to claim 1, characterized in that: In step S502, the method further includes: After completing step S501, the infrared image is converted into the LAB color space to enhance the color contrast. Contrast-limited adaptive histogram equalization is applied for contrast enhancement. The enhanced image is then converted into a grayscale image, smoothed by Gaussian filtering, and divided into foreground and background using adaptive threshold segmentation. The morphological filtering algorithm is then used to further remove noise points.

6. The method for detecting building exterior wall defects by fusing visible light and infrared images according to claim 4, characterized in that: In step S5, removing the interferents includes: Draw a mask at the interference area and cover it on the image for filtering.

7. The method for detecting building exterior wall defects by fusing visible light and infrared images according to claim 4, characterized in that: In step S501, the U-net network model is used to remove the interference objects marked on the infrared image.

8. The method for detecting building exterior wall defects by fusing visible light and infrared images according to claim 1, characterized in that: In step S6, it includes: For infrared images and visible light images, use labeling software to classify and label them according to the characteristics and defects of the disease, and save the corresponding label files.

9. The method for detecting building exterior wall defects by fusing visible light and infrared images according to claim 1, characterized in that: In step S8, it includes: The defect recognition results based on appearance features in visible light images are fused with the defect recognition results based on thermal features in infrared images to obtain the true situation of the disease.

Images