Building Facade Defect Detection Method Based on Deep Learning Multimodal Image Fusion

Abstract

The present invention discloses a method, device and storage medium for detecting building exterior wall defects based on deep learning multi-modal image fusion. The method includes: collecting visible light images and infrared thermal images of building exterior wall defects; manually screening the visible light images and the infrared thermal images and performing registration and fusion of the visible light images and the infrared images; manually annotating the fused image set by using a semantic segmentation annotation tool; constructing a semantic segmentation network for building exterior wall defect recognition based on the Res-UNet network and performing model training; and using the trained semantic segmentation network model for building exterior wall defect recognition to identify defects in building exterior wall images. The method of the present invention improves the detection efficiency and reduces the detection cost by collecting visible light images and infrared thermal images of building exterior wall defects; overcomes the problem that image recognition technologies based on a single modality often only target certain specific defects, and combines multiple modality image information to achieve accurate recognition of various defects in building exterior walls.

Description

Technical Field

[0001] The present invention relates to the technical field of building exterior wall defect identification, and particularly to a building exterior wall defect detection method based on deep learning multi-modal image fusion. Background Art

[0002] In the early stage, the detection of building exterior wall defects mainly relied on manual inspection. In this process, inspectors needed to come to the construction site and use corresponding detection equipment to measure and record the position, size, length, width, etc. of the defects observed on the surface of the building exterior wall. The manual inspection method requires the use of professional detection equipment to approach the building surface, and requires inspectors to have professional knowledge and rich experience. With the increase in building height and building volume, problems such as gradually increasing work intensity, low detection efficiency, and insufficient detection accuracy of the manual inspection method have become increasingly prominent. At the same time, for high-rise and super high-rise buildings, manual inspection poses a danger of working at heights. In addition, a large part of the defect detection results are affected by subjective factors such as the work experience and work status of the staff. Different inspectors may obtain different detection results, and false detections and missed detections are usually prone to occur.

[0003] With the wide application of artificial intelligence technology in various fields, domestic and foreign scholars have carried out in-depth research on image-based building exterior wall defect detection to improve the automation and intelligence of the operation and maintenance of existing buildings. At present, the non-destructive detection technology of building exterior wall defects based on image processing can be mainly divided into: 1) building exterior wall defect detection based on visible light images; 2) building exterior wall defect detection based on infrared images.

[0004] The current visible light image-based recognition methods often target specific defects, such as cracks, peeling, leakage, and corrosion. Although the current technology can already identify defects well and with high accuracy, since the defects on the building exterior wall are often interrelated and the defects on the building exterior wall can be transformed under certain conditions, multiple defects on the building exterior wall still cannot be detected simultaneously. The infrared thermal imaging-based defect detection method targets internal defects that cannot be directly observed by visible light in the building. Although the detection effect is obvious, a large part of the detection results are affected by external environments such as illumination time, climate temperature, and radiation rays. Summary of the Invention

[0005] In view of the above problems, the present invention provides a building exterior wall defect detection method, device, and storage medium based on deep learning multi-modal image fusion. The method combines visible light and infrared images, combines the image information of the two modalities, breaks the limitations of using single-modal data, and realizes the accurate identification of various defects on the building exterior wall; solves the problem that the traditional building exterior wall defect recognition methods based on visible light images or infrared images often only target a specific defect.

[0006] In the first aspect of the present invention, a method for detecting building exterior wall defects based on deep learning multi-modal image fusion is provided. The method includes the following steps:

[0007] According to the terrain around the building to be detected and the size information of the building facade, route planning is carried out for the aerial photography equipment, and visible light images and infrared thermal images of the building exterior wall defects are collected according to the route planning.

[0008] Manually screen the visible light images and the infrared thermal images, and preprocess and expand the screened visible light images and infrared thermal images into effective image datasets respectively.

[0009] Crop the visible light effective image dataset and the infrared effective image dataset to the corresponding sizes, and perform registration and fusion of the visible light images and the infrared images to form a fused image set.

[0010] Unify various defect types of the building exterior wall, and use a semantic segmentation annotation tool to perform manual annotation of each image and each pixel in the fused image set to form a dataset of building exterior wall defects.

[0011] Construct a semantic segmentation network for identifying building exterior wall defects based on the Res-UNet network, and use the dataset of building exterior wall defects to train the semantic segmentation network model for identifying building exterior wall defects.

[0012] Use the trained semantic segmentation network model for identifying building exterior wall defects to identify defects in the building exterior wall image, and calculate the ratio of the corresponding defect area to the entire image area.

[0013] A further technical solution of the present invention is that: cropping the visible light effective image dataset and the infrared effective image dataset to the corresponding sizes, and performing registration and fusion of the visible light images and the infrared images to form a fused image set, which specifically includes:

[0014] Crop the infrared effective image, remove the useless information at the edge, and leave the defect information in the center.

[0015] Taking the visible light effective image as the base map and the infrared effective image as the upper layer image, adjust the transparency of the infrared effective image to 50%, and crop the visible light effective image according to the size of the infrared effective image to retain the corresponding visible light image.

[0016] Add an infrared effective image channel on the basis of the three channels of the visible light effective image to achieve the purpose of fusing visible light and infrared thermal images.

[0017] A further technical solution of the present invention is that: the semantic segmentation annotation tool is the Eiseg semantic segmentation annotation tool.

[0018] A further technical solution of the present invention is that the semantic segmentation network for building exterior wall defect recognition is constructed based on the Res-UNet network. The structure of the semantic segmentation network for building exterior wall defect recognition adopts the Encoder-Decoder form. The size of the input image of the semantic segmentation network for building exterior wall defect recognition is 512×512×4, where 512×512 is the size of the input image, and 4 represents the number of channels of the input image.

[0019] A further technical solution of the present invention is that when training the semantic segmentation network model for building exterior wall defect recognition, the Dice coefficient is introduced as the loss function to measure the difference between the actual variable value and the predicted value.

[0020] In the second aspect of the present invention, a building exterior wall defect detection device based on deep learning multi-modal image fusion includes:

[0021] An image acquisition module, which is used to plan the flight route of the aerial photography device according to the terrain around the building to be detected and the size information of the building facade, and collect visible light images and infrared thermal images of the building exterior wall defects according to the flight route plan;

[0022] An image screening module, which is used to manually screen the visible light images and the infrared thermal images, and preprocess and expand the screened visible light images and infrared thermal images respectively into effective image datasets;

[0023] An image fusion module, which is used to crop the visible light effective image dataset and the infrared effective image dataset to the corresponding size, and perform registration and fusion of the visible light image and the infrared image to form a fused image set;

[0024] An image annotation module, which is used to unify various defect types of the building exterior wall, and use a semantic segmentation annotation tool to perform manual annotation of each image and each pixel of the fused image set to form a dataset of building exterior wall defects;

[0025] A network model construction and training module, which is used to construct a semantic segmentation network for building exterior wall defect recognition based on the Res-UNet network, and use the dataset of building exterior wall defects to train the semantic segmentation network model for building exterior wall defect recognition;

[0026] A defect recognition module, which is used to use the trained semantic segmentation network model for building exterior wall defect recognition to recognize the defects of the building exterior wall image, and calculate the ratio of the corresponding defect area to the entire image area.

[0027] In a third aspect of the present invention, there is provided a device for detecting building exterior wall defects based on deep learning multi-modal image fusion, comprising: a processor; and a memory, wherein the memory stores a computer-executable program which, when executed by the processor, performs the above-mentioned method for detecting building exterior wall defects based on deep learning multi-modal image fusion.

[0028] In a fourth aspect of the present invention, there is provided a storage medium having stored thereon a program which, when executed by a processor, causes the processor to perform the above-mentioned method for detecting building exterior wall defects based on deep learning multi-modal image fusion.

[0029] The method, device and storage medium for detecting building exterior wall defects based on deep learning multi-modal image fusion provided by the present invention solve the problems of time-consuming, laborious, high cost and certain danger existing in traditional manual detection. By using an unmanned aerial vehicle to collect visible light images and infrared thermal images of building exterior wall defects, the detection efficiency is improved, the detection cost is reduced, and accidents during the detection process are prevented; based on single-modal image recognition technology, it often only targets certain specific defects and cannot achieve accurate detection for various types of defects. Among them, the technology for detecting building exterior wall defects based on visible light images has good detection effects on apparent defects such as cracks, peeling, and leakage, and the technology for detecting building exterior wall defects based on infrared images has good detection effects on hidden defects such as hollowing.

[0030] In summary, the beneficial effects of the present invention are mainly as follows:

[0031] 1) An aerial photography device is used to carry a high-resolution camera and an infrared thermal imaging camera to collect images of building exterior wall defects, which is not restricted by the height and shape of the building, and can realize the detection of exterior wall defects of super high-rise buildings and special-shaped buildings;

[0032] 2) Compared with the manual detection method, the present invention solves the problems of time-consuming, laborious, high cost and certain danger existing in the manual detection technology, improves the detection efficiency, reduces the detection cost, and reduces the safety risk of detecting high-altitude building defects;

[0033] 3) The present invention collects visible light images and infrared thermal images of building exterior wall defects through an aerial photography device, fuses the image information of the two modalities, makes a dataset of building exterior wall defect images, constructs a semantic segmentation network based on deep learning, and trains a building exterior wall defect recognition model through the deep learning network to realize the automatic and intelligent recognition of the defect position and area size, greatly improving the detection accuracy and the level of intelligence;

[0034] 4) The present invention proposes a method for detecting building exterior wall defects based on deep learning multi-modal image fusion. By fusing visible light and infrared images and combining the image information of the two modalities, the limitation of using single-modal data is broken, so as to achieve accurate identification of various defects on the building exterior wall; it solves the problem that the conventional methods for identifying building exterior wall defects based on visible light images or infrared images often only target a specific defect.

[0035] 5) The method for detecting building exterior wall defects based on deep learning multi-modal image fusion proposed by the present invention can realize regular and rapid assessment and detection of various types of defects on the building exterior walls in urban blocks. For buildings located in areas with large pedestrian flows, urban blocks after experiencing extremely bad weather, and old buildings with existing hollowing and peeling phenomena, the test and assessment frequency can be increased; the present invention solves the problem that various defects on the building exterior wall cannot be accurately and comprehensively detected, provides accurate data for early warning and repair and reinforcement of various defects in old buildings, can reduce and even avoid related accidents, and can ensure the safety of the people. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a schematic flowchart of the method for detecting building exterior wall defects based on deep learning multi-modal image fusion in Embodiment 1 of the present invention;

[0037] Figure 2 is a schematic structural diagram of the device for detecting building exterior wall defects based on deep learning multi-modal image fusion in Embodiment 2 of the present invention;

[0038] Figure 3 is the architecture of a computer device in Embodiment 3 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] The present invention will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. In addition, it should be noted that only the parts related to the present invention rather than all the structures are shown in the drawings for the convenience of description.

[0040] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts depict the steps as sequential processes, many of the steps can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0041] Embodiments of the present invention provide the following embodiments for a method, apparatus, and storage medium for detecting building exterior wall defects based on deep learning multi-modal image fusion:

[0042] Based on Embodiment 1 of the present invention

[0043] This embodiment is used to illustrate the method for detecting building exterior wall defects based on deep learning multi-modal image fusion. Refer to Figure 1 , which is a schematic flow diagram of the method for detecting building exterior wall defects based on deep learning multi-modal image fusion. In the specific implementation process, the implementation device includes a multi-rotor drone with a flight path planning system, a gimbal, an infrared thermal imaging camera, a high-resolution camera, and a server with high computing power; the gimbal is installed on the aircraft platform of the multi-rotor drone, and there is a shock absorption device between the gimbal and the multi-rotor unmanned aircraft. There is an ordinary camera and an infrared thermal imaging camera on the gimbal. The method specifically includes the following steps:

[0044] S110. Perform route planning for the aerial photography equipment according to the terrain around the building to be detected and the size information of the building facade, and collect visible light images and infrared thermal images of the building exterior wall defects according to the route planning;

[0045] In the specific implementation process, perform route planning for the multi-rotor drone according to the terrain around the building to be detected and the size information of the building facade, and collect visible light images and infrared thermal images of the building exterior wall defects according to the predetermined planned route;

[0046] S120. Manually screen the visible light images and the infrared thermal images, and preprocess and expand the screened visible light images and infrared thermal images into effective image datasets respectively;

[0047] In the specific implementation process, through manual inspection, clean the basic images with unfocused shooting, blurred shooting, uneven brightness, incomplete images, problematic formats, and defective contents; increase the number of effective images in the existing dataset through rotation, mirroring, scaling, cropping, stitching, brightness change, color balance shift, gray scale change, etc., to improve the generalization ability and robustness of the model;

[0048] S130. Crop the visible light effective image dataset and the infrared effective image dataset to the corresponding sizes, and perform registration and fusion of the visible light images and the infrared images to form a fused image set;

[0049] In the specific implementation process, crop the visible light images and the infrared images to the corresponding sizes, and perform registration and fusion of the visible light and infrared images;

[0050] S140. Unify various defect types of the building exterior wall, and use a semantic segmentation annotation tool to perform manual annotation on each image and each pixel of the fused image set to form a data set of building exterior wall defects;

[0051] In the specific implementation process, unify the naming, numbering, color, format, and annotation format of various defect types of the building exterior wall, and use semantic segmentation annotation tools such as Eiseg to perform manual annotation on each image and each pixel, and unify the format to form a data set of building exterior wall defects.

[0052] S150. Build a semantic segmentation network for building exterior wall defect recognition based on the Res-UNet network, and use the data set of building exterior wall defects to train the semantic segmentation network model for building exterior wall defect recognition;

[0053] In the specific implementation process, build a semantic segmentation network for building exterior wall defect recognition based on the Res-UNet network, and perform network model training on the collected building exterior wall data set on the server with the environment set up to obtain the relevant parameters of the semantic segmentation model for building exterior wall defects, which are used as the benchmark for building exterior wall defect detection;

[0054] S160. Use the trained semantic segmentation network model for building exterior wall defect recognition to identify defects in the building exterior wall image, and calculate the ratio of the area of the corresponding defect to the area of the entire image.

[0055] In the specific implementation process, use the semantic segmentation model obtained in S150 to identify defects in the building exterior wall image, segment and extract defects such as cracks, hollowing, peeling, and leakage, and calculate the ratio of the area of the corresponding defect to the area of the entire image.

[0056] Specifically, S130 is mainly divided into three parts. First, crop the infrared thermal image to remove the useless information on the edge and leave the defect information in the center; use the visible light image as the base map and the infrared thermal image as the upper layer image, adjust the transparency of the infrared thermal image to 50%, and crop the visible light image according to the size of the infrared thermal image to retain the corresponding visible light image; finally, add the infrared thermal image channel on the basis of the three channels of the visible light image to achieve the purpose of quickly fusing the visible light and infrared thermal images;

[0057] Preferably, in S140, the various defect types of the building exterior wall are mainly divided into: cracks in the building exterior wall, hollowing of the concrete protective layer, hollowing of tiles, peeling of the concrete protective layer, peeling of tiles, leakage, etc.; the Eiseg semantic segmentation annotation tool used can quickly and accurately label the true values of building exterior wall defects, greatly improving the annotation rate;

[0058] Preferably, the artificial intelligence recognition algorithm adopted in S150 is an improved Res-UNet network structure. This network structure adopts the form of Encoder-Decoder and is in the shape of a 'U', so it is called UNet. Moreover, each block has a residual connection part, so it is called Res-Unet. The size of the input image of this network is 512×512×4, where 512×512 is the size of the input image, and 4 represents the number of channels of the input image. The number of input channels of the traditional Unet is 3. Since the visible light image and the infrared image are fused, the number of channels of the input image becomes 4 (R, G, B, T).

[0059] Furthermore, the loss function is used to measure the difference between the actual variable value and the predicted value, and plays a guiding role in the process of model training and optimization. The various weight parameters of the model are adjusted and improved through the feedback of the loss function. The smaller the loss value, the more accurate the predicted value. This model introduces the Dice coefficient, and its calculation formula is as follows:

[0060]

[0061] The Dice coefficient essentially measures the overlapping part of two samples. The range of this index is from 0 to 1, where '1' represents complete overlap. In order to form a loss function that can be minimized, 1-Dice is used as the loss function.

[0062] The evaluation indicators include Accuracy, Precision, Recall, and F1Score. The higher the value of each parameter, the better the defect segmentation effect of the network and the better the comprehensive performance. The calculation formulas are as follows:

[0063]

[0064]

[0065]

[0066]

[0067] Based on Embodiment 2 of the present invention

[0068] A building exterior wall defect detection device 200 provided by the second embodiment of the present invention can execute the building exterior wall defect detection method provided by the first embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. This device can be implemented in the form of software and / or hardware (integrated circuit), and is generally integrated in a server or a terminal device. Figure 22 is a schematic diagram of the structure of a building exterior wall defect detection device 200 based on deep learning multimodal image fusion in Example 2 of the present invention. Figure 2 The building exterior wall defect detection device 200 based on deep learning multimodal image fusion according to the embodiment of the present invention may specifically include:

[0069] The image acquisition module 210 is used to plan a flight path for the aerial photography device according to the terrain around the inspected building and the size information of the building facade, and to collect visible light images and infrared thermal images of defects on the building exterior wall according to the flight path plan;

[0070] An image screening module 220, configured to manually screen the visible light image and the infrared thermal image, pre-process the screened visible light image and the infrared thermal image, and expand them into a valid image data set;

[0071] An image fusion module 230 is used to crop the visible light effective image data set and the infrared effective image data set to corresponding sizes, and perform registration and fusion of the visible light image and the infrared image to form a fused image set;

[0072] An image annotation module 240 is used to unify multiple types of building exterior wall defects, and use a semantic segmentation annotation tool to manually annotate the fused image set image by image and pixel by pixel to form a data set of building exterior wall defects;

[0073] A network model construction and training module 250 is used to construct a semantic segmentation network for building exterior wall defect recognition based on a Res-UNet network, and to train a semantic segmentation network model for building exterior wall defect recognition using the data set of building exterior wall defects;

[0074] The defect recognition module 260 is used to use the trained building exterior wall defect recognition semantic segmentation network model to perform defect recognition on the building exterior wall image and calculate the ratio of the corresponding defect area to the entire image area.

[0075] In addition to the above-mentioned units, the building exterior wall defect detection device 200 based on deep learning multimodal image fusion may also include other components. However, since these components are not related to the content of the embodiment of the present disclosure, their illustration and description are omitted here.

[0076] The specific working process of the building exterior wall defect detection device 200 based on deep learning multimodal image fusion refers to the description of the above-mentioned building exterior wall defect detection method based on deep learning multimodal image fusion Example 1, and will not be repeated here.

[0077] Embodiment 3 based on the present invention

[0078] The system according to the embodiment of the present invention can also be used byFigure 3 implemented with the architecture of the computing device shown. Figure 3 The architecture of the computing device is shown. As Figure 3 shown, computer system 301, system bus 303, one or more CPUs 304, input / output 302, memory 305, etc. Memory 305 can store various data or files used for computer processing and / or communication, as well as program instructions executed by the CPU, including the program instructions for implementing the method of Embodiment 1. Figure 3 The architecture shown is only exemplary. When implementing different devices, one or more components in Figure 3 are adjusted according to actual needs. Memory 305, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the building exterior wall defect detection method based on deep learning multi-modal image fusion in the embodiments of the present invention (for example, the image acquisition module 210, image screening module 220, image fusion module 230, image annotation module 240, network model construction and training module 250, and defect recognition module 260 in the building exterior wall defect detection device 200 based on deep learning multi-modal image fusion). One or more CPUs 304 execute various functional applications and data processing of the system of the present invention by running the software programs, instructions, and modules stored in memory 305, that is, implement the above-mentioned building exterior wall defect detection method based on deep learning multi-modal image fusion, and the method includes:

[0079] Planning the flight path of the aerial photography device according to the terrain around the building to be detected and the size information of the building facade, and collecting visible light images and infrared thermal images of the building exterior wall defects according to the flight path plan;

[0080] Manually screening the visible light images and the infrared thermal images, and preprocessing and expanding the screened visible light images and infrared thermal images respectively into effective image datasets;

[0081] Cropping the visible light effective image dataset and the infrared effective image dataset to the corresponding sizes, and performing registration and fusion of the visible light images and the infrared images to form a fused image set;

[0082] Unifying various defect types of the building exterior wall, and using a semantic segmentation annotation tool to perform manual annotation of each image and each pixel in the fused image set to form a dataset of building exterior wall defects;

[0083] Constructing a building exterior wall defect recognition semantic segmentation network based on the Res-UNet network, and using the dataset of building exterior wall defects to train the building exterior wall defect recognition semantic segmentation network model;

[0084] Use the trained semantic segmentation network model for identifying building exterior wall defects to identify defects in the building exterior wall image,

[0085] and calculate the ratio of the corresponding defect area to the area of the entire image.

[0086] Certainly, for the server provided by the embodiments of the present invention, its processor is not limited to performing the method operations as described above, and can also perform related operations in the building exterior wall defect detection method based on deep learning multi-modal image fusion provided by any embodiment of the present invention.

[0087] The memory 305 may mainly include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory 305 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some instances, the memory 305 may further include a memory remotely set relative to one or more CPUs 304, and these remote memories can be connected to the device through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0088] The input / output 302 can be used to receive input digital or character information, and generate key signal inputs related to the user settings and function control of the device. The input / output 302 may further include a display device such as a display screen.

[0089] Based on Embodiment 4 of the present invention

[0090] The embodiments of the present invention can also be implemented as a computer-readable storage medium. A computer program is stored on the computer-readable storage medium according to Embodiment 4. When the computer program is executed by a processor, the building exterior wall defect detection method according to Embodiment 1 of the present invention described with reference to the above drawings can be executed.

[0091] Certainly, for a storage medium provided by the embodiments of the present invention containing computer-executable instructions, the computer-executable instructions are not limited to the method operations as described above, and can also perform related operations in the building exterior wall defect detection method based on deep learning multi-modal image fusion provided by any embodiment of the present invention.

[0092] The computer-readable storage medium of the embodiments of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0093] The computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0094] The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the above.

[0095] The computer program code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or terminal. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (for example, by connecting through an Internet service provider via the Internet).

[0096] In summary, as can be seen from the embodiments, a building exterior wall defect detection method, device and storage medium based on deep learning multi-modal image fusion provided by the present invention solve the problems of time-consuming, laborious, high cost and certain danger existing in traditional manual detection. The visible light images and infrared thermal images of the building exterior wall defects are collected by drones to improve the detection efficiency, reduce the detection cost and prevent accidents during the detection process. The image recognition technology based on a single modality often only targets certain specific defects and cannot accurately detect various types of defects. Among them, the building exterior wall defect detection technology based on visible light images has good detection effects on apparent defects such as cracks, peeling and leakage, and the building exterior wall defect detection technology based on infrared images has good detection effects on hidden defects such as hollowing. In summary, the beneficial effects of the present invention are mainly as follows: The aerial photography equipment is used to carry a high-resolution camera and an infrared thermal imaging camera to collect the building exterior wall defect images, which is not limited by the height and shape of the building, and can realize the detection of the exterior wall defects of super high-rise buildings and special-shaped buildings. Compared with the manual detection method, the present invention solves the problems of time-consuming, laborious, high cost and certain danger existing in the manual detection technology, improves the detection efficiency, reduces the detection cost and reduces the safety risk of the high-altitude building defect detection. The present invention collects the visible light images and infrared thermal images of the building exterior wall defects through the aerial photography equipment, fuses the image information of the two modalities, makes a building exterior wall defect image dataset, constructs a semantic segmentation network based on deep learning, and trains a building exterior wall defect recognition model through the deep learning network to realize the automatic and intelligent recognition of the defect position and area size, greatly improving the detection accuracy and the intelligent level. The present invention proposes a building exterior wall defect detection method based on deep learning multi-modal image fusion. By fusing visible light and infrared images and combining the image information of the two modalities, the limitation of using a single modality data is broken to realize the accurate recognition of various defects of the building exterior wall. It solves the problem that the previous building exterior wall defect recognition methods based on visible light images or infrared images often only target a specific defect. The building exterior wall defect detection method based on deep learning multi-modal image fusion proposed by the present invention can realize the regular and rapid assessment and detection of various types of defects of the building exterior wall in urban blocks. For buildings located in areas with large pedestrian flow, urban blocks after experiencing extremely bad weather, and old buildings with existing hollowing and peeling phenomena, the test assessment frequency can be increased. The present invention solves the problem that various defects of the building exterior wall cannot be accurately and comprehensively detected, provides accurate data for the early warning and repair and reinforcement of various defects of old buildings, can reduce and even avoid related accidents, and can ensure the safety of the people.

[0097] Note that the above is only a preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein. Various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, it may also include more other equivalent embodiments, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A building exterior wall defect detection method based on deep learning multi-modal image fusion, characterized in that The method includes the following steps: Plan the flight path of the aerial photography device according to the terrain around the building to be detected and the size information of the building facade, and collect visible light images and infrared thermal images of the building exterior wall defects according to the flight path plan; Manually screen the visible light images and the infrared thermal images, and preprocess and expand the screened visible light images and infrared thermal images into effective image datasets respectively; Crop the visible light effective image dataset and the infrared effective image dataset to the corresponding sizes, and perform registration and fusion of the visible light images and the infrared images to form a fused image set; Unify various defect types of the building exterior wall, and use a semantic segmentation annotation tool to perform manual annotation of each image and each pixel in the fused image set to form a dataset of building exterior wall defects; Construct a semantic segmentation network for building exterior wall defect recognition based on the Res-UNet network, and use the dataset of building exterior wall defects to train the semantic segmentation network model for building exterior wall defect recognition; Use the trained semantic segmentation network model for building exterior wall defect recognition to identify defects in the building exterior wall image, and calculate the ratio of the corresponding defect area to the entire image area; Crop the visible light effective image dataset and the infrared effective image dataset to the corresponding sizes, and perform registration and fusion of the visible light images and the infrared images to form a fused image set, specifically including: Crop the infrared effective image to remove the useless information at the edges and leave the defect information in the center; Use the visible light effective image as the base map and the infrared effective image as the upper layer image, adjust the transparency of the infrared effective image to 50%, and crop the visible light effective image according to the size of the infrared effective image to retain the corresponding visible light image; Add an infrared effective image channel on the basis of the three channels of the visible light effective image to achieve the purpose of fusing visible light and infrared thermal images.

2. The method for detecting building exterior wall defects based on deep learning multi-modal image fusion according to claim 1, wherein The semantic segmentation annotation tool is the Eiseg semantic segmentation annotation tool.

3. The method for detecting building exterior wall defects based on deep learning multi-modal image fusion according to claim 1, wherein The semantic segmentation network for building exterior wall defect recognition is constructed based on the Res-UNet network. The structure of the semantic segmentation network for building exterior wall defect recognition adopts the Encoder-Decoder form. The size of the input image of the semantic segmentation network for building exterior wall defect recognition is 512×512×4, where 512×512 is the size of the input image, and 4 represents the number of channels of the input image.

4. The building exterior wall defect detection method based on deep learning multi-modal image fusion according to claim 1, characterized in that, When training the semantic segmentation network model for building exterior wall defect recognition, the model introduces the Dice coefficient as the loss function to measure the difference between the actual variable value and the predicted value.

5. An architectural exterior wall defect detection device based on deep learning multi-modal image fusion, characterized in that, Including: An image acquisition module, which is used to plan the flight path of the aerial photography device according to the terrain around the building to be detected and the size information of the building facade, and collect visible light images and infrared thermal images of the building exterior wall defects according to the flight path plan; An image screening module, which is used to manually screen the visible light images and the infrared thermal images, and preprocess and expand the screened visible light images and infrared thermal images into effective image datasets respectively; An image fusion module, configured to crop the visible light effective image dataset and the infrared effective image dataset to corresponding sizes, and perform registration and fusion of the visible light image and the infrared image to form a fused image set; An image annotation module, configured to unify various defect types of building facades, and use a semantic segmentation annotation tool to perform manual annotation of each image and each pixel in the fused image set to form a dataset of building facade defects; A network model construction and training module, configured to construct a semantic segmentation network for building facade defect recognition based on the Res-UNet network, and use the dataset of building facade defects to train the semantic segmentation network model for building facade defect recognition; A defect recognition module, configured to use the trained semantic segmentation network model for building facade defect recognition to recognize defects in building facade images, and calculate the ratio of the corresponding defect area to the entire image area; Cropping the visible light effective image dataset and the infrared effective image dataset to corresponding sizes, and performing registration and fusion of the visible light image and the infrared image to form a fused image set, specifically including: Cropping the infrared effective image to remove useless information at the edges and leaving the defect information in the center; Taking the visible light effective image as the base image and the infrared effective image as the upper image, adjusting the transparency of the infrared effective image to 50%, and cropping the visible light effective image according to the size of the infrared effective image to retain the corresponding visible light image; Adding an infrared effective image channel on the basis of the three channels of the visible light effective image to achieve the purpose of fusing the visible light and infrared thermal images.

6. An architectural exterior wall defect detection device based on deep learning multi-modal image fusion, characterized in that, Including: A processor; And a memory, wherein a computer executable program is stored in the memory, and when the computer executable program is executed by the processor, the building facade defect detection method based on deep learning multi-modal image fusion according to any one of claims 1-4 is executed.

7. A storage medium, on which a program is stored, characterized in that, When the program is executed by the processor, the building facade defect detection method based on deep learning multi-modal image fusion according to any one of claims 1-4 is implemented.

Images