Building facade deterioration detection method based on dual-discriminator fusion image generation network

Abstract

The invention discloses a building facade degradation detection method based on a double-discriminator fusion image generation network, and the method comprises the steps: obtaining a visible light and infrared high-quality detection image pair of a specified building facade, and enabling a preprocessed image to serve as a data set for training an outer wall disease detection model; constructing a double-discriminator fusion image generation network, obtaining a fusion image generator with good performance by using the obtained data set training network, and carrying out quantitative evaluation on the quality of a fusion image in the training process to ensure that the fusion image can better store texture details in a visible light image and thermal feature information in a red thermal image; and performing image instance segmentation on the fused image to detect different types of diseases. According to the method, the visual and accurate building outer wall disease fusion image can be obtained, the practicability is high, the limitation of single-mode disease detection is overcome, multiple outer wall diseases can be recognized, and the outer wall disease detection efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of detecting internal and external damage of building facade materials, and in particular to a building facade deterioration detection method based on a dual-discriminator fusion image generation network. Background Art

[0002] The exterior of a building may deteriorate with cracks, dents, leaks, and other deterioration phenomena due to reasons such as aging materials, construction errors, and natural disasters, which will affect its service life and the safety of residents. In recent years, drones and advanced imaging equipment (such as RGB cameras, infrared cameras, and laser scanners) have been combined to inspect building facades. Visible light imaging and infrared imaging can provide different visual information. The former is suitable for surface damage, while the latter can detect damage hidden under the surface, but they still face limitations in image processing and data fusion technology. Traditional image fusion methods still have problems with information loss and insufficient fusion in the process of multimodal image fusion, which limits their application in complex scenes. How to improve the accuracy of multimodal image fusion and automated detection has become a key research issue. Summary of the Invention

[0003] The present invention aims to address the shortcomings of existing technologies by providing a method for detecting building facade degradation based on a dual-discriminator fusion image generation network. This method improves the accuracy of multimodal image fusion and detection, providing support for more efficient and intelligent building degradation detection.

[0004] The object of the present invention is achieved through the following technical solutions: In a first aspect, an embodiment of the present invention provides a method for detecting building facade degradation based on a dual-discriminator fusion image generation network, comprising the following steps:

[0005] (1) Obtain visible light and infrared image pairs of building facades and preprocess them to construct a dataset;

[0006] (2) constructing a dual-discriminator fusion image generation network, wherein the dual-discriminator fusion image generation network includes a fusion image generator and two image discriminators; using the data set to train the dual-discriminator fusion image generation network, during the training process, with minimizing the loss function of the dual-discriminator fusion image generation network as the optimization goal, adjusting the parameters of the fusion image generator and the image discriminator until a preset training round is reached and the quantitative evaluation index is greater than or equal to its reference threshold, so as to obtain a trained fusion image generator;

[0007] (3) Use the trained fusion image generator to obtain the fusion image of the visible light and infrared image pairs to be detected, and perform image instance segmentation on the fusion image to detect different types of diseases.

[0008] Furthermore, the step (1) specifically includes:

[0009] First, the terrain and facade dimensions around the building to be measured are investigated; then the drone's shooting route is planned; then a drone equipped with a visible light camera and an infrared camera is used to shoot along the planned shooting route to obtain visible light and infrared image pairs of the building's facade; secondly, the visible light and infrared image pairs are preprocessed to delete unqualified image data such as focusing errors, excessive viewing angle deviations, repeated shots, and overexposure, thereby obtaining preprocessed visible light and infrared image pairs; finally, a dataset is constructed based on the preprocessed visible light and infrared image pairs.

[0010] Furthermore, the fused image generator is configured to generate a fused image based on an input pair of visible light and infrared images; the fused image generator comprises an encoding network and a decoding network, wherein the encoding network comprises a plurality of convolution blocks and a plurality of residual blocks, each of the convolution blocks comprising a convolution layer, a normalization layer, and a ReLU activation function; the decoding network comprises a plurality of convolution blocks and an output module, each of the convolution blocks comprising a convolution layer, a normalization layer, and a ReLU activation function, and the output module comprises a convolution layer and a Tanh activation function;

[0011] The visible light and infrared images are spliced ​​and input into the fusion image generator, which passes through the encoding network and decoding network in sequence to obtain the generated fusion image.

[0012] Furthermore, the two image discriminators have the same structure. The function of the image discriminators is to predict the probability of the source of the input image and output the prediction result in the form of a scalar. One of the image discriminators serves as a visible light image discriminator, which is used to predict the probability that the input image comes from a real visible light image and a fused image; the other image discriminator serves as an infrared image discriminator, which is used to predict the probability that the input image comes from a real infrared image and a fused image; each of the image discriminators includes multiple convolution blocks and output blocks, each of the convolution blocks includes a convolution layer, a normalization layer and a ReLU activation function, and the output block includes a convolution layer and a Sigmoid activation function;

[0013] The fused image output by the fusion image generator and its corresponding real visible light image are input into the visible light image discriminator to obtain the probability that the fused image comes from the real visible light image and the fused image; the fused image output by the fusion image generator and its corresponding real infrared image are input into the infrared image discriminator to obtain the probability that the fused image comes from the real infrared image and the fused image.

[0014] Furthermore, the dual discriminator fusion image generation network is trained using the dataset, specifically including:

[0015] The visible light and infrared image pairs in the dataset are spliced ​​and input into the dual-discriminator fusion image generation network. First, the fusion image generator is input to obtain the fusion image. Then, the fusion image and its corresponding real visible light image are input into the visible light image discriminator to obtain the probability that the fusion image comes from the real visible light image and the fusion image. The fusion image and its corresponding real infrared image are input into the infrared image discriminator to obtain the probability that the fusion image comes from the real infrared image and the fusion image.

[0016] The structural similarity loss and the saliency-weighted pixel loss are calculated based on the fused image and its corresponding real visible light image and infrared image, and the generation loss function is calculated based on the structural similarity loss and the saliency-weighted pixel loss; the loss function of the visible light image discriminator and the loss function of the infrared image discriminator are calculated based on the probability that the fused image comes from the real visible light image and the fused image, and the probability that the fused image comes from the real infrared image and the fused image, respectively; the loss function of the dual-discriminator fusion image generation network is calculated based on the generation loss function, the loss function of the visible light image discriminator, and the loss function of the infrared image discriminator;

[0017] With the optimization goal of minimizing the loss function of the dual-discriminator fusion image generation network, the parameters of the fusion image generator, visible light image discriminator, and infrared image discriminator are adjusted until the preset training rounds are reached and the quantitative evaluation index of the image fusion quality is greater than or equal to its reference threshold. The training is stopped to obtain the trained fusion image generator.

[0018] Furthermore, the calculation formula of the structural similarity loss is:

[0019]

[0020] Among them, L SSIM represents the structural similarity loss, V represents the visible light image, I represents the infrared image, F represents the fused image, SSIM V,F SSIM represents the structural similarity index between the visible light image and the fused image. I,F It represents the structural similarity index between the infrared image and the fused image and is calculated using the following formula:

[0021]

[0022] Among them, SSIM m,n represents the structural similarity index between image m and image n, m k and n k denotes the brightness value of the kth pixel on image m and image n respectively, μ denotes the mean value, σ denotes the standard deviation, c1, c2 and c3 denote the parameters introduced to stabilize the measurement;

[0023] The calculation formula of the pixel loss based on the saliency weight is:

[0024]

[0025] Among them, L pixel represents pixel loss, ||·||1 represents L1 norm, S I(k) Indicates the pixel saliency of the infrared image I, S V(k) It represents the pixel saliency of the visible light image V and is calculated by the following formula:

[0026]

[0027] Among them, S m(k) represents the pixel saliency of image m, m(k) represents the value of the kth pixel on image m; H m (g) represents the histogram distribution of pixel value g;

[0028] The calculation formula for the generated loss function is:

[0029] L G =L SSIM +L pixel

[0030] Among them, L G Represents the generation loss function;

[0031] The calculation formulas of the loss function of the visible light image discriminator and the loss function of the infrared image discriminator are respectively:

[0032]

[0033] in, represents the loss function of the visible light image discriminator, Denotes the loss function of infrared image discriminator, D v (V) represents the predicted probability of the visible light image discriminator outputting the visible light image V, D i (I) represents the predicted probability of the infrared image I output by the infrared image discriminator, G(I,V) represents the fused image output by the fused image generator, and D v (G(I,V)) represents the predicted probability of the visible light image discriminator for the fused image output, D i (G(I,V)) represents the predicted probability of the infrared image discriminator for the fused image output;

[0034] The calculation formula of the loss function of the dual discriminator fusion image generation network is:

[0035]

[0036] Among them, L loss Represents the loss function of the dual-discriminator fusion image generation network.

[0037] Furthermore, the quantitative evaluation indicators specifically include:

[0038] ① Information entropy: an indicator used to measure the information richness of the fused image. Its calculation formula is:

[0039]

[0040] Among them, EN represents information entropy, p g represents the probability of the g-th gray level, and n represents the total number of gray levels;

[0041] ② Average gradient: an indicator used to describe the texture details of the fused image. Its calculation formula is:

[0042]

[0043] Where AG represents the average gradient, W and H represent the width and height of the fused image respectively, and x i,j Represents the pixel value at position (i, j);

[0044] ③ Mutual information: A metric used to evaluate the similarity between the fused image and the source image. Its calculation formula is:

[0045]

[0046] Among them, MI represents the mutual information, p V (v), p I (i') and p F (f) represents the probability of the g-th gray level in the visible light image, infrared image and fused image, respectively, p I,F (i',f) represents the grayscale joint probability distribution of the infrared image and the fused image, p V,F (v,f) represents the joint probability distribution of the grayscale of the visible light image and the fused image.

[0047] Furthermore, in step (3), after using the trained fusion image generator to obtain the fusion image of the visible light and infrared image pair to be detected, the public Mask-RCNN-based algorithm is used to segment and detect the defects on the building facade in the fusion image, and different types of defects on the exterior wall can be marked with different colors and represented with clear outlines.

[0048] A second aspect of an embodiment of the present invention provides a building facade deterioration detection device based on a dual-discriminator fusion image generation network, comprising one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned building facade deterioration detection method based on the dual-discriminator fusion image generation network.

[0049] A third aspect of an embodiment of the present invention provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the program is used to implement the above-mentioned building facade deterioration detection method based on a dual-discriminator fusion image generation network.

[0050] The beneficial effects of the present invention are as follows: the present invention adopts a generative adversarial network with dual discriminators to enhance the complementary information fusion of infrared (IR) and visible light images. The generated fused image (IVIF) can simultaneously retain thermal features and visual texture details, thereby optimizing the fusion effect and achieving higher image contrast and detail presentation; the present invention adopts multiple comprehensive indicators to quantitatively evaluate the image fusion quality to ensure the quality of the fused image, which is beneficial to the next step of detection; the present invention adopts a multi-task learning method to perform target detection and instance segmentation on the fused image, thereby improving the effectiveness of facade degradation detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of a building facade deterioration detection method based on a dual-discriminator fusion image generation network of the present invention;

[0052] Figure 2 is a structural diagram of the dual discriminator fusion image generation network of the present invention;

[0053] Figure 3 It is a structural schematic diagram of the building facade deterioration detection device based on the dual-discriminator fusion image generation network of the present invention. DETAILED DESCRIPTION

[0054] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims. It should be understood that the foregoing general description and the detailed description that follows are exemplary and illustrative only and do not limit the present application.

[0055] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0056] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of..." or "when..." or "in response to determination." Moreover, the term "comprises," "comprising," or any other variant thereof is intended to cover non-exclusive inclusion, so that the process or method comprising a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process or method. In the absence of further restrictions, the elements defined by the statement "comprising a..." do not exclude the presence of other identical elements in the process, method, article, or device comprising the elements.

[0057] The present invention will be described in detail below with reference to the accompanying drawings. Unless there is any conflict, the features of the following embodiments and implementations may be combined with each other.

[0058] See also Figure 1 The building facade deterioration detection method based on a dual-discriminator fusion image generation network of the present invention specifically includes the following steps:

[0059] (1) Obtain visible light and infrared image pairs of building facades and preprocess them to construct a dataset.

[0060] Specifically, the terrain and facade dimensions around the building to be tested are first investigated; then the drone's shooting route is planned to ensure the acquisition of high-quality detection image pairs; then a drone equipped with a visible light camera and an infrared camera is used to shoot along the planned shooting route to obtain visible light and infrared image pairs of the building's facade; secondly, the visible light and infrared image pairs are preprocessed to delete unqualified image data such as focusing errors, excessive viewing angle deviations, repeated shooting, and overexposure, to obtain preprocessed visible light and infrared image pairs; finally, a dataset is constructed based on the preprocessed visible light and infrared image pairs for subsequent training of the exterior wall disease detection model.

[0061] (2) Construct a dual-discriminator fusion image generation network, where the dual-discriminator fusion image generation network includes a fusion image generator and two image discriminators, such as Figure 2 As shown in the figure, the dual discriminator fusion image generation network is trained using the dataset. During the training process, the parameters of the fusion image generator and the image discriminator are adjusted with the optimization goal of minimizing the loss function of the dual discriminator fusion image generation network until the preset training rounds are reached and the quantitative evaluation index is greater than or equal to its reference threshold, so as to obtain a trained fusion image generator.

[0062] In this embodiment, the function of the fusion image generator is to generate a fusion image with greater information content, good visual effect, and the same dimensions as the input image based on the input visible light and infrared image pair of the building facade. The fusion image generator includes an encoding network and a decoding network, wherein the encoding network includes multiple convolution blocks and multiple residual blocks, each convolution block includes a convolution layer, a normalization layer, and a ReLU activation function; the decoding network includes multiple convolution blocks and an output module, each convolution block includes a convolution layer, a normalization layer, and a ReLU activation function, and the output module includes a convolution layer and a Tanh activation function, such as Figure 2 The visible light and infrared images are stitched together and input into the fused image generator, which passes through the encoding network and decoding network in sequence to obtain the generated fused image.

[0063] Specifically, if Figure 2 As shown in the figure, the encoding network can convert the spliced ​​visible light and infrared image pairs into a low-dimensional representation, compress the matrix, and retain the information with the largest amount of information, that is, retain the most favorable information about exterior wall diseases in the visible light and infrared images; the encoding network part contains two initial convolution blocks C7-64 and C3-128 and 5 residual blocks Res-256, C7-64 and C3-128 respectively represent 7×7 convolution layers using 64 filters and 3×3 convolution layers using 128 filters, and Res-256 represents 256 filters. The decoding network then reconstructs it into a fused image and outputs it. In the decoding network, it passes through two convolution blocks U-128 and U-64 and the output module C7-3 in sequence. U-128 and U-64 respectively represent the use of 128 filters in a 3×3 convolutional layer and 64 filters in a 3×3 convolutional layer. C7-3 represents the use of 3 filters in a 7×7 convolutional layer (corresponding to the three RGB channels respectively). The final output image is generated through the 7×7 convolutional layer. The Tanh activation function is used to normalize the output value to the range of [-1,1], which is suitable for the task of generating a fused image. That is, the decoding network can output a fused image of visible light and infrared images with the same dimension as the input image, thereby obtaining a fused image that preserves the texture details in the visible light image and the thermal features in the infrared image.

[0064] In this embodiment, the two image discriminators have the same structure. The image discriminators are used to predict the probability of the source of the input image and output the prediction result in the form of a scalar. One of the image discriminators is used as a visible light image discriminator to predict the probability that the input image comes from a real visible light image and a fused image; the other image discriminator is used as an infrared image discriminator to predict the probability that the input image comes from a real infrared image and a fused image. Figure 2 As shown, each image discriminator includes multiple convolution blocks and output blocks. Each convolution block includes a convolution layer, a normalization layer, and a ReLU activation function, and the output block includes a convolution layer and a Sigmoid activation function. The fused image output by the fusion image generator and its corresponding real visible light image are input into the visible light image discriminator to obtain the probability that the fused image comes from the real visible light image and the fused image. The fused image output by the fusion image generator and its corresponding real infrared image are input into the infrared image discriminator to obtain the probability that the fused image comes from the real infrared image and the fused image.

[0065] It should be understood that the image discriminator specifically compares the similarity between the fused image and the corresponding modality source image to determine whether the generated fused image meets the similarity requirements in brightness distribution or texture distribution with the source image. If the similarity is sufficient, it can be judged as true, and this can be determined specifically by probability. Specifically, the visible light image discriminator outputs the probability that the fused image is derived from the real visible light image and the fused image. If the probability that the fused image is derived from the real visible light image is greater than the probability that the fused image is derived from the generated fused image, the judgment result is true; otherwise, the judgment result is false. Similarly, the infrared image discriminator outputs the probability that the fused image is derived from the real infrared image and the fused image. If the probability that the fused image is derived from the real infrared image is greater than the probability that the fused image is derived from the generated fused image, the judgment result is true; otherwise, the judgment result is false.

[0066] Specifically, the image discriminator undergoes multiple convolutional blocks, C3-64, C3-128, C3-256, and C3-512, and outputs blocks, C3-1. C3-64, C3-128, C3-256, and C3-512 represent 3×3 convolutional layers using 64, 128, 256, and 512 filters, respectively, while C3-1 represents a 3×3 convolutional layer using a single filter. The visible and infrared image discriminators share the same network architecture. The visible image discriminator distinguishes whether the gradient distribution (i.e., texture representation) of the fused image is consistent with that of the visible image, while the infrared image discriminator assesses the similarity of the brightness distribution between the fused image and the infrared image. To ensure that the fused image contains both the intensity information of the infrared image and the texture details of the visible image, the fused image generator must simultaneously meet the requirements of both image discriminators during the generation process.

[0067] In this embodiment, a dual-discriminator fusion image generation network is trained using a dataset. Specifically, the training includes: concatenating visible light and infrared image pairs from the dataset and inputting them into the dual-discriminator fusion image generation network, first entering the fusion image generator to obtain a fused image; then inputting the fused image and its corresponding real visible light image into the visible light image discriminator to obtain the probability that the fused image is derived from the real visible light image and the fused image; inputting the fused image and its corresponding real infrared image into the infrared image discriminator to obtain the probability that the fused image is derived from the real infrared image and the fused image; calculating a structural similarity loss and a saliency weighted (S-DW)-based pixel loss based on the fused image and its corresponding real visible light image and infrared image; calculating a generation loss function based on the structural similarity loss and the saliency weighted pixel loss; calculating a loss function for the visible light image discriminator and a loss function for the infrared image discriminator based on the probabilities that the fused image is derived from the real visible light image and the fused image, and the probabilities that the fused image is derived from the real infrared image and the fused image, respectively; and calculating a loss function for the dual-discriminator fusion image generation network based on the generation loss function, the loss function for the visible light image discriminator, and the loss function for the infrared image discriminator. With the optimization goal of minimizing the loss function of the dual-discriminator fusion image generation network, the parameters of the fusion image generator, visible light image discriminator, and infrared image discriminator are adjusted until the preset training rounds are reached and the quantitative evaluation index of the image fusion quality is greater than or equal to its reference threshold. The training is stopped to obtain the trained fusion image generator.

[0068] It's important to note that the fused image generator and image discriminator are trained using a zero-sum game approach. The fused image generator attempts to generate images that are as realistic as possible to deceive the image discriminator, while the image discriminator strives to distinguish between real and generated images. During training, the fused image generator and image discriminator continuously improve their respective capabilities until they reach a dynamic equilibrium where the fused image generator is able to produce high-quality images while the image discriminator cannot easily distinguish between real and fake images, ensuring that the images generated by the fused image generator are sufficiently realistic.

[0069] Furthermore, the calculation formula of the structural similarity loss is:

[0070]

[0071] Among them, L SSIM represents the structural similarity loss, V represents the visible light image, I represents the infrared image, F represents the fused image, SSIM V,F Represents the structural similarity index (SSIM) between the visible light image and the fused image, SSIM I,FIt represents the structural similarity index between the infrared image and the fused image and is calculated using the following formula:

[0072]

[0073] Among them, SSIM m,n represents the structural similarity index between image m and image n, m k and n k where represents the brightness value of the kth pixel in image m and image n, μ represents the mean, σ represents the standard deviation, and c1, c2, and c3 represent parameters introduced to stabilize the metric. The structural similarity index is used to evaluate the similarity between two images, quantifying differences and deformations based on the similarity of brightness, contrast, and structural details.

[0074] Furthermore, the calculation formula of pixel loss based on saliency weight is:

[0075]

[0076] Among them, L pixel represents pixel loss, ||·||1 represents L1 norm, S I(k) Indicates the pixel saliency of the infrared image I, S V(k) It represents the pixel saliency of the visible light image V and is calculated by the following formula:

[0077]

[0078] Among them, S m(k) represents the pixel saliency of image m, m(k) represents the value of the kth pixel on image m; H m (g) represents the histogram distribution of pixel value g.

[0079] Furthermore, the calculation formula for generating the loss function is:

[0080] L G =L SSIM +L pixel

[0081] Among them, L G Represents the generated loss function.

[0082] Furthermore, the calculation formulas of the loss function of the visible light image discriminator and the loss function of the infrared image discriminator are:

[0083]

[0084] in, represents the loss function of the visible light image discriminator, Denotes the loss function of infrared image discriminator, Dv (V) represents the predicted probability of the visible light image discriminator outputting the visible light image V, D i (I) represents the predicted probability of the infrared image I output by the infrared image discriminator, G(I,V) represents the fused image output by the fused image generator, and D v (G(I,V)) represents the predicted probability of the visible light image discriminator for the fused image output, D i (G(I,V)) represents the predicted probability of the infrared image discriminator for the fused image output.

[0085] Furthermore, the loss function of the dual discriminator fusion image generation network is calculated as:

[0086]

[0087] Among them, L loss Represents the loss function of the dual-discriminator fusion image generation network.

[0088] It should be noted that during the training process, the fused image generator, visible light image discriminator, and infrared image discriminator are continuously optimized to optimize the quality of the fused image generated by the fused image generator and maximize the image discriminator's ability to predict the source of the input image. The fused image generator utilizes enhanced complementary fusion of visible light and infrared images to help generate a fused image that preserves the overall structure and maintains an intensity distribution similar to the source images. During the training process, the fusion quality of the fused image needs to be quantitatively evaluated. Training can only be stopped when the quantitative evaluation index of image fusion quality is greater than or equal to its reference threshold. Because the evaluation results obtained by qualitative evaluation are relatively subjective and can only be used as a reference, the present invention uses a quantitative evaluation method system based on qualitative evaluation to provide a comprehensive assessment method for the fusion quality of fused images. It objectively evaluates the information content and edge preservation of the fused image, ensuring that the fused image can effectively preserve the information of each modality source image, the texture details in the visible light image, and the thermal characteristics in the red-hot image, thereby ensuring the effectiveness of the next step of segmentation detection.

[0089] Furthermore, the quantitative evaluation indicators specifically include:

[0090] ① Information entropy (EN): An indicator used to measure the information richness of the fused image. Its calculation formula is:

[0091]

[0092] Among them, EN represents information entropy, p g represents the probability of the gth gray level, and n represents the total number of gray levels. The higher the value of information entropy, the more information content in the fused image, that is, the better the fusion quality of the fused image.

[0093] ② Average gradient (AG): An indicator used to describe the texture details of the fused image. Its calculation formula is:

[0094]

[0095] Where AG represents the average gradient, W and H represent the width and height of the fused image respectively, and x i,j Represents the pixel value at position (i, j). The higher the average gradient, the clearer the fused image is, that is, the better the fusion quality of the fused image.

[0096] ③ Mutual Information (MI): A metric used to evaluate the similarity between the fused image and the source image. Its calculation formula is:

[0097]

[0098] Among them, MI represents the mutual information, p V (v), p I (i') and p F (f) represents the probability of the g-th gray level in the visible light image, infrared image and fused image, respectively, p I,F (i',f) represents the grayscale joint probability distribution of the infrared image and the fused image, p V,F (v,f) represents the joint grayscale probability distribution of the visible light image and the fused image. A higher value of the mutual information indicates that more source image information is retained in the fused image, that is, the fusion quality of the fused image is better.

[0099] (3) Use the trained fusion image generator to obtain the fusion image of the visible light and infrared image pairs to be detected, and perform image instance segmentation on the fusion image to detect different types of diseases.

[0100] Furthermore, in step (3), after using the trained fusion image generator to obtain a fusion image of the visible light and infrared image pair to be detected, the publicly available Mask-RCNN-based algorithm is used to segment and detect building facade defects in the fusion image. Different types of exterior wall defects can be labeled with different colors and represented with clear outlines, intuitively displaying the type of defect, with high accuracy, fast detection speed, and stable performance in complex scenes. Therefore, the method of the present invention can be used to detect defects on various complex building exterior walls.

[0101] Corresponding to the aforementioned embodiment of the building facade deterioration detection method based on a dual-discriminator fusion image generation network, the present invention also provides an embodiment of a building facade deterioration detection device based on a dual-discriminator fusion image generation network.

[0102] See also Figure 3An embodiment of the present invention provides a building facade deterioration detection device based on a dual-discriminator fusion image generation network, comprising one or more processors and a memory coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the building facade deterioration detection method based on the dual-discriminator fusion image generation network in the above embodiment.

[0103] The embodiment of the building facade deterioration detection device based on the dual discriminator fusion image generation network of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 3 As shown, this is a hardware structure diagram of a device with data processing capability for detecting building facade degradation based on a dual discriminator fusion image generation network according to the present invention, except Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0104] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0105] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0106] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the building facade deterioration detection method based on the dual-discriminator fusion image generation network in the above embodiment is implemented.

[0107] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0108] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A building facade deterioration detection method based on a dual-discriminator fusion image generation network, characterized in that: The following steps are involved: (1) Obtain visible light and infrared image pairs of building facades and preprocess them to construct a dataset; (2) constructing a dual-discriminator fusion image generation network, wherein the dual-discriminator fusion image generation network includes a fusion image generator and two image discriminators; The dual-discriminator fusion image generation network is trained using the dataset. During the training process, the parameters of the fusion image generator and the image discriminator are adjusted with the goal of minimizing the loss function of the dual-discriminator fusion image generation network until the preset training rounds are reached and the quantitative evaluation index is greater than or equal to its reference threshold, thereby obtaining a trained fusion image generator. (3) Use the trained fusion image generator to obtain the fusion image of the visible light and infrared image pairs to be detected, and perform image instance segmentation on the fusion image to detect different types of diseases.

2. The building facade deterioration detection method based on a dual discriminator fusion image generation network according to claim 1 is characterized in that: The step (1) specifically includes: First, the terrain and facade dimensions around the building to be measured are investigated; then the drone's shooting route is planned; then a drone equipped with a visible light camera and an infrared camera is used to shoot along the planned shooting route to obtain visible light and infrared image pairs of the building's facade; secondly, the visible light and infrared image pairs are preprocessed to delete unqualified image data such as focusing errors, excessive viewing angle deviations, repeated shots, and overexposure, thereby obtaining preprocessed visible light and infrared image pairs; finally, a dataset is constructed based on the preprocessed visible light and infrared image pairs.

3. The building facade deterioration detection method based on a dual discriminator fusion image generation network according to claim 1, characterized in that: The fused image generator is configured to generate a fused image based on an input pair of visible light and infrared images; the fused image generator comprises an encoding network and a decoding network, wherein the encoding network comprises a plurality of convolution blocks and a plurality of residual blocks, each of the convolution blocks comprising a convolution layer, a normalization layer, and a ReLU activation function; the decoding network comprises a plurality of convolution blocks and an output module, each of the convolution blocks comprising a convolution layer, a normalization layer, and a ReLU activation function, and the output module comprises a convolution layer and a Tanh activation function; The visible light and infrared images are spliced ​​and input into the fusion image generator, which passes through the encoding network and decoding network in sequence to obtain the generated fusion image.

4. The building facade deterioration detection method based on a dual discriminator fusion image generation network according to claim 1, characterized in that: The two image discriminators have the same structure. The function of the image discriminators is to predict the probability of the source of the input image and output the prediction result in the form of a scalar. One of the image discriminators is used as a visible light image discriminator to predict the probability that the input image comes from a real visible light image and a fused image; the other image discriminator is used as an infrared image discriminator to predict the probability that the input image comes from a real infrared image and a fused image; each of the image discriminators includes multiple convolution blocks and output blocks, each of the convolution blocks includes a convolution layer, a normalization layer and a ReLU activation function, and the output block includes a convolution layer and a Sigmoid activation function; The fused image output by the fusion image generator and its corresponding real visible light image are input into the visible light image discriminator to obtain the probability that the fused image comes from the real visible light image and the fused image; the fused image output by the fusion image generator and its corresponding real infrared image are input into the infrared image discriminator to obtain the probability that the fused image comes from the real infrared image and the fused image.

5. The building facade deterioration detection method based on a dual discriminator fusion image generation network according to claim 1, characterized in that: The method of using the data set to train the dual discriminator fusion image generation network specifically includes: The visible light and infrared image pairs in the dataset are spliced ​​and input into the dual-discriminator fusion image generation network. First, the fusion image generator is input to obtain the fusion image. Then, the fusion image and its corresponding real visible light image are input into the visible light image discriminator to obtain the probability that the fusion image comes from the real visible light image and the fusion image. The fusion image and its corresponding real infrared image are input into the infrared image discriminator to obtain the probability that the fusion image comes from the real infrared image and the fusion image. The structural similarity loss and the saliency-weighted pixel loss are calculated based on the fused image and its corresponding real visible light image and infrared image, and the generation loss function is calculated based on the structural similarity loss and the saliency-weighted pixel loss; the loss function of the visible light image discriminator and the loss function of the infrared image discriminator are calculated based on the probability that the fused image comes from the real visible light image and the fused image, and the probability that the fused image comes from the real infrared image and the fused image, respectively; the loss function of the dual-discriminator fusion image generation network is calculated based on the generation loss function, the loss function of the visible light image discriminator, and the loss function of the infrared image discriminator; With the optimization goal of minimizing the loss function of the dual-discriminator fusion image generation network, the parameters of the fusion image generator, visible light image discriminator, and infrared image discriminator are adjusted until the preset training rounds are reached and the quantitative evaluation index of the image fusion quality is greater than or equal to its reference threshold. The training is stopped to obtain the trained fusion image generator.

6. The building facade deterioration detection method based on a dual discriminator fusion image generation network according to claim 5, characterized in that: The calculation formula of the structural similarity loss is: Among them, L SSIM represents the structural similarity loss, V represents the visible light image, I represents the infrared image, F represents the fused image, SSIM V,F SSIM represents the structural similarity index between the visible light image and the fused image. I,F It represents the structural similarity index between the infrared image and the fused image and is calculated using the following formula: Among them, SSIM m,n represents the structural similarity index between image m and image n, m k and n k denotes the brightness value of the kth pixel on image m and image n respectively, μ denotes the mean value, σ denotes the standard deviation, c1, c2 and c3 denote the parameters introduced to stabilize the measurement; The calculation formula of the pixel loss based on the saliency weight is: Among them, L pixel represents pixel loss, ||·||1 represents L1 norm, S I(k) Indicates the pixel saliency of the infrared image I, S V(k) It represents the pixel saliency of the visible light image V and is calculated by the following formula: Among them, S m(k) represents the pixel saliency of image m, m(k) represents the value of the kth pixel on image m; H m (g) represents the histogram distribution of pixel value g; The calculation formula for the generated loss function is: L G =L SSIM +L pixel Among them, L G Represents the generation loss function; The calculation formulas of the loss function of the visible light image discriminator and the loss function of the infrared image discriminator are respectively: in, represents the loss function of the visible light image discriminator, Denotes the loss function of infrared image discriminator, D v (V) represents the predicted probability of the visible light image discriminator outputting the visible light image V, D i (I) represents the predicted probability of the infrared image I output by the infrared image discriminator, G(I,V) represents the fused image output by the fused image generator, and D v (G(I,V)) represents the predicted probability of the visible light image discriminator for the fused image output, D i (G(I,V)) represents the predicted probability of the infrared image discriminator for the fused image output; The calculation formula of the loss function of the dual discriminator fusion image generation network is: Among them, L loss Represents the loss function of the dual-discriminator fusion image generation network.

7. The building facade deterioration detection method based on a dual discriminator fusion image generation network according to claim 1, characterized in that: The quantitative evaluation indicators specifically include: ① Information entropy: an indicator used to measure the information richness of the fused image. Its calculation formula is: Among them, EN represents information entropy, p g represents the probability of the g-th gray level, and n represents the total number of gray levels; ② Average gradient: an indicator used to describe the texture details of the fused image. Its calculation formula is: Where AG represents the average gradient, W and H represent the width and height of the fused image respectively, and x i,j Represents the pixel value at position (i, j); ③ Mutual information: A metric used to evaluate the similarity between the fused image and the source image. Its calculation formula is: Among them, MI represents the mutual information, p V (v), p I (i') and p F (f) represents the probability of the g-th gray level in the visible light image, infrared image and fused image, respectively, p I,F (i',f) represents the grayscale joint probability distribution of the infrared image and the fused image, p V,F (v,f) represents the joint probability distribution of the grayscale of the visible light image and the fused image.

8. The building facade deterioration detection method based on a dual discriminator fusion image generation network according to claim 1, characterized in that: In step (3), after using the trained fusion image generator to obtain the fusion image of the visible light and infrared image pair to be detected, the public Mask-RCNN-based algorithm is used to segment and detect the defects on the building facade in the fusion image, and different types of defects on the exterior wall can be marked with different colors and represented with clear outlines.

9. A building facade deterioration detection device based on a dual-discriminator fusion image generation network, comprising one or more processors and a memory, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the building facade deterioration detection method based on a dual-discriminator fusion image generation network according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that A program is stored thereon, which, when executed by a processor, is used to implement the building facade deterioration detection method based on a dual-discriminator fusion image generation network according to any one of claims 1 to 8.

Images