A Method for Detection and Health Assessment of External Wall Defects Based on Fusion-Segmentation Joint Network

Abstract

This invention discloses a method for detecting and assessing the health of exterior walls based on a fusion-segmentation joint network. The method includes: acquiring infrared thermal images and visible light images of exterior wall defects; constructing an infrared image fusion dataset after preprocessing; constructing a fusion-detection joint network comprising an image fusion network module and a segmentation detection network module, and achieving collaborative optimization through a joint loss function; pre-training using the MSRS dataset to obtain the optimal parameters for dynamic pixel fusion weights and loss function weights; freezing the parameters of the image fusion network module and optimizing the segmentation detection network module using the self-built fusion dataset; segmenting and detecting defects in the building exterior wall images using the trained fusion-detection joint network; and statistically analyzing the defect area and distribution based on the segmentation detection results to comprehensively evaluate the health status of the exterior walls. This invention achieves automated defect segmentation detection and facade health assessment, improving the accuracy of defect detection.

Description

Technical Field

[0001] This invention relates to the field of infrared image fusion and building exterior wall defect recognition technology, and in particular to a method for exterior wall defect detection and health assessment based on a fusion-segmentation joint network. Background Technology

[0002] Common defects on building facades, such as wall cracks, hollow areas, water seepage, and peeling, not only affect the safety and stability of buildings but also threaten public safety. Therefore, comprehensive automated detection and health assessment of surface and subsurface defects on building facades are of great significance.

[0003] Infrared thermal imaging and visible light imaging have become mainstream technologies for defect detection, but each has significant limitations. Visible light image-based identification methods can currently only identify specific surface defects, such as cracks and peeling. Infrared thermal imaging can detect hidden defects such as hollow areas and water seepage, but it suffers from low contrast and susceptibility to environmental interference. Many image fusion technologies attempt to combine the advantages of infrared and visible light to compensate for the shortcomings of a single modality. However, traditional fusion methods are often optimized for visual effects rather than designed for the needs of downstream tasks, resulting in poor fusion performance in defect detection. The key challenge is to achieve efficient complementarity of information from both modalities during fusion, considering the complex scenes of building facades, to achieve efficient and accurate defect detection. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for detecting and assessing the health of exterior walls based on a fusion-segmentation joint network. This invention utilizes a dynamically weighted fusion network and a highly efficient segmentation network for collaborative optimization during the detection process, overcoming the limitations of single-modal imaging and the adaptability issues of fusion, thus improving the accuracy of automatic detection and enabling a scientific and quantitative assessment of the health status of building facades based on the detection results.

[0005] The objective of this invention is achieved through the following technical solution: The first aspect of this invention provides a method for detecting and assessing the health of exterior wall defects based on a fusion-segmentation joint network, comprising the following steps:

[0006] (1) Collect infrared thermal images and visible light images of external wall defects and preprocess them to construct a fused image dataset; label the fused image pairs in the fused image dataset and amplify them. After amplification, randomly divide them into training set and test set according to proportion.

[0007] (2) Construct a fusion-detection joint network, which includes a pixel-weighted image fusion network module and a segmentation detection network module. The image fusion network module is used to fuse the fused image pair, i.e., the registered infrared thermal image and the visible light image, pixel by pixel to obtain the final fused image. The segmentation detection network module is used to process the final fused image to obtain the category, localization bounding box and pixel-level segmentation mask for each instance.

[0008] (3) The fusion-detection joint network is pre-trained using the publicly available MSRS dataset. During the pre-training process, the parameters of the fusion-detection joint network are adjusted with the goal of minimizing the joint loss function in order to obtain a pre-trained fusion-detection joint network.

[0009] (4) Use the training set in step (1) to train the pre-trained fusion-detection joint network. During the training process, with the goal of minimizing the joint loss function, freeze the parameters of the image fusion network module and only adjust the parameters of the segmentation and detection network module to obtain the final trained fusion-detection joint network.

[0010] (5) Input the infrared thermal imaging images and visible light images to be tested in the test set into the finally trained fusion-detection joint network to perform disease segmentation and detection, and obtain the segmentation and detection results, including the category, localization bounding box and pixel-level segmentation mask of each instance.

[0011] (6) Based on the segmentation and detection results obtained in step (5), the area and distribution density of the disease are statistically analyzed, and the health of the exterior wall is evaluated by a comprehensive scoring index.

[0012] Furthermore, step (1) specifically includes the following sub-steps:

[0013] (1.1) Use infrared thermal imaging cameras and visible light cameras to collect infrared thermal imaging images and visible light images of external wall defects;

[0014] (1.2) Preprocessing of infrared thermal imaging images and visible light images is as follows: First, filter and remove erroneous image data such as focusing errors, excessive viewing angle deviations, and repeated shooting in infrared thermal imaging images and visible light images; then register and crop the infrared thermal imaging images and visible light images of different resolutions to obtain registered infrared thermal imaging images and visible light images.

[0015] (1.3) Construct a fused image dataset based on the registered infrared thermal imaging image and the visible light image;

[0016] (1.4) Use annotation tools to perform multi-class target mask segmentation annotation on the fused image pairs in the fused image dataset at the pixel level, and use them as semantic segmentation labels; and use a variety of data augmentation methods to augment the fused image dataset. After augmentation, the dataset is randomly divided into training set and test set according to the proportion.

[0017] Furthermore, the image fusion network module specifically includes:

[0018] First, the registered infrared thermal image and the visible light image are converted to grayscale and then stitched together along the channel direction. Next, convolutional operations through multiple convolutional layers are used to obtain the weights of each pixel in the infrared thermal image and the visible light image. Then, these weights are combined with the source image using element-wise multiplication and addition operations to generate a grayscale fused image. Finally, the color mode of the grayscale fused image is converted to RGB to obtain a color fused image, which serves as the final fused image, represented as:

[0019]

[0020] Among them, I vif W represents the pixel values ​​of the final fused image. irt I represents the weighting coefficient of each pixel in an infrared thermal imaging image. irt W represents the pixel value of an infrared thermal imaging image. vis I represents the weight coefficient of each pixel in a visible light image. vis Represents the pixel values ​​of a visible light image. This indicates element-wise multiplication.

[0021] Furthermore, the segmentation detection network module specifically includes:

[0022] The segmentation detection network module employs a Mask R-CNN network, which processes targets of different scales through a feature pyramid. The segmentation detection network module is used to process the final fused image output by the image fusion network module to achieve instance segmentation and obtain the category, local bounding box, and pixel-level segmentation mask for each instance.

[0023] Furthermore, step (3) specifically includes:

[0024] Infrared thermal imaging images and visible light images from the publicly available MSRS dataset are input into a fusion-detection joint network. The final fused image is obtained through the image fusion network module, and the category, local bounding box, and pixel-level segmentation mask of each instance in the final fused image are obtained through the segmentation and detection network module.

[0025] The fusion loss function is calculated based on the final fused image output by the image fusion network module and its corresponding source image, where the source image refers to the pair of infrared thermal imaging image and visible light image; the semantic segmentation loss function is calculated based on the category, local bounding box, and pixel-level segmentation mask of each instance output by the segmentation and detection network module and its corresponding semantic segmentation label in the MSRS dataset; and the joint loss function is calculated based on the fusion loss function and the semantic segmentation loss function.

[0026] During pre-training, the parameters of the fusion-detection joint network are adjusted with the goal of minimizing the joint loss function. The weights of each pixel in the infrared thermal imaging image and the visible light image in the image fusion network module, as well as the weights of texture loss and semantic segmentation loss function in the joint loss function, are dynamically adjusted until a preset training epoch is reached or the loss threshold is less than a preset threshold. The optimal weights of each pixel in the infrared thermal imaging image and the visible light image, as well as the optimal weights of texture loss and semantic segmentation loss function, are obtained, resulting in the pre-trained fusion-detection joint network.

[0027] Furthermore, the joint loss function includes a fusion loss function and a semantic segmentation loss function, and its calculation formula is as follows:

[0028] L total =L vif +βL detect

[0029] Among them, L total L represents the joint loss function. vif L represents the fusion loss function. detect Let β represent the semantic segmentation loss function, and let β represent the weights of the semantic segmentation loss function.

[0030] The fusion loss function includes brightness loss and texture loss, and its calculation formula is as follows:

[0031] L vif =L int +αL txt

[0032] Among them, L int L represents the loss of brightness. txt α represents the texture loss, and α represents the weight of the texture loss.

[0033] The formula for calculating the brightness loss is:

[0034]

[0035] Where H and W represent the height and width of the infrared thermal image or the visible light image, respectively, ||·||1 represents the L1 norm, and max(Iirt ,I vis This represents the maximum pixel value taken from both the infrared thermal image and the visible light image at each pixel location;

[0036] The formula for calculating the texture loss is:

[0037]

[0038] in, This represents the Sobel gradient operator. and These represent the gradient magnitudes of the fused image, the infrared thermal image, and the visible light image, respectively. Indicates taking and The maximum of these two gradient magnitudes.

[0039] The semantic segmentation loss function includes classification loss, bounding box loss, and mask loss, and its calculation formula is as follows:

[0040] L detect =L seg +L cls +L mask

[0041] Among them, L seg L represents the classification loss. cls L represents the bounding box loss. mask This indicates the loss of the mask.

[0042] Furthermore, step (6) specifically includes:

[0043] The segmentation detection results obtained in step (5) include the category, location bounding box and pixel-level segmentation mask of each instance. Based on the category, location bounding box and pixel-level segmentation mask of all instances, the segmentation results of instances are statistically analyzed by category. The area and location distribution density of each category are calculated according to the pixel-level segmentation mask. The comprehensive scoring index is calculated according to the area and location distribution density of each category. The health of the exterior wall is evaluated by the calculated comprehensive scoring index.

[0044] Furthermore, the comprehensive scoring index is calculated and obtained through the following steps:

[0045] (6.1) Calculate the percentage of damaged area, which is defined as the ratio of the total damaged area to the total wall area. The formula is as follows:

[0046]

[0047] Among them, R defect Indicates the percentage of the area affected by disease; The area of ​​the i-th disease, where N1 represents the total number of diseases; Let A represent the area of ​​the j-th ancillary facility, and N2 represent the total number of ancillary facilities; wall This indicates the total area of ​​the wall surface;

[0048] (6.2) Calculate the average area of ​​a single disease, which is defined as the ratio of the total disease area to the number of diseases. The calculation formula is as follows:

[0049]

[0050] Among them, C defect This represents the average area of ​​a single disease.

[0051] (6.3) Calculate the standard deviation of the distance between the centroids of the disease. Specifically, calculate the Euclidean distance between each centroid of the disease and the standard deviation of the distance between all centroids. The formula for calculating the standard deviation of the distance between the centroids of the disease is as follows:

[0052]

[0053] Among them, D defect d represents the standard deviation of the centroid of the disease. ij Let x represent the Euclidean distance between the centroids of the i-th and j-th diseases. i and y i Let x and y represent the x and y coordinates of the centroid of the i-th disease, respectively. j and y j Let x and y represent the x and y coordinates of the j-th centroid of the disease, respectively. Represents all Euclidean distances d ij The average value;

[0054] (6.4) Calculate the comprehensive scoring index based on the proportion of diseased area, the average area of ​​a single disease, and the standard deviation of the distance between the centroid of the disease. The calculation formula is as follows:

[0055]

[0056] Where h represents the comprehensive scoring index, and D max The reference threshold represents the standard deviation of the disease distance, and w1, w2, and w3 are the weights of different sub-scoring items.

[0057] A second aspect of the present invention provides an external wall defect detection and health assessment device based on a fusion-segmentation joint network, comprising one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described external wall defect detection and health assessment method based on a fusion-segmentation joint network.

[0058] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-described method for detecting and assessing external wall defects based on a fusion-segmentation joint network.

[0059] The beneficial effects of this invention are as follows: This invention adopts a fusion-segmentation joint network framework and introduces dynamic fusion weights. The introduction of dynamic fusion weights, combined with the instance segmentation capability of Mask R-CNN, achieves both efficient image fusion and object detection and segmentation capabilities. Based on the MSRS dataset and transfer learning technology, this invention balances the performance requirements of fusion and segmentation tasks in multi-scene training, significantly reducing computational complexity while ensuring the accuracy of detection results. This invention proposes a facade health calculation index, achieving precise quantification of facade health status and providing a scientific basis for building maintenance. This invention utilizes two different perceptual modalities, overcoming the problem that image recognition technologies based on a single modality often only target certain specific defects. Through the fusion-segmentation joint network architecture and refined training strategies, it satisfies the requirement for fused images to support downstream tasks, realizing automated defect segmentation detection and facade health assessment in the field of building defect detection, improving work efficiency and enhancing the accuracy of defect detection. Attached Figure Description

[0060] Figure 1 This is a flowchart of the method for detecting and assessing the health of exterior walls based on a fusion-segmentation joint network according to the present invention;

[0061] Figure 2 This is a schematic diagram of the fusion-segmentation joint network of the present invention;

[0062] Figure 3 This is a schematic diagram of a structural device for detecting and assessing external wall defects based on a fusion-segmentation joint network according to the present invention. Detailed Implementation

[0063] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit this application.

[0064] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0065] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to determination," or "includes." Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process or method. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0066] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0067] See Figure 1 The method for detecting and assessing the health of exterior walls based on a fusion-segmentation joint network of the present invention specifically includes the following steps:

[0068] (1) Collect infrared thermal imaging images and visible light images of external wall defects and preprocess them to construct a fused image dataset; label the fused image pairs in the fused image dataset and amplify them. After amplification, randomly divide them into training set and test set according to proportion.

[0069] (1.1) Use an infrared thermal imaging camera and a visible light camera to collect infrared thermal images and visible light images of exterior wall defects.

[0070] Specifically, drones equipped with infrared thermal imaging cameras and visible light cameras can be used to collect infrared thermal imaging images and visible light images of exterior wall defects at multiple locations.

[0071] (1.2) Preprocessing of infrared thermal imaging images and visible light images is performed as follows: First, filter and remove erroneous image data such as focusing errors, excessive viewing angle deviations, and repeated shooting in infrared thermal imaging images and visible light images; then, register and crop the infrared thermal imaging images and visible light images of different resolutions to obtain registered infrared thermal imaging images and visible light images.

[0072] (1.3) Construct a fused image dataset based on the registered infrared thermal imaging image and the visible light image.

[0073] (1.4) Using annotation tools, the fused image pairs in the fused image dataset are masked and labeled at the pixel level to represent multiple targets such as walls, wall-mounted ancillary facilities, and defects, serving as semantic segmentation labels. Various data augmentation methods are then used to augment the fused image dataset, which is then randomly divided into training and testing sets according to a set ratio. The fused image pairs are the registered infrared thermal image and the visible light image.

[0074] It should be understood that various data augmentation methods, such as random rotation, random flipping, and translation, are used to process the fused image pairs to increase the number of fused image pairs and thus expand the fused image dataset. After expansion, it can be divided into training and testing sets in a ratio, such as 7:3 or 8:2, for subsequent model training and testing.

[0075] (2) Construct a fusion-detection joint network, which includes a pixel-weighted image fusion network module and a segmentation detection network module, such as... Figure 2 As shown, the image fusion network module is used to fuse the fused image pair, namely the registered infrared thermal image and the visible light image, pixel by pixel to obtain the final fused image; the segmentation detection network module is used to process the final fused image to obtain the category, localization bounding box and pixel-level segmentation mask for each instance.

[0076] In this embodiment, as Figure 2 As shown, the image fusion network module specifically includes: first, the registered infrared thermal imaging image and the visible light image are grayscaled and then stitched together along the channel direction; then, the weights of each pixel in the infrared thermal imaging image and the visible light image are obtained through convolution operations of multiple convolutional layers, which can be used for subsequent pixel-level fusion; then, the weights of each pixel in the infrared thermal imaging image and the visible light image are combined with the source image through element-wise multiplication and addition operations to generate a grayscale fused image; finally, the color mode of the grayscale fused image is converted to RGB color to obtain a color fused image, which is the final fused image.

[0077] It should be noted that the registered infrared thermal image and visible light image input to the image fusion network module can be either grayscale or color images. If it's a grayscale image, the infrared thermal image and visible light image are directly stitched together. If it's a color image, the infrared thermal image and visible light image need to be converted to grayscale and color space converted before stitching. Specifically, the infrared thermal image can be directly converted to grayscale. For the visible light image, it needs to be converted to the YCbCr color space first, and then converted to grayscale. Next, the weights of each pixel in the infrared thermal image and the visible light image are obtained, and then the two modalities are fused. During fusion, the Y channel of the visible light image and the infrared thermal image is used to fuse them, resulting in a grayscale fused image. After obtaining the grayscale fused image, the Y channel of the visible image is replaced with the grayscale fusion result. Finally, the image is converted back from YCbCr to RGB color space to obtain a color fused image, which serves as the input to the subsequent segmentation and detection network module.

[0078] Furthermore, the final fused image is represented as follows:

[0079]

[0080] Among them, I vif W represents the pixel values ​​of the final fused image. irt I represents the weighting coefficient of each pixel in an infrared thermal imaging image. irt W represents the pixel value of an infrared thermal imaging image. vis I represents the weight coefficient of each pixel in a visible light image. vis Represents the pixel values ​​of a visible light image. This indicates element-wise multiplication.

[0081] In this embodiment, as Figure 2 As shown, the segmentation detection network module specifically includes: the segmentation detection network module adopts the Mask RCNN network, which processes targets of different scales through the Feature Pyramid (FPN), and has good segmentation results from large wall objects to small lesions; the segmentation detection network module is used to process the final fused image output by the image fusion network module to achieve instance segmentation, and obtain the category, local bounding box and pixel-level segmentation mask of each instance.

[0082] It should be understood that the Mask R-CNN network is a small, flexible, and general-purpose object instance segmentation framework that can not only detect objects in an image but also provide a high-quality segmentation result for each object. This Mask R-CNN network has multi-task joint training capabilities: ① identifying the category of each instance, such as walls, ancillary facilities, and defects; ② accurately predicting the local bounding box of each instance; ③ generating a pixel-level segmentation mask result for each instance, which is a black and white mask image. The output of the segmentation and detection network module includes the category, local bounding box, and pixel-level segmentation mask for each instance, which can be directly used for building health assessment.

[0083] (3) The fusion-detection joint network is pre-trained using the publicly available MSRS dataset. During the pre-training process, the parameters of the fusion-detection joint network are adjusted with the goal of minimizing the joint loss function in order to obtain a pre-trained fusion-detection joint network.

[0084] It should be understood that the MSRS dataset is a publicly available infrared-visible light fusion dataset, which contains pairs of infrared thermal imaging images and visible light images, as well as the corresponding fused images. The dataset also includes semantic segmentation labels, making it a good dataset for fusion networks with downstream tasks.

[0085] Specifically, infrared thermal imaging images and visible light image pairs from the publicly available MSRS dataset are input into a fusion-detection joint network. The image fusion network module obtains the final fused image, and the segmentation-detection network module obtains the category, localized bounding box, and pixel-level segmentation mask for each instance in the final fused image. A fusion loss function is calculated based on the final fused image output by the image fusion network module and its corresponding source image, where the source image refers to the infrared thermal imaging image and visible light image pair. A semantic segmentation loss function is calculated based on the category, localized bounding box, and pixel-level segmentation mask for each instance output by the segmentation-detection network module and its corresponding semantic segmentation label from the MSRS dataset. A joint loss function is calculated based on the fusion loss function and the semantic segmentation loss function. During pre-training, the parameters of the fusion-detection joint network are adjusted with the optimization objective of minimizing the joint loss function, and the weight W of each pixel in the infrared thermal imaging image in the image fusion network module is dynamically adjusted. irt The weight W of each pixel in the visible light image visThe system calculates the weights α for texture loss and β for semantic segmentation loss in the joint loss function until a preset training epoch is reached or the loss threshold is less than a preset threshold. This process obtains the optimal weights for each pixel in the infrared thermal imaging image and the visible light image, as well as the optimal weights for texture loss and semantic segmentation loss. The resulting pre-trained fusion-detection joint network ensures a balance between image fusion and instance segmentation tasks, and guarantees that the fused image retains significant target information and texture details from the source image.

[0086] It should be understood that the amount of data in the training and test sets constructed in step (1) is insufficient. Therefore, in the current step (3), the publicly available MSRS dataset is used to pre-train the fusion-detection joint network. The main purpose is to obtain the optimal weights for each pixel in the infrared thermal imaging image and the optimal weights for each pixel in the visible light image, so as to ensure the best fusion effect of the image fusion network module. Then, the training set constructed in step (1) is used to further optimize and train the segmentation and detection network module.

[0087] Furthermore, the fusion loss function includes brightness loss and texture loss, and its calculation formula is as follows:

[0088] L vif =L int +αL txt

[0089] Among them, L vif L represents the fusion loss function. int L represents the loss of brightness. txt α represents the texture loss, and α represents the weight of the texture loss, used to balance the global brightness consistency and detail texture representation of the fused image.

[0090] Furthermore, the formula for calculating the brightness loss is:

[0091]

[0092] Where H and W represent the height and width of the infrared thermal image or the visible light image, respectively, ||·||1 represents the L1 norm, and max(I irt ,I vis This means taking the maximum pixel value from the infrared thermal image and the visible light image at each pixel location, thus ensuring that the fused image retains the maximum amount of information.

[0093] Furthermore, the formula for calculating texture loss is:

[0094]

[0095] in, This represents the Sobel gradient operator. and These represent the gradient magnitudes of the fused image, the infrared thermal image, and the visible light image, respectively. Indicates taking and The maximum of these two gradient magnitudes is used to preserve edge information in the source image.

[0096] It should be understood that, due to the lack of a ground truth value, the image fusion network module's fusion loss function is generally defined using operations on the fused image and the source images. The fused image should combine the advantages of visible light and infrared images, possessing rich texture information and comprehensive intensity information. Therefore, the fusion loss function used in the training process comprises two parts: brightness loss L... int and texture loss L txt The weight α is a constant used to balance the global brightness consistency and detail texture representation of the fused image, in order to obtain better visual quality and evaluation metrics of the fused image; the brightness loss L int The difference between the fused image and the source image is measured at the pixel level; while the texture loss L... txt This ensures that the fused image maintains the optimal brightness distribution while preserving the rich texture details in the source image.

[0097] Furthermore, the semantic segmentation loss function includes classification loss, bounding box loss, and mask loss, and its calculation formula is as follows:

[0098] L detect =L seg +L cls +L mask

[0099] Among them, L detect L represents the semantic segmentation loss function. seg L represents the classification loss. cls L represents the bounding box loss. mask This represents the mask loss. Classification loss ensures the accuracy of instance category identification; bounding box loss improves the accuracy of target localization; mask loss optimizes pixel-level segmentation of instances, using a binary sigmoid cross-entropy loss for each pixel.

[0100] Furthermore, the joint loss function includes a fusion loss function and a semantic segmentation loss function, and its calculation formula is as follows:

[0101] L total =L vif +βL detect

[0102] Among them, L total Let represent the joint loss function, and β represent the weights of the semantic segmentation loss function.

[0103] It should be understood that by incorporating the semantic segmentation loss function into the total loss function of the fusion-detection joint network, and using this total loss function as the joint loss function, high-level semantic segmentation and classification information is guided back to the image fusion network module to generate a fused image rich in semantic information. This improves the performance of the fused image in high-level vision tasks, meeting the needs of downstream high-level vision tasks. The total joint loss function includes the fusion loss function and the semantic segmentation loss function. The weight β is used to balance the weight of the segmentation and detection network module's loss term in the total loss function, optimizing the fused image's support for downstream tasks and ensuring a balance in model performance between upstream and downstream networks. For the training strategy of the above joint optimization framework, the image fusion network module and the segmentation and detection network module are trained collaboratively in an end-to-end manner, achieving collaborative optimization between the two modules. This allows the parameter updates of these two components to influence each other during the training phase.

[0104] (4) Use the training set in step (1) to train the pre-trained fusion-detection joint network. During the training process, with the goal of minimizing the joint loss function, freeze the parameters of the image fusion network module and only adjust the parameters of the segmentation and detection network module to obtain the final trained fusion-detection joint network.

[0105] Specifically, the infrared thermal imaging images and visible light image pairs from the training set are input into a pre-trained fusion-detection joint network. The final fused image is obtained through the image fusion network module, and the category, localization bounding box, and pixel-level segmentation mask of each instance in the final fused image are obtained through the segmentation and detection network module. Based on the weights of the optimal texture loss and the optimal semantic segmentation loss function obtained in step (3), the fusion loss function is calculated according to the final fused image output by the image fusion network module and its corresponding source image, where the source image refers to the infrared thermal imaging image and visible light image pair. The semantic segmentation loss function is calculated according to the category, localization bounding box, and pixel-level segmentation mask of each instance output by the segmentation and detection network module and its corresponding semantic segmentation label in the training set. The joint loss function is calculated based on the fusion loss function and the semantic segmentation loss function. During training, with the goal of minimizing the joint loss function, the parameters of the image fusion network module are frozen, that is, the optimal weights of each pixel in the infrared thermal imaging image and the optimal weights of each pixel in the visible light image are frozen in step (3). Only the parameters of the segmentation and detection network module are adjusted until the preset training rounds are reached or the loss threshold is less than the preset threshold, so as to obtain the finally trained fusion-detection joint network.

[0106] It should be noted that after obtaining the initial pre-trained weights and optimal hyperparameters of each loss function term of the pre-trained fusion-detection joint network through step (3), the quality of the image fusion network module has reached a high level through pre-training and meets the requirements of the segmentation and detection network module. At this time, the parameters of the image fusion network module are frozen, and the pre-trained fusion-detection joint network is further trained and optimized using transfer learning combined with the training set constructed in step (1). This optimizes the parameters of the segmentation and detection network module, refines the segmentation and detection network module, and improves the accuracy of the downstream segmentation and detection network module. By further optimizing the segmentation and detection network module, the computational complexity can be effectively reduced, focusing on improving the segmentation accuracy, and finally obtaining the ideal parameters of the fusion-detection joint network.

[0107] (5) Input the infrared thermal imaging images and visible light images to be tested in the test set into the finally trained fusion-detection joint network to perform disease segmentation and detection, and obtain the segmentation and detection results, including the category, localization bounding box and pixel-level segmentation mask of each instance.

[0108] It should be noted that by using the final trained fusion-detection joint network obtained in step (4) to perform defect segmentation and detection on the building exterior wall image, the category, localization bounding box and pixel-level segmentation mask of each instance can be obtained.

[0109] (6) Based on the segmentation and detection results obtained in step (5), the area and distribution density of the disease are statistically analyzed, and the health of the exterior wall is evaluated by a comprehensive scoring index.

[0110] Specifically, the segmentation detection results obtained in step (5) include the category, location bounding box and pixel-level segmentation mask of each instance. Based on the category, location bounding box and pixel-level segmentation mask of all instances, the segmentation results of instances are statistically analyzed by category. The area and location distribution density of each category are calculated based on the pixel-level segmentation mask. The comprehensive scoring index is calculated based on the area and location distribution density of each category. The health of the exterior wall is evaluated by the calculated comprehensive scoring index. The evaluation of the health of the exterior wall in this way is more scientific and accurate.

[0111] Furthermore, the comprehensive scoring index is calculated through the following steps:

[0112] (6.1) Calculate the percentage of damaged area, which is defined as the ratio of the total damaged area to the total wall area. The formula is as follows:

[0113]

[0114] Among them, R defect Indicates the percentage of the area affected by disease; The area of ​​the i-th disease, where N1 represents the total number of diseases. The specific number and area of ​​diseases can be obtained from the segmentation and detection results. Their positions are determined by the coordinates (x, y) of the center point of each bounding box. i ,y i ) Calculations are used to analyze the spatial distribution characteristics of diseases; Let A represent the area of ​​the j-th ancillary facility, and N2 represent the total number of ancillary facilities. Specifically, the number and area of ​​ancillary facilities can be calculated based on the segmentation and detection results. wall This indicates the total area of ​​the wall surface, which can be calculated based on the results of segmented testing. The percentage of damaged area directly reflects the severity of the damage.

[0115] (6.2) Calculate the average area of ​​a single disease, which is defined as the ratio of the total disease area to the number of diseases. The calculation formula is as follows:

[0116]

[0117] Among them, C defect This represents the average area of ​​a single disease, which can reflect the average severity of a single disease. Considering that the larger the area, the more severe the impact of the disease on the wall structure may be, a reference threshold can be set.

[0118] (6.3) Calculate the standard deviation of the distance between the centroids of the disease. Specifically, calculate the Euclidean distance between each centroid of the disease and the standard deviation of the distance between all centroids. The formula for calculating the standard deviation of the distance between the centroids of the disease is as follows:

[0119]

[0120] Among them, D defect d represents the standard deviation of the centroid of the disease. ij Let x represent the Euclidean distance between the centroids of the i-th and j-th diseases. i and y i Let x and y represent the x and y coordinates of the centroid of the i-th disease, respectively. j and y j Let x and y represent the x and y coordinates of the j-th centroid of the disease, respectively. Represents all Euclidean distances d ij The average value. The standard deviation of the centroid distance of the disease can accurately reflect the uniformity of the spatial distribution of the disease. If the disease is concentrated in a certain area, the risk is concentrated, affecting the structural integrity and reducing the score; if the distribution is discrete, the risk is lower and the score is higher.

[0121] (6.4) Based on the scoring logic of disease area proportion, average area of ​​individual diseases, and standard deviation of disease centroid distance: a larger disease area proportion will lower the health score, a higher average area of ​​individual diseases will lower the health score, and a lower standard deviation of disease centroid distance will lower the health score. Combining these three indicators, a comprehensive score index is obtained, and its calculation formula is as follows:

[0122]

[0123] Where h represents the comprehensive scoring index, the larger the value, the healthier the exterior wall; D max The reference threshold represents the standard deviation of the disease distance; w1, w2, and w3 are the weights of different sub-score items, and their sum is 1. The weights can be adjusted according to the actual situation to focus on different evaluation items.

[0124] Corresponding to the aforementioned embodiments of the external wall disease detection and health assessment method based on fusion-segmentation joint network, the present invention also provides embodiments of an external wall disease detection and health assessment device based on fusion-segmentation joint network.

[0125] See Figure 3 The present invention provides an external wall defect detection and health assessment device based on a fusion-segmentation joint network, comprising one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the external wall defect detection and health assessment method based on a fusion-segmentation joint network in the above embodiment.

[0126] The embodiment of the external wall defect detection and health assessment device based on the fusion-segmentation joint network of the present invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware structure diagram of any device with data processing capabilities, including the external wall defect detection and health assessment device based on a fusion-segmentation joint network according to the present invention. (Except for...) Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0127] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0128] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0129] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the method for detecting and assessing external wall defects based on a fusion-segmentation joint network as described in the above embodiments.

[0130] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0131] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting and assessing the health of exterior walls based on a fusion-segmentation joint network, characterized in that, Includes the following steps: (1) Collect infrared thermal images and visible light images of exterior wall defects, and preprocess them to construct a fused image dataset; label the fused image pairs in the fused image dataset and amplify them; after amplification, randomly divide them into training set and test set according to proportion; step (1) specifically includes the following sub-steps: (1.1) Use an infrared thermal imaging camera and a visible light camera to collect infrared thermal images and visible light images of external wall defects; (1.2) Preprocessing of infrared thermal imaging images and visible light images is as follows: First, filter out erroneous image data such as focusing errors, excessive viewing angle deviations, and repeated shooting in infrared thermal imaging images and visible light images; then register and crop the infrared thermal imaging images and visible light images of different resolutions to obtain registered infrared thermal imaging images and visible light images. (1.3) Construct a fused image dataset based on the registered infrared thermal imaging image and the visible light image; (1.4) Use annotation tools to perform multi-class target mask segmentation annotation on the fused image pairs in the fused image dataset at the pixel level, and use them as semantic segmentation labels; and use a variety of data augmentation methods to augment the fused image dataset. After augmentation, the dataset is randomly divided into training set and test set according to the proportion. (2) Construct a fusion-detection joint network, which includes a pixel-weighted image fusion network module and a segmentation detection network module. The image fusion network module is used to fuse the fused image pair, i.e., the registered infrared thermal image and the visible light image, pixel by pixel to obtain the final fused image. The segmentation detection network module is used to process the final fused image to obtain the category, localization bounding box and pixel-level segmentation mask of each instance. The image fusion network module specifically includes: First, the registered infrared thermal image and the visible light image are converted to grayscale and then stitched together along the channel direction. Next, convolutional operations through multiple convolutional layers are used to obtain the weights of each pixel in the infrared thermal image and the visible light image. Then, these weights are combined with the source image using element-wise multiplication and addition operations to generate a grayscale fused image. Finally, the color mode of the grayscale fused image is converted to RGB to obtain a color fused image, which serves as the final fused image, represented as: in, This represents the pixel values ​​of the final merged image. This represents the weighting coefficient for each pixel in an infrared thermal imaging image. Represents the pixel values ​​of an infrared thermal imaging image. This represents the weight coefficient of each pixel in a visible light image. Represents the pixel values ​​of a visible light image. This indicates element-wise multiplication; The segmentation detection network module specifically includes: The segmentation detection network module uses a Mask R-CNN network, which processes targets of different scales through a feature pyramid. The segmentation detection network module is used to process the final fused image output by the image fusion network module to achieve instance segmentation and obtain the category, local bounding box and pixel-level segmentation mask of each instance. (3) The fusion-detection joint network is pre-trained using the publicly available MSRS dataset. During the pre-training process, the parameters of the fusion-detection joint network are adjusted with the goal of minimizing the joint loss function to obtain a pre-trained fusion-detection joint network. Step (3) specifically includes: Infrared thermal imaging images and visible light images from the publicly available MSRS dataset are input into a fusion-detection joint network. The final fused image is obtained through the image fusion network module, and the category, local bounding box, and pixel-level segmentation mask of each instance in the final fused image are obtained through the segmentation and detection network module. The fusion loss function is calculated based on the final fused image output by the image fusion network module and its corresponding source image, where the source image refers to the pair of infrared thermal imaging image and visible light image; the semantic segmentation loss function is calculated based on the category, local bounding box, and pixel-level segmentation mask of each instance output by the segmentation and detection network module and its corresponding semantic segmentation label in the MSRS dataset; and the joint loss function is calculated based on the fusion loss function and the semantic segmentation loss function. During pre-training, the parameters of the fusion-detection joint network are adjusted with the goal of minimizing the joint loss function. The weights of each pixel in the infrared thermal imaging image and the visible light image in the image fusion network module, as well as the weights of the texture loss and semantic segmentation loss function in the joint loss function, are dynamically adjusted until a preset training epoch is reached or the preset loss threshold is reached. The optimal weights of each pixel in the infrared thermal imaging image and the visible light image, as well as the optimal weights of the texture loss and the optimal weights of the semantic segmentation loss function, are obtained, resulting in the pre-trained fusion-detection joint network. (4) Train the pre-trained fusion-detection joint network using the training set from step (1). During training, minimize the joint loss function as the optimization objective, freeze the parameters of the image fusion network module, and only adjust the parameters of the segmentation and detection network module to obtain the final trained fusion-detection joint network; step (4) specifically includes: The infrared thermal imaging images and visible light images in the training set are input into a pre-trained fusion-detection joint network. The final fused image is obtained through the image fusion network module, and the category, local bounding box and pixel-level segmentation mask of each instance in the final fused image are obtained through the segmentation and detection network module. Based on the optimal texture loss weights and optimal semantic segmentation loss function weights obtained in step (3), the fusion loss function is calculated according to the final fused image output by the image fusion network module and its corresponding source image, where the source image refers to the infrared thermal imaging image and the visible light image pair; the semantic segmentation loss function is calculated according to the category, localization bounding box and pixel-level segmentation mask of each instance output by the segmentation detection network module and its corresponding semantic segmentation label in the training set; the joint loss function is calculated according to the fusion loss function and the semantic segmentation loss function. During training, with the goal of minimizing the joint loss function, the parameters of the image fusion network module are frozen, and only the parameters of the segmentation and detection network module are adjusted until the preset training rounds are reached or the loss threshold is less than the preset threshold, thus obtaining the final trained fusion-detection joint network. (5) Input the infrared thermal imaging images and visible light images to be tested in the test set into the finally trained fusion-detection joint network to perform disease segmentation and detection, and obtain the segmentation and detection results, including the category, localization bounding box and pixel-level segmentation mask of each instance; (6) Based on the segmentation and detection results obtained in step (5), the area and distribution density of the disease are statistically analyzed, and the health of the exterior wall is evaluated by a comprehensive scoring index.

2. The method for detecting and assessing the health of exterior walls based on a fusion-segmentation joint network according to claim 1, characterized in that, The joint loss function includes a fusion loss function and a semantic segmentation loss function, and its calculation formula is as follows: in, Denotes the joint loss function. Represents the fusion loss function. This represents the semantic segmentation loss function. The weights represent the semantic segmentation loss function. The fusion loss function includes brightness loss and texture loss, and its calculation formula is as follows: in, Indicates brightness loss. Indicates texture loss, The weights representing the texture loss; The formula for calculating the brightness loss is: Where H and W represent the height and width of the infrared thermal image or the visible light image, respectively. Describing the L1 norm, This represents the maximum pixel value obtained from both the infrared thermal image and the visible light image at each pixel location; The formula for calculating the texture loss is: in, This represents the Sobel gradient operator. , and These represent the gradient magnitudes of the fused image, the infrared thermal image, and the visible light image, respectively. Indicates taking and The maximum of these two gradient magnitudes; The semantic segmentation loss function includes classification loss, bounding box loss, and mask loss, and its calculation formula is as follows: in, Represents classification loss. Indicates the bounding box loss. This indicates the loss of the mask.

3. The method for detecting and assessing the health of exterior walls based on a fusion-segmentation joint network according to claim 1, characterized in that, Step (6) specifically includes: The segmentation detection results obtained in step (5) include the category, location bounding box and pixel-level segmentation mask of each instance. Based on the category, location bounding box and pixel-level segmentation mask of all instances, the segmentation results of instances are statistically analyzed by category. The area and location distribution density of each category are calculated according to the pixel-level segmentation mask. The comprehensive scoring index is calculated according to the area and location distribution density of each category. The health of the exterior wall is evaluated by the calculated comprehensive scoring index.

4. The method for detecting and assessing the health of exterior walls based on a fusion-segmentation joint network according to claim 1 or 3, characterized in that, The comprehensive scoring index is calculated through the following steps: (6.1) Calculate the percentage of damaged area, which is defined as the ratio of the total damaged area to the total wall area. The calculation formula is as follows: in, Indicates the percentage of the area affected by disease; The area of ​​the i-th disease. Indicates the total number of diseases; This represents the area of ​​the j-th ancillary facility. Indicates the total number of ancillary facilities; Indicates the total area of ​​the wall surface; (6.2) Calculate the average area of ​​a single disease, which is defined as the ratio of the total area of ​​diseases to the number of diseases. The calculation formula is as follows: in, This represents the average area of ​​a single disease. (6.3) Calculate the standard deviation of the distance between the centroids of the disease. Specifically, calculate the Euclidean distance between each centroid of the disease and the standard deviation of the distance between all centroids. The formula for calculating the standard deviation of the distance between the centroids of the disease is as follows: in, This indicates the distance from the centroid of the disease to the standard deviation. Let represent the Euclidean distance between the centroids of the i-th disease and the j-th disease. and Let x and y represent the x and y coordinates of the centroid of the i-th disease, respectively. and Let x and y represent the x and y coordinates of the j-th centroid of the disease, respectively. Represents all Euclidean distances The average value; (6.4) Calculate the comprehensive scoring index based on the proportion of diseased area, the average area of ​​a single disease, and the standard deviation of the distance between the centroid of the disease. The calculation formula is as follows: Where h represents the comprehensive scoring index, A reference threshold representing the standard deviation of the distance from the disease. , and The weights of different sub-scoring items.

5. A device for detecting and assessing the health of exterior walls based on a fusion-segmentation joint network, comprising one or more processors and a memory, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the method for detecting and assessing external wall defects based on a fusion-segmentation joint network as described in any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the method for detecting and assessing the health of exterior walls based on a fusion-segmentation joint network as described in any one of claims 1-4.

Images