Exterior wall disease detection and health assessment method based on fusion-segmentation joint network

Abstract

The invention discloses an outer wall disease detection and health assessment method based on a fusion-segmentation joint network, and the method comprises the steps: collecting an infrared thermal image and a visible light image of an outer wall disease, and constructing an infrared image fusion data set after preprocessing; constructing a fusion-detection joint network comprising an image fusion network module and a segmentation detection network module, and realizing collaborative optimization through a joint loss function; pre-training is carried out by using an MSRS data set so as to obtain the best parameters of the dynamic pixel fusion weight and the loss function weight item; freezing parameters of the image fusion network module, and optimizing the segmentation detection network module in combination with a self-established fusion data set; using the trained fusion-detection joint network to carry out disease segmentation detection on the building exterior wall image; counting the disease area and distribution condition according to the segmentation detection result, and comprehensively evaluating the health degree of the outer wall. According to the invention, automatic disease segmentation detection and external facade health assessment are realized, and the disease detection accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of infrared image fusion and building exterior wall defect recognition, and in particular to an exterior wall disease detection and health assessment method based on a fusion-segmentation joint network. Background Art

[0002] Common building facade defects such as cracks, hollowing, water seepage, and spalling not only impact the safety and stability of buildings but also threaten public safety. Comprehensive automated detection and health assessment of facade surface and subsurface defects is crucial.

[0003] Infrared thermal imaging and visible light imaging have become mainstream technologies for defect detection, but each has significant limitations. Recognition methods based on visible light images can currently only identify specific surface defects, such as cracks and detachments. Infrared thermal imaging can reveal hidden defects such as hollowing and water seepage, but it suffers from low contrast and susceptibility to environmental interference. Many image fusion technologies currently attempt to combine the advantages of infrared and visible light to compensate for the shortcomings of a single modality. However, traditional fusion methods are often optimized for visual effects rather than designed for downstream tasks, resulting in poor fusion results for defect detection. A key challenge is how to efficiently complement the two modal information in the fusion process based on the complex scenes of building facades to achieve efficient and accurate defect detection. Summary of the Invention

[0004] The present invention aims to address the shortcomings of existing technologies by providing a method for exterior wall defect detection and health assessment based on a combined fusion-segmentation network. This method utilizes a dynamically weighted fusion network and an efficient segmentation network for collaborative optimization during the detection process. This method addresses the limitations of single-modality imaging and the adaptability of fusion, improves automatic detection accuracy, and can scientifically and quantitatively assess the health of building facades based on the detection results.

[0005] The object of the present invention is achieved through the following technical solutions: In a first aspect, an embodiment of the present invention provides a method for detecting and assessing exterior wall defects based on a fusion-segmentation joint network, comprising the following steps:

[0006] (1) Collect infrared thermal imaging images and visible light images of exterior wall defects and preprocess them to construct a fused image dataset; annotate the fused image pairs in the fused image dataset and amplify them. After amplification, the dataset is randomly divided into a training set and a test set according to the proportion;

[0007] (2) constructing a fusion-detection joint network, which includes a pixel-weighted image fusion network module and a segmentation detection network module, wherein the image fusion network module is used to fuse the fused image pair, i.e., the registered infrared thermal imaging image and the visible light image, pixel by pixel to obtain a final fused image; and the segmentation detection network module is used to process the final fused image to obtain the category, positioning bounding box and pixel-level segmentation mask of each instance;

[0008] (3) The fusion-detection joint network is pre-trained using the public MSRS dataset. During the pre-training process, the parameters of the fusion-detection joint network are adjusted with the goal of minimizing the joint loss function to obtain a pre-trained fusion-detection joint network.

[0009] (4) Using the training set in step (1) to train the pre-trained fusion-detection joint network, during the training process, the parameters of the image fusion network module are frozen with the optimization goal of minimizing the joint loss function, and only the parameters of the segmentation detection network module are adjusted to obtain the final trained fusion-detection joint network;

[0010] (5) Input the infrared thermal imaging image and visible light image pairs to be tested in the test set into the finally trained fusion-detection joint network to perform disease segmentation detection and obtain the segmentation detection results, which include the category, positioning bounding box and pixel-level segmentation mask of each instance;

[0011] (6) According to the segmentation detection results obtained in step (5), the disease area and distribution density are counted, and the health of the exterior wall is evaluated using a comprehensive scoring index.

[0012] Furthermore, the step (1) specifically includes the following sub-steps:

[0013] (1.1) Use infrared thermal imaging cameras and visible light cameras to collect infrared thermal imaging images and visible light images of exterior wall defects;

[0014] (1.2) Preprocessing the infrared thermal imaging images and visible light images: First, filter and remove erroneous image data caused by focusing errors, excessive viewing angle deviations, and repeated shots; then, register and resize the infrared thermal imaging images and visible light images of different resolutions to obtain registered infrared thermal imaging images and visible light images;

[0015] (1.3) Constructing a fused image dataset based on the registered infrared thermal imaging image and visible light image;

[0016] (1.4) Use annotation tools to perform mask segmentation annotation of multiple categories of targets on the fused image pairs in the fused image dataset at the pixel level as semantic segmentation labels; and use multiple data augmentation methods to amplify the fused image dataset. After amplification, the dataset is randomly divided into training and test sets in proportion.

[0017] Furthermore, the image fusion network module specifically includes:

[0018] First, the registered infrared thermal imaging image and visible light image are grayscaled and then spliced ​​along the channel direction; then, the weight of each pixel of the infrared thermal imaging image and the weight of each pixel of the visible light image are obtained through the convolution operation of multiple convolutional layers; then, the weight of each pixel of the infrared thermal imaging image and the weight of each pixel of the visible light image are combined with the source image through element-level multiplication and addition operations to generate a grayscale fused image; finally, the color mode of the grayscale fused image is converted to RGB color to obtain a color fused image as the final fused image, which is expressed as:

[0019]

[0020] Among them, I vif Represents the pixel value of the final fused image, W irt Represents the weight coefficient of each pixel of the infrared thermal imaging image, I irt Represents the pixel value of the infrared thermal imaging image, W vis Represents the weight coefficient of each pixel of the visible light image, I vis Represents the pixel value of the visible light image, Represents element-wise multiplication.

[0021] Furthermore, the segmentation detection network module specifically includes:

[0022] The segmentation detection network module adopts the Mask RCNN network, which processes objects of different scales through a feature pyramid. The segmentation detection network module is used to process the final fused image output by the image fusion network module to achieve instance segmentation and obtain the category, positioning bounding box and pixel-level segmentation mask of each instance.

[0023] Furthermore, the step (3) specifically includes:

[0024] The infrared thermal imaging image and visible light image pairs from the public MSRS dataset are input into the joint fusion-detection network. The image fusion network module is used to obtain the final fused image. The segmentation detection network module is used to obtain the category, localization bounding box, and pixel-level segmentation mask of each instance in the final fused image.

[0025] The fusion loss function is calculated based on the final fused image output by the image fusion network module and its corresponding source image, where the source image refers to the infrared thermal imaging image and the visible light image pair; the semantic segmentation loss function is calculated based on the category, positioning bounding box and pixel-level segmentation mask of each instance output by the segmentation detection network module and the corresponding semantic segmentation label in the MSRS dataset; the joint loss function is calculated based on the fusion loss function and the semantic segmentation loss function;

[0026] During the pre-training process, the parameters of the fusion-detection joint network are adjusted with the minimization of the joint loss function as the optimization goal, and the weight of each pixel of the infrared thermal imaging image and the weight of each pixel of the visible light image in the image fusion network module, as well as the weight of the texture loss and the weight of the semantic segmentation loss function in the joint loss function are dynamically adjusted until the preset training rounds are reached or the preset loss threshold is less than, and the optimal weight of each pixel of the infrared thermal imaging image and the optimal weight of each pixel of the visible light image, as well as the optimal weight of the texture loss and the optimal weight of the semantic segmentation loss function are obtained to obtain the pre-trained fusion-detection joint network.

[0027] Furthermore, the joint loss function includes a fusion loss function and a semantic segmentation loss function, and its calculation formula is:

[0028] L total =L vif +βL detect

[0029] Among them, L total represents the joint loss function, L vif represents the fusion loss function, L detect represents the semantic segmentation loss function, and β represents the weight of the semantic segmentation loss function;

[0030] The fusion loss function includes brightness loss and texture loss, and its calculation formula is:

[0031] L vif =L int +αL txt

[0032] Among them, L int Indicates brightness loss, L txt represents texture loss, α represents the weight of texture loss;

[0033] The calculation formula for the brightness loss is:

[0034]

[0035] Where H and W represent the height and width of the infrared thermal imaging image or visible light image, respectively, ||·||1 represents the L1 norm, and max(Iirt ,I vis ) represents taking the maximum value of the pixel values ​​of the infrared thermal imaging image and the visible light image at each pixel position;

[0036] The calculation formula of the texture loss is:

[0037]

[0038] in, represents the Sobel gradient operator, and Represent the gradient amplitudes of the fused image, infrared thermal imaging image, and visible light image, respectively. Indicates taking and The maximum of the two gradient magnitudes.

[0039] The semantic segmentation loss function includes classification loss, bounding box loss and mask loss, and its calculation formula is:

[0040] L detect =L seg +L cls +L mask

[0041] Among them, L seg represents the classification loss, L cls represents the bounding box loss, L mask Represents mask loss.

[0042] Furthermore, the step (6) specifically includes:

[0043] The segmentation detection result obtained in step (5) includes the category, positioning bounding box and pixel-level segmentation mask of each instance. Based on the categories, positioning bounding boxes and pixel-level segmentation masks of all instances, the instance segmentation results are statistically analyzed by category. The area and position distribution density of each category are calculated based on the pixel-level segmentation mask. A comprehensive scoring index is calculated based on the area and position distribution density of each category. The health of the exterior wall is evaluated using the calculated comprehensive scoring index.

[0044] Furthermore, the comprehensive scoring index is calculated and obtained through the following steps:

[0045] (6.1) Calculate the proportion of damaged area, which is defined as the ratio of the total damaged area to the total wall area. The calculation formula is:

[0046]

[0047] Among them, R defect Indicates the proportion of diseased area; The area of ​​the ith disease, N1 represents the total number of diseases; represents the area of ​​the jth ancillary facility, N2 represents the total number of ancillary facilities; A wall Indicates the total area of ​​the wall;

[0048] (6.2) Calculate the average area of ​​a single disease, which is defined as the ratio of the total disease area to the number of diseases. The calculation formula is:

[0049]

[0050] Among them, C defect It represents the average area of ​​a single disease;

[0051] (6.3) Calculate the standard deviation of the disease centroid distance. Specifically, calculate the Euclidean distance between each disease centroid and the standard deviation of all centroid distances to obtain the standard deviation of the disease centroid distance. The calculation formula is:

[0052]

[0053] Among them, D defect represents the standard deviation of the distance from the disease centroid, d ij represents the Euclidean distance between the centroid of the i-th disease and the j-th disease, x i and y i Represents the horizontal and vertical coordinates of the centroid of the ith disease, x j and y j are the horizontal and vertical coordinates of the jth disease centroid, represents all Euclidean distances d ij The average value of

[0054] (6.4) The comprehensive scoring index is calculated based on the disease area ratio, the average area of ​​a single disease, and the standard deviation of the disease centroid distance. The calculation formula is:

[0055]

[0056] Among them, h represents the comprehensive scoring index, D max It represents the reference threshold of the disease distance standard deviation, and w1, w2 and w3 are the weights of different sub-scoring items.

[0057] A second aspect of an embodiment of the present invention provides an exterior wall disease detection and health assessment device based on a fusion-segmentation joint network, comprising one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned exterior wall disease detection and health assessment method based on a fusion-segmentation joint network.

[0058] A third aspect of an embodiment of the present invention provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the program is used to implement the above-mentioned exterior wall disease detection and health assessment method based on the fusion-segmentation joint network.

[0059] The beneficial effects of the present invention are as follows: the present invention adopts a fusion-segmentation joint network framework and introduces dynamic fusion weights. The introduction of dynamic fusion weights is combined with the instance segmentation capability of Mask R-CNN, which has both efficient image fusion and target detection and segmentation capabilities; the present invention is based on the MSRS data set and transfer learning technology, and balances the performance requirements of fusion and segmentation tasks in multi-scene training, significantly reducing the computational complexity while ensuring the accuracy of the detection results; the present invention proposes a facade health calculation index to achieve accurate quantification of the facade health status, providing a scientific basis for building maintenance; the present invention utilizes two different perceptual modal information, overcomes the problem that image recognition technology based on a single modality often only targets certain specific defects, and through the fusion-segmentation joint network architecture and refined training strategies, meets the support of fused images for downstream tasks, and realizes automated disease segmentation detection and facade health assessment in the field of building disease detection, thereby improving work efficiency and the accuracy of disease detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a flow chart of the exterior wall disease detection and health assessment method based on fusion-segmentation joint network of the present invention;

[0061] Figure 2 It is a schematic diagram of the structure of the fusion-segmentation joint network of the present invention;

[0062] Figure 3 It is a structural diagram of the exterior wall disease detection and health assessment device based on the fusion-segmentation joint network of the present invention. DETAILED DESCRIPTION

[0063] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims. It should be understood that the foregoing general description and the detailed description that follows are exemplary and illustrative only and do not limit the present application.

[0064] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0065] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of..." or "when..." or "in response to determination." Moreover, the term "comprises," "comprising," or any other variant thereof is intended to cover non-exclusive inclusion, so that the process or method comprising a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process or method. In the absence of further restrictions, the elements defined by the statement "comprising a..." do not exclude the presence of other identical elements in the process, method, article, or device comprising the elements.

[0066] The present invention will be described in detail below with reference to the accompanying drawings. Unless there is any conflict, the features of the following embodiments and implementations may be combined with each other.

[0067] See also Figure 1 The exterior wall disease detection and health assessment method based on the fusion-segmentation joint network of the present invention specifically includes the following steps:

[0068] (1) Infrared thermal imaging images and visible light images of exterior wall defects are collected and preprocessed to construct a fused image dataset; the fused image pairs in the fused image dataset are annotated and amplified, and after the amplification is completed, they are randomly divided into training and test sets according to proportion.

[0069] (1.1) Use infrared thermal imaging cameras and visible light cameras to collect infrared thermal imaging images and visible light images of exterior wall defects.

[0070] Specifically, drones equipped with infrared thermal imaging cameras and visible light cameras can be used to collect infrared thermal imaging images and visible light images of exterior wall defects at multiple locations.

[0071] (1.2) Preprocessing the infrared thermal imaging images and visible light images. Specifically, first, filter out and remove erroneous image data such as focusing errors, excessive viewing angle deviations, and repeated shots in the infrared thermal imaging images and visible light images; then, align and resize the infrared thermal imaging images and visible light images of different resolutions to obtain the aligned infrared thermal imaging images and visible light images.

[0072] (1.3) Construct a fused image dataset based on the registered infrared thermal imaging image and visible light image.

[0073] (1.4) Use annotation tools to perform pixel-level mask segmentation annotations on the fused image pairs in the fused image dataset for multiple categories of objects, such as walls, wall accessories, and defects, as semantic segmentation labels. Use various data augmentation methods to augment the fused image dataset, and after augmentation, randomly divide it into training and test sets in proportion. The fused image pairs are the registered infrared thermal imaging images and visible light images.

[0074] It should be understood that various data augmentation methods, such as random rotation, random flipping, and translation, are used to process the fused image pairs to increase the number of fused image pairs and thus expand the fused image dataset. After the augmentation is completed, the dataset can be divided into training and test sets in a ratio, such as 7:3 or 8:2, for subsequent model training and testing.

[0075] (2) Construct a fusion-detection joint network, which includes an image fusion network module based on pixel weighting and a segmentation detection network module, such as Figure 2 As shown in the figure, the image fusion network module is used to fuse the fused image pair, that is, the registered infrared thermal imaging image and the visible light image, pixel by pixel to obtain the final fused image; the segmentation detection network module is used to process the final fused image to obtain the category, positioning bounding box and pixel-level segmentation mask of each instance.

[0076] In this embodiment, Figure 2 As shown in the figure, the image fusion network module specifically includes: first, the registered infrared thermal imaging image and the visible light image are grayscaled and then spliced ​​along the channel direction; then, the weight of each pixel of the infrared thermal imaging image and the weight of each pixel of the visible light image are obtained through the convolution operation of multiple convolutional layers, which can be used for subsequent pixel-level fusion; then, the weight of each pixel of the infrared thermal imaging image and the weight of each pixel of the visible light image are combined with the source image through element-level multiplication and addition operations to generate a grayscale fusion image; finally, the color mode of the grayscale fusion image is converted to RGB color to obtain a color fusion image as the final fusion image.

[0077] It should be noted that the registered infrared thermal imaging image and visible light image input to the image fusion network module can be grayscale or color images. If they are grayscale images, the infrared thermal imaging image and visible light image are directly spliced ​​together. If they are color images, they need to undergo color space conversion and grayscale conversion before splicing. Infrared thermal imaging images can be directly converted to grayscale. For visible light images, they need to be converted to YCbCr color space first, then to grayscale. Next, the weight of each pixel in the infrared thermal imaging image and each pixel in the visible light image are obtained, and the two modal images are fused. During the fusion process, the visible light image and the infrared thermal imaging image are fused using the Y channel to obtain a grayscale fused image. After obtaining the grayscale fused image, the Y channel of the visible image is replaced with the grayscale fused result. Finally, the image is converted back from YCbCr to RGB color space to obtain a color fused image, which serves as the input to the subsequent segmentation and detection network module.

[0078] Furthermore, the final fused image is expressed as:

[0079]

[0080] Among them, I vif Represents the pixel value of the final fused image, W irt Represents the weight coefficient of each pixel of the infrared thermal imaging image, I irt Represents the pixel value of the infrared thermal imaging image, W vis Represents the weight coefficient of each pixel of the visible light image, I vis Represents the pixel value of the visible light image, Represents element-wise multiplication.

[0081] In this embodiment, Figure 2 As shown in the figure, the segmentation detection network module specifically includes: the segmentation detection network module adopts the Mask RCNN network, and the Mask RCNN network processes targets of different scales through the feature pyramid (FPN), and has a good segmentation effect from large-scale wall objects to small-scale diseases; the segmentation detection network module is used to process the final fused image output by the image fusion network module to achieve instance segmentation and obtain the category, positioning bounding box and pixel-level segmentation mask of each instance.

[0082] It should be understood that the Mask RCNN network is an existing, compact, and flexible general-purpose object instance segmentation framework that not only detects objects in an image but also generates high-quality segmentation results for each object. The Mask RCNN network is capable of joint multi-task training: ① identifying the category of each instance, such as wall, ancillary facilities, or damage; ② accurately predicting the location bounding box of each instance; and ③ generating a pixel-level segmentation mask for each instance, which is a black and white mask. The output of the segmentation detection network module includes the category, location bounding box, and pixel-level segmentation mask for each instance, which can be directly used for building health assessment.

[0083] (3) The fusion-detection joint network is pre-trained using the public MSRS dataset. During the pre-training process, the parameters of the fusion-detection joint network are adjusted with the optimization goal of minimizing the joint loss function to obtain the pre-trained fusion-detection joint network.

[0084] It should be understood that the MSRS dataset is a public infrared-visible light fusion dataset, which contains infrared thermal imaging images, visible light image pairs, and corresponding fused images. The dataset also has semantic segmentation labels, which can provide a good dataset for fusion networks with downstream tasks.

[0085] Specifically, the infrared thermal imaging image and visible light image pairs in the public MSRS dataset are input into the fusion-detection joint network, and the final fused image is obtained through the image fusion network module. The category, positioning bounding box and pixel-level segmentation mask of each instance in the final fused image are obtained through the segmentation detection network module. The fusion loss function is calculated based on the final fused image output by the image fusion network module and the corresponding source image, where the source image refers to the infrared thermal imaging image and visible light image pair; the semantic segmentation loss function is calculated based on the category, positioning bounding box and pixel-level segmentation mask of each instance output by the segmentation detection network module and the corresponding semantic segmentation label in the MSRS dataset; the joint loss function is calculated based on the fusion loss function and the semantic segmentation loss function. During the pre-training process, the parameters of the fusion-detection joint network are adjusted with the minimization of the joint loss function as the optimization goal, and the weight W of each pixel of the infrared thermal imaging image in the image fusion network module is dynamically adjusted. irt and the weight W of each pixel of the visible light image visAs well as the weight α of the texture loss and the weight β of the semantic segmentation loss function in the joint loss function, until the preset training rounds are reached or the loss threshold is less than the preset value, the optimal weight of each pixel of the infrared thermal imaging image and the optimal weight of each pixel of the visible light image, as well as the optimal weight of the texture loss and the optimal weight of the semantic segmentation loss function are obtained, and a pre-trained fusion-detection joint network is obtained, which can ensure that the pre-trained fusion-detection joint network achieves a balance between image fusion and instance segmentation tasks, and ensures that the fused image retains the salient target information and texture details of the source image.

[0086] It should be understood that the amount of data in the training set and test set constructed in step (1) is not large enough. Therefore, in the current step (3), the public MSRS dataset is first used to pre-train the fusion-detection joint network. The main purpose is to obtain the optimal weight for each pixel of the infrared thermal imaging image and the optimal weight for each pixel of the visible light image to ensure the best fusion effect of the image fusion network module. After that, the training set constructed in step (1) is used to further optimize the segmentation detection network module.

[0087] Furthermore, the fusion loss function includes brightness loss and texture loss, and its calculation formula is:

[0088] L vif =L int +αL txt

[0089] Among them, L vif represents the fusion loss function, L int Indicates brightness loss, L txt Represents texture loss, α represents the weight of texture loss, which is used to balance the global brightness consistency and detail texture performance of the fused image.

[0090] Furthermore, the calculation formula for brightness loss is:

[0091]

[0092] Where H and W represent the height and width of the infrared thermal imaging image or visible light image, respectively, ||·||1 represents the L1 norm, and max(I irt ,I vis ) means taking the maximum pixel value of the infrared thermal imaging image and the visible light image at each pixel position, thereby ensuring that the fused image retains the maximum amount of information.

[0093] Furthermore, the calculation formula of texture loss is:

[0094]

[0095] in, represents the Sobel gradient operator, and Represent the gradient amplitudes of the fused image, infrared thermal imaging image, and visible light image, respectively. Indicates taking and The maximum of these two gradient magnitudes is used to preserve edge information in the source image.

[0096] It should be understood that due to the lack of true value, the image fusion network module’s fusion loss function is generally defined by using the fusion image and the source image operation. The fusion image should combine the advantages of visible light and infrared images, and have rich texture information and comprehensive intensity information. Therefore, the fusion loss function used in the training process contains two parts: brightness loss L int and texture loss L txt , the weight α is a constant used to balance the global brightness consistency and detail texture performance of the fused image to obtain better fused image visual quality and evaluation indicators; the brightness loss L int Measuring the difference between the fused image and the source image at the pixel level; while the texture loss L txt It ensures that the fused image can maintain the best brightness distribution while retaining the rich texture details in the source image.

[0097] Furthermore, the semantic segmentation loss function includes classification loss, bounding box loss and mask loss, and its calculation formula is:

[0098] L detect =L seg +L cls +L mask

[0099] Among them, L detect represents the semantic segmentation loss function, L seg represents the classification loss, L cls represents the bounding box loss, L mask Represents mask loss. Classification loss ensures the accuracy of instance category recognition; bounding box loss improves the accuracy of target positioning; mask loss optimizes pixel-level segmentation of instances, using a binary sigmoid cross entropy loss for each pixel.

[0100] Furthermore, the joint loss function includes the fusion loss function and the semantic segmentation loss function, and its calculation formula is:

[0101] L total =L vif +βL detect

[0102] Among them, L total represents the joint loss function, and β represents the weight of the semantic segmentation loss function.

[0103] It should be understood that by introducing the semantic segmentation loss function into the total loss function of the fusion-detection joint network, the total loss function is used as a joint loss function to guide the high-level semantic segmentation and classification information to flow back to the image fusion network module to generate a fused image rich in semantic information, thereby improving the performance of the fused image in high-level visual tasks to meet the needs of downstream high-level visual tasks. Among them, the total joint loss function includes the fusion loss function and the semantic segmentation loss function, and the weight β is used to balance the weight of the loss term of the segmentation detection network module in the total loss function, optimize the support of the fused image for downstream tasks and ensure the balance of performance of the model between the upstream and downstream networks. For the training strategy of the above-mentioned joint optimization framework, the image fusion network module and the segmentation detection network module are trained collaboratively in an end-to-end manner to achieve collaborative optimization of the image fusion network module and the segmentation detection network module, so that the parameter updates of the two components can affect each other during the training phase.

[0104] (4) Using the training set in step (1), the pre-trained fusion-detection joint network is trained. During the training process, the optimization goal is to minimize the joint loss function, freeze the parameters of the image fusion network module, and only adjust the parameters of the segmentation detection network module to obtain the final trained fusion-detection joint network.

[0105] Specifically, the infrared thermal imaging image and visible light image pairs in the training set are input into the pre-trained fusion-detection joint network, the final fusion image is obtained through the image fusion network module, and the category, positioning bounding box and pixel-level segmentation mask of each instance in the final fusion image are obtained through the segmentation detection network module. Based on the optimal texture loss weight and the optimal semantic segmentation loss function weight obtained in step (3), the fusion loss function is calculated according to the final fusion image output by the image fusion network module and the corresponding source image, where the source image refers to the infrared thermal imaging image and visible light image pair; the semantic segmentation loss function is calculated according to the category, positioning bounding box and pixel-level segmentation mask of each instance output by the segmentation detection network module and the corresponding semantic segmentation label in the training set; and the joint loss function is calculated according to the fusion loss function and the semantic segmentation loss function. During the training process, the optimization goal is to minimize the joint loss function, freeze the parameters of the image fusion network module, that is, freeze the optimal weight of each pixel of the infrared thermal imaging image and the optimal weight of each pixel of the visible light image in step (3), and only adjust the parameters of the segmentation and detection network module until the preset training rounds are reached or the loss threshold is less than the preset value, and the final trained fusion-detection joint network is obtained.

[0106] It should be noted that after obtaining the initial pre-trained weights of the entire pre-trained fusion-detection joint network and the optimal hyperparameters of the weights of each loss function item through step (3), the quality of the image fusion network module has reached a high level through pre-training and meets the requirements of the segmentation detection network module. At this time, the parameters of the image fusion network module are frozen, and the pre-trained fusion-detection joint network is further trained and optimized using transfer learning combined with the training set constructed in step (1), the parameters of the segmentation detection network module are optimized, the segmentation detection network module is finely optimized, and the accuracy of the downstream segmentation detection network module is improved. By further optimizing the segmentation detection network module, the computational complexity can be effectively reduced, focusing on improving the segmentation accuracy, and ultimately obtaining the parameters of the ideal fusion-detection joint network.

[0107] (5) The infrared thermal imaging image and visible light image pairs to be tested in the test set are input into the finally trained fusion-detection joint network to perform disease segmentation detection and obtain the segmentation detection results, which include the category, positioning bounding box and pixel-level segmentation mask of each instance.

[0108] It should be noted that the final trained fusion-detection joint network obtained in step (4) is used to perform disease segmentation detection on the building exterior wall image, and finally the category, positioning bounding box and pixel-level segmentation mask of each instance can be obtained.

[0109] (6) According to the segmentation detection results obtained in step (5), the disease area and distribution density are counted, and the health of the exterior wall is evaluated using a comprehensive scoring index.

[0110] Specifically, the segmentation detection result obtained in step (5) includes the category, positioning bounding box and pixel-level segmentation mask of each instance. Based on the categories, positioning bounding boxes and pixel-level segmentation masks of all instances, the results of instance segmentation are statistically analyzed by category. The area and position distribution density of each category are calculated based on the pixel-level segmentation mask. A comprehensive scoring index is calculated based on the area and position distribution density of each category. The calculated comprehensive scoring index is used to evaluate the health of the exterior wall. In this way, the health of the exterior wall is evaluated more scientifically and accurately.

[0111] Furthermore, the comprehensive scoring index is calculated through the following steps:

[0112] (6.1) Calculate the proportion of damaged area, which is defined as the ratio of the total damaged area to the total wall area. The calculation formula is:

[0113]

[0114] Among them, R defect Indicates the proportion of diseased area; The area of ​​the i-th disease, N1 represents the total number of diseases. Specifically, the number and area of ​​the diseases can be obtained according to the segmentation detection results. The location is determined by the center point coordinates of each positioning bounding box (x i ,y i ) calculations, used to analyze the spatial distribution characteristics of diseases; represents the area of ​​the jth ancillary facility, N2 represents the total number of ancillary facilities, and the number and area of ​​ancillary facilities can be counted according to the segmentation detection results; A wall Represents the total area of ​​the wall. Specifically, the total area of ​​the wall can be calculated based on the segmentation detection results. The proportion of the damaged area can directly reflect the severity of the disease.

[0115] (6.2) Calculate the average area of ​​a single disease, which is defined as the ratio of the total disease area to the number of diseases. The calculation formula is:

[0116]

[0117] Among them, C defect It represents the average area of ​​a single defect, which can reflect the average severity of a single defect. Considering that the larger the area, the more serious the impact of the defect on the wall structure may be, a reference threshold can be set.

[0118] (6.3) Calculate the standard deviation of the disease centroid distance. Specifically, calculate the Euclidean distance between each disease centroid and the standard deviation of all centroid distances to obtain the standard deviation of the disease centroid distance. The calculation formula is:

[0119]

[0120] Among them, D defect represents the standard deviation of the distance from the disease centroid, d ij represents the Euclidean distance between the centroid of the i-th disease and the j-th disease, x i and y i Represents the horizontal and vertical coordinates of the centroid of the ith disease, x j and y j are the horizontal and vertical coordinates of the jth disease centroid, represents all Euclidean distances d ij The standard deviation of the disease centroid distance can accurately reflect the uniformity of the spatial distribution of diseases. If the disease is concentrated in a certain area, the risk is concentrated, affecting the structural integrity, and the score is reduced; if the distribution is discrete, the risk is low and the score is improved.

[0121] (6.4) Based on the scoring logic of the disease area ratio, the average area of ​​a single disease, and the standard deviation of the disease centroid distance, a larger disease area ratio will reduce the health score, a higher average area of ​​a single disease will reduce the health score, and a lower standard deviation of the disease centroid distance will reduce the health score. Combining these three indicators, we get the comprehensive score index, which is calculated as follows:

[0122]

[0123] Among them, h represents the comprehensive scoring index, and the larger its value, the healthier the exterior wall; D max It represents the reference threshold of the standard deviation of disease distance; w1, w2 and w3 are the weights of different sub-scoring items and their sum is 1. The weights can be adjusted according to actual conditions to focus on different evaluation items.

[0124] Corresponding to the aforementioned embodiment of the exterior wall disease detection and health assessment method based on the fusion-segmentation joint network, the present invention also provides an embodiment of the exterior wall disease detection and health assessment device based on the fusion-segmentation joint network.

[0125] See also Figure 3 An embodiment of the present invention provides an exterior wall defect detection and health assessment device based on a fusion-segmentation joint network, comprising one or more processors and a memory coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the exterior wall defect detection and health assessment method based on a fusion-segmentation joint network in the above embodiment.

[0126] The embodiment of the exterior wall disease detection and health assessment device based on the fusion-segmentation joint network of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 3 As shown, this is a hardware structure diagram of any device with data processing capability where the exterior wall disease detection and health assessment device based on the fusion-segmentation joint network of the present invention is located. Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0127] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0128] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0129] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for exterior wall disease detection and health assessment based on a fusion-segmentation joint network in the above embodiment is implemented.

[0130] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0131] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for exterior wall disease detection and health assessment based on a fusion-segmentation joint network, characterized in that: The following steps are involved: (1) Collect infrared thermal imaging images and visible light images of exterior wall defects and preprocess them to construct a fused image dataset; annotate the fused image pairs in the fused image dataset and amplify them. After amplification, the dataset is randomly divided into a training set and a test set according to the proportion; (2) constructing a fusion-detection joint network, which includes a pixel-weighted image fusion network module and a segmentation detection network module, wherein the image fusion network module is used to fuse the fused image pair, i.e., the registered infrared thermal imaging image and the visible light image, pixel by pixel to obtain a final fused image; and the segmentation detection network module is used to process the final fused image to obtain the category, positioning bounding box and pixel-level segmentation mask of each instance; (3) The fusion-detection joint network is pre-trained using the public MSRS dataset. During the pre-training process, the parameters of the fusion-detection joint network are adjusted with the goal of minimizing the joint loss function to obtain a pre-trained fusion-detection joint network. (4) Using the training set in step (1) to train the pre-trained fusion-detection joint network, during the training process, the parameters of the image fusion network module are frozen with the optimization goal of minimizing the joint loss function, and only the parameters of the segmentation detection network module are adjusted to obtain the final trained fusion-detection joint network; (5) Input the infrared thermal imaging image and visible light image pairs to be tested in the test set into the finally trained fusion-detection joint network to perform disease segmentation detection and obtain the segmentation detection results, which include the category, positioning bounding box and pixel-level segmentation mask of each instance; (6) According to the segmentation detection results obtained in step (5), the disease area and distribution density are counted, and the health of the exterior wall is evaluated using a comprehensive scoring index.

2. The exterior wall disease detection and health assessment method based on fusion-segmentation joint network according to claim 1 is characterized in that: The step (1) specifically includes the following sub-steps: (1.1) Use infrared thermal imaging cameras and visible light cameras to collect infrared thermal imaging images and visible light images of exterior wall defects; (1.2) Preprocessing the infrared thermal imaging images and visible light images: First, filter and remove erroneous image data caused by focusing errors, excessive viewing angle deviations, and repeated shots; then, register and resize the infrared thermal imaging images and visible light images of different resolutions to obtain registered infrared thermal imaging images and visible light images; (1.3) Constructing a fused image dataset based on the registered infrared thermal imaging image and visible light image; (1.4) Use annotation tools to perform mask segmentation annotation of multiple categories of targets on the fused image pairs in the fused image dataset at the pixel level as semantic segmentation labels; and use multiple data augmentation methods to amplify the fused image dataset. After amplification, the dataset is randomly divided into training and test sets in proportion.

3. The exterior wall disease detection and health assessment method based on fusion-segmentation joint network according to claim 1 is characterized in that: The image fusion network module specifically includes: First, the registered infrared thermal imaging image and visible light image are grayscaled and then spliced ​​along the channel direction; then, the weight of each pixel of the infrared thermal imaging image and the weight of each pixel of the visible light image are obtained through the convolution operation of multiple convolutional layers; then, the weight of each pixel of the infrared thermal imaging image and the weight of each pixel of the visible light image are combined with the source image through element-level multiplication and addition operations to generate a grayscale fused image; finally, the color mode of the grayscale fused image is converted to RGB color to obtain a color fused image as the final fused image, which is expressed as: Among them, I vif Represents the pixel value of the final fused image, W irt Represents the weight coefficient of each pixel of the infrared thermal imaging image, I irt Represents the pixel value of the infrared thermal imaging image, W vis Represents the weight coefficient of each pixel of the visible light image, I vis Represents the pixel value of the visible light image, Represents element-wise multiplication.

4. The exterior wall disease detection and health assessment method based on fusion-segmentation joint network according to claim 1 is characterized in that: The segmentation detection network module specifically includes: The segmentation detection network module adopts the Mask RCNN network, which processes objects of different scales through a feature pyramid. The segmentation detection network module is used to process the final fused image output by the image fusion network module to achieve instance segmentation and obtain the category, positioning bounding box and pixel-level segmentation mask of each instance.

5. The exterior wall disease detection and health assessment method based on fusion-segmentation joint network according to claim 1 is characterized in that: The step (3) specifically includes: The infrared thermal imaging image and visible light image pairs from the public MSRS dataset are input into the joint fusion-detection network. The image fusion network module is used to obtain the final fused image. The segmentation detection network module is used to obtain the category, localization bounding box, and pixel-level segmentation mask of each instance in the final fused image. The fusion loss function is calculated based on the final fused image output by the image fusion network module and its corresponding source image, where the source image refers to the infrared thermal imaging image and the visible light image pair; the semantic segmentation loss function is calculated based on the category, positioning bounding box and pixel-level segmentation mask of each instance output by the segmentation detection network module and the corresponding semantic segmentation label in the MSRS dataset; the joint loss function is calculated based on the fusion loss function and the semantic segmentation loss function; During the pre-training process, the parameters of the fusion-detection joint network are adjusted with the minimization of the joint loss function as the optimization goal, and the weight of each pixel of the infrared thermal imaging image and the weight of each pixel of the visible light image in the image fusion network module, as well as the weight of the texture loss and the weight of the semantic segmentation loss function in the joint loss function are dynamically adjusted until the preset training rounds are reached or the preset loss threshold is less than, and the optimal weight of each pixel of the infrared thermal imaging image and the optimal weight of each pixel of the visible light image, as well as the optimal weight of the texture loss and the optimal weight of the semantic segmentation loss function are obtained to obtain the pre-trained fusion-detection joint network.

6. The exterior wall disease detection and health assessment method based on fusion-segmentation joint network according to claim 1 or 5 is characterized in that: The joint loss function includes a fusion loss function and a semantic segmentation loss function, and its calculation formula is: L total =L vif +βL detect Among them, L total represents the joint loss function, L vif represents the fusion loss function, L detect represents the semantic segmentation loss function, and β represents the weight of the semantic segmentation loss function; The fusion loss function includes brightness loss and texture loss, and its calculation formula is: L vif =L int +αL txt Among them, L int Indicates brightness loss, L txt represents texture loss, α represents the weight of texture loss; The calculation formula for the brightness loss is: Where H and W represent the height and width of the infrared thermal imaging image or visible light image, respectively, ||·||1 represents the L1 norm, and max(I irt ,I vis ) represents taking the maximum value of the pixel values ​​of the infrared thermal imaging image and the visible light image at each pixel position; The calculation formula of the texture loss is: in, represents the Sobel gradient operator, and Represent the gradient amplitudes of the fused image, infrared thermal imaging image, and visible light image, respectively. Indicates taking and The maximum of the two gradient magnitudes. The semantic segmentation loss function includes classification loss, bounding box loss and mask loss, and its calculation formula is: L detect =L seg +L cls +L mask Among them, L seg represents the classification loss, L cls represents the bounding box loss, L mask Represents mask loss.

7. The exterior wall disease detection and health assessment method based on fusion-segmentation joint network according to claim 1 is characterized in that: The step (6) specifically includes: The segmentation detection result obtained in step (5) includes the category, positioning bounding box and pixel-level segmentation mask of each instance. Based on the categories, positioning bounding boxes and pixel-level segmentation masks of all instances, the instance segmentation results are statistically analyzed by category. The area and position distribution density of each category are calculated based on the pixel-level segmentation mask. A comprehensive scoring index is calculated based on the area and position distribution density of each category. The health of the exterior wall is evaluated using the calculated comprehensive scoring index.

8. The exterior wall disease detection and health assessment method based on fusion-segmentation joint network according to claim 1 or 7 is characterized in that: The comprehensive scoring index is calculated and obtained through the following steps: (6.1) Calculate the proportion of damaged area, which is defined as the ratio of the total damaged area to the total wall area. The calculation formula is: Among them, R defect Indicates the proportion of diseased area; The area of ​​the ith disease, N1 represents the total number of diseases; represents the area of ​​the jth ancillary facility, N2 represents the total number of ancillary facilities; A wall Indicates the total area of ​​the wall; (6.2) Calculate the average area of ​​a single disease, which is defined as the ratio of the total disease area to the number of diseases. The calculation formula is: Among them, C defect It represents the average area of ​​a single disease; (6.3) Calculate the standard deviation of the disease centroid distance. Specifically, calculate the Euclidean distance between each disease centroid and the standard deviation of all centroid distances to obtain the standard deviation of the disease centroid distance. The calculation formula is: Among them, D defect represents the standard deviation of the distance from the disease centroid, d ij represents the Euclidean distance between the centroid of the i-th disease and the j-th disease, x i and y i Represents the horizontal and vertical coordinates of the centroid of the ith disease, x j and y j are the horizontal and vertical coordinates of the jth disease centroid, represents all Euclidean distances d ij The average value of (6.4) The comprehensive scoring index is calculated based on the disease area ratio, the average area of ​​a single disease, and the standard deviation of the disease centroid distance. The calculation formula is: Among them, h represents the comprehensive scoring index, D max It represents the reference threshold of the disease distance standard deviation, and w1, w2 and w3 are the weights of different sub-scoring items.

9. A device for detecting and assessing exterior wall defects based on a fusion-segmentation joint network, comprising one or more processors and a memory, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the exterior wall disease detection and health assessment method based on a fusion-segmentation joint network according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by the processor, it is used to implement the exterior wall disease detection and health assessment method based on the fusion-segmentation joint network according to any one of claims 1 to 8.

Images