A building facade hollowing multi-modal detection method and related device

Abstract

The application provides a building facade hollowing multi-modal detection method and related device, and the specific steps are as follows: inputting a thermal infrared image and a visible light image into a building facade hollowing multi-modal detection model to obtain a building facade hollowing position and a hollowing position classification; the building facade hollowing multi-modal detection model comprises a coding-decoding semantic segmentation network with a fusion thermal gradient attention module and a hollowing position auxiliary classification network guided based on visible light image boundary information, the coding-decoding semantic segmentation network is used for extracting temperature distribution features of the thermal infrared image to obtain a hollowing area segmentation mask; and the hollowing position auxiliary classification network is used for extracting boundary information of the visible light image and combining the hollowing area segmentation mask to obtain the hollowing position classification. The application designs an automatic nondestructive detection technology based on a deep learning network, and realizes a more efficient, visual and non-contact detection process.

Description

Technical Field

[0001] This invention belongs to the field of building facade inspection technology, specifically to a multimodal detection method and related device for building facade hollowness. Background Technology

[0002] Building facades, as a crucial external component of building structures, encompass various cladding structures such as walls, facing bricks, stone curtain walls, and paint layers. They are widely used in various civil and industrial buildings, including residential buildings, commercial complexes, industrial plants, and public venues. Throughout a building's entire lifecycle, building facades not only fulfill basic functions such as providing shelter from wind and rain and thermal insulation, but also directly impact the building's aesthetic appearance and operational safety. With the advancement of urbanization, the stock of existing buildings continues to grow, and many early-built buildings have entered the aging maintenance phase. Structural health monitoring of building facades has gradually become a core aspect of building operation and maintenance management. In recent years, various regions have increasingly emphasized building safety, and building facade defect detection has been incorporated into routine building safety inspections. The application scenarios of related detection technologies have expanded from traditional periodic maintenance to post-disaster emergency detection, assessment of old community renovations, and quality acceptance of new buildings, providing key technical support for ensuring building safety and extending building lifespan.

[0003] In the current practical application of building facade inspection, there are many practical problems that urgently need to be solved. On the one hand, the causes of building facade deterioration are complex and the hazards are serious. Affected by a combination of natural environmental factors (such as wind, sun, rain, and freeze-thaw cycles), material aging (such as the degradation of adhesive properties and cracking of cladding materials), construction defects (such as incomplete bonding and improper base treatment), and natural disasters such as earthquakes and typhoons, building facade deterioration occurs frequently, and the deterioration process is often insidious and develops rapidly. This deterioration not only reduces the structural stability of the building and shortens its service life, but may also lead to safety accidents such as cladding peeling and wall collapse, directly threatening the lives and property of residents and passersby. On the other hand, hollowing, a typical early defect of building facades, is particularly difficult to detect. Hollow areas refer to a defect where the bonding between the building facade cladding (such as tiles, stone, paint, etc.) and the base layer (such as concrete walls, masonry walls, etc.) fails, resulting in localized detachment and forming hollow areas. This defect is a crucial early warning indicator for assessing the risk of further deterioration of the building facade. Timely detection of hollow areas is essential for early intervention in the deterioration process and preventing safety accidents. However, because hollow areas are located between the cladding and the base layer and are internal defects, they cannot be directly observed and identified by the naked eye. This presents a natural technical obstacle to detection and makes the timely detection of early hollow area defects a major pain point in building operation and maintenance management.

[0004] To address the challenge of multimodal detection of hollow areas in building facades, various detection technologies have been developed in the industry, with the traditional manual tapping method being the most widely used. The core principle of this method is to utilize the difference in acoustic signals generated when tapping a hollow area compared to a normally bonded area to determine the presence of hollow areas. The specific procedure involves the inspector using a tapping tool (such as a small hammer or screwdriver) to tap the building facade point by point. By listening to the differences in the crispness and dullness of the tapping sound, and combining this with personal experience, the inspector determines whether a hollow area exists at the detection location. In addition to traditional methods, the industry has gradually explored and applied some auxiliary detection technologies, such as ultrasonic testing (UT, which uses the differences in the propagation speed and reflection characteristics of ultrasonic waves in different media to determine the presence of hollow areas by receiving reflected signals), infrared thermography (IRT, which detects hollow areas by capturing abnormal temperature field distribution based on the difference in thermal conduction characteristics between hollow areas and normal areas), and impact echo testing (IET, which analyzes internal structural defects by exciting impact elastic waves and utilizing the reflection and propagation characteristics of the waves). These technologies have begun to be applied in specific scenarios, attempting to compensate for the shortcomings of traditional manual inspection and improve the scientific nature and accuracy of inspection. However, existing detection technologies still have significant limitations and have not completely solved the core pain point of multimodal detection of hollow areas in building facades.

[0005] In summary, current multimodal detection technologies for building facade hollowness cannot simultaneously meet the practical application requirements of high efficiency, accuracy, safety, and low cost. Developing a hollowness detection technology that can adapt to different building types and environmental conditions, and has both high detection accuracy and high operational safety, has become an urgent need in the fields of civil engineering inspection and building operation and maintenance management. It is also a key breakthrough to address the major challenges of current building facade deterioration detection. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a multimodal detection method and related device for hollow building facades. It is based on deep learning network design for automated non-destructive testing technology, achieving a more efficient, visualized, and non-contact testing process.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a multimodal detection method for hollow areas in building facades, the specific steps of which are as follows: By inputting thermal infrared images and visible light images into the multimodal detection model for building facade voids, the location and classification of void locations on the building facade are obtained. The multimodal detection model for building facade hollowness includes an encoding / decoding semantic segmentation network that integrates a thermal gradient attention module and a hollowness location auxiliary classification network guided by visible light image boundary information. The encoding / decoding semantic segmentation network is used to extract the temperature distribution features of the thermal infrared image to obtain the hollowness region segmentation mask; the hollowness location auxiliary classification network is used to extract the boundary information of the visible light image and combine it with the hollowness region segmentation mask to obtain the hollowness location classification.

[0008] Furthermore, the encoder-decoder semantic segmentation network with integrated thermal gradient attention module includes an encoder-decoder semantic segmentation network and a thermal gradient attention module. The encoder-decoder semantic segmentation network adopts a deep convolutional encoder-decoder structure, wherein the encoder extracts features from the thermal infrared image, the thermal gradient attention module obtains feature maps of different levels of the encoder of the encoder-decoder semantic segmentation network, calculates the temperature gradient magnitude in the feature map using the Sobel operator, and normalizes it to obtain the thermal gradient attention mask; the decoder fuses the thermal gradient attention mask with the original features of each decoder layer to obtain the hollow area segmentation mask.

[0009] Furthermore, the specific steps for obtaining the thermal gradient attention mask in the thermal gradient attention module are as follows: 1) Calculate gradient information Assume the original thermal infrared image is T(x,y), a two-dimensional temperature field, where x and y represent the spatial location of the image. Calculate the gradients of the feature map F of each encoder layer in the horizontal (x-direction) and vertical (y-direction):

[0010]

[0011]

[0012] Among them, G T The thermal gradient vector represents the rate of change of the temperature field in the space of the tile surface; x and y are the spatial coordinates, respectively. 2) Calculate the gradient magnitude to characterize the hollow boundary: Implementing the thermal gradient G using the Sobel operator T The first-order partial derivative yields the gradient magnitude G:

[0013]

[0014] 3) Constructing an attention map based on the thermal gradient magnitude map The thermal gradient magnitude is normalized to the [0,1] interval to obtain the normalized gradient map. , denoted as:

[0015] Normalized gradient map Upsampled to the spatial dimensions of feature maps at various scales in the encoder, resulting in a fused gradient map. , denoted as:

[0016] in, The features of the i-th layer encoder; fused gradient maps Expand to the same channel dimension as the feature map and perform channel-by-channel multiplication:

[0017] in, It is the sigmoid activation function. This indicates pixel-wise and channel-wise multiplication. This is the weighted thermal gradient attention mask; thermal gradient attention mask Original features of the decoder layer The input to the decoder is obtained by fusion. .

[0018] Furthermore, the specific steps of the cavity location classification network guided by visible light image boundary information to obtain cavity location classification are as follows: Visible light images Gaussian filtering followed by Canny edge detection yields edge map E:

[0019] Then, Hough line segment detection is performed based on the edge map E to extract all tile bounding boxes {B}. k}:

[0020] Hollow area segmentation mask is represented as Perform connected component analysis on M to extract all hollow regions. Each region Defined as a set of pixels:

[0021] For each hollow area Find the tile bounding box corresponding to its position. And calculate its relative position coordinate center point:

[0022] Map it to the current tile bounding box Normalized coordinates are used to generate a fused image:

[0023] Based on the spatial relationship between the location of the hollow area and the tile boundary, the hollow areas are divided into corner hollow areas, edge hollow areas, and center hollow areas, as detailed below:

[0024] Furthermore, the multimodal detection model for building facade hollowness also includes an uncertainty quantification method for a thermal infrared hollowness segmentation model based on Bayesian probability. This method uses a Bayesian deep neural network to evaluate the uncertainty of each pixel in the hollowness region segmentation mask and outputs an uncertainty map to guide manual secondary verification.

[0025] Furthermore, thermal infrared and visible light images were acquired using a FOTRIC 288+ infrared thermal imager.

[0026] This invention also discloses a multimodal detection method system for hollow building facades, which performs the steps of the multimodal detection method for hollow building facades as described in the above claims, wherein the system includes: The hollow detection module is used to input thermal infrared images and visible light images into the multimodal detection model of building facade hollowness to obtain the location and classification of hollowness in the building facade. The multimodal detection model for building facade hollowness includes an encoding / decoding semantic segmentation network that integrates a thermal gradient attention module and a hollowness location auxiliary classification network guided by visible light image boundary information. The encoding / decoding semantic segmentation network is used to extract the temperature distribution features of the thermal infrared image to obtain the hollowness region segmentation mask; the hollowness location auxiliary classification network is used to extract the boundary information of the visible light image and combine it with the hollowness region segmentation mask to obtain the hollowness location classification.

[0027] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for multimodal detection of hollow building facades.

[0028] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for multimodal detection of hollow building facades.

[0029] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the multimodal detection method for hollow building facades.

[0030] Compared with the prior art, the present invention has at least the following beneficial effects: This invention provides a multimodal detection method for hollow areas in building facades, addressing the core pain point of thermal infrared non-destructive testing (TIB) for hollow areas in ceramic building facade tiles. It significantly improves detection accuracy and reliability, overcoming the limitations of traditional detection methods. While TIB non-destructive testing of hollow areas in ceramic building facade tiles offers advantages such as good visualization, high detection efficiency, and low cost, heat diffusion easily leads to blurred infrared thermal image boundaries, drastically reducing segmentation accuracy—a problem that has long plagued the industry. Simultaneously, traditional semantic segmentation models have many shortcomings in this detection task, failing to meet the demands for accurate detection. To address these issues, this invention constructs a TIHSNet hollow area detection framework based on an encoding / decoding architecture and innovatively introduces a thermal gradient attention module. By modeling the spatial temperature change characteristics in thermal infrared images, it accurately captures the temperature difference between hollow areas and intact areas, effectively overcoming the problem of blurred hollow thermal image boundaries caused by heat diffusion and significantly improving the segmentation accuracy of hollow area boundaries. Experimental results show that the multimodal detection model for building facade hollowness has mIoU = 0.896, Precision = 0.961, Recall = 0.949, and F1-Score = 0.944, which are significantly improved compared with traditional image processing methods and existing deep learning methods.

[0031] This invention fills several research gaps in the field, accurately matches actual engineering needs, and provides a new technical path and standardized support for multimodal detection of hollow areas in building facades. In engineering applications, construction workers often focus on the location and type of hollow areas, but current methods cannot classify hollow areas, making it difficult to support subsequent repair work. To address this gap, this invention innovatively integrates tile boundary information from visible light images and designs a hollow area location-assisted classification network. Through steps such as Gaussian filtering, Canny edge detection, and Hough line segment detection, tile boundaries are extracted. Combined with connected component analysis of the hollow area segmentation mask, accurate classification of central hollow areas, edge hollow areas, and corner hollow areas is achieved, effectively meeting the needs of engineering applications and providing an important basis for subsequent targeted hollow area repair strategies and improving repair efficiency and quality.

[0032] Furthermore, this invention incorporates an uncertainty quantification mechanism, constructs a dual-output system of predicted value and confidence score, quantifies the uncertainty of the model using a Bayesian deep neural network, and performs uncertainty quantification on four representative semantic segmentation models through variational inference approximating the Bayesian deep neural network. It was found that high uncertainty regions often appear in hard-to-define hollow boundaries, and the uncertainty increases with the degree of boundary ambiguity. By filtering data above the uncertainty threshold for manual intervention, it can not only provide early warning of small-scale hollow false detection problems—avoiding unnecessary workload and even life safety threats—but also help improve the boundary extraction accuracy, significantly improving the reliability of the detection results. At the same time, it enhances the interpretability of the model and meets the stringent requirements for detection accuracy in engineering practice.

[0033] On the other hand, addressing the scarcity of publicly available datasets for building facade hollowness detection in the field of thermal infrared and visible light fusion, this invention constructs and publishes the first multimodal dataset for building facade hollowness detection, XAUAT-2000. This dataset covers different types of walls and different hollowness locations, providing standardized data support for subsequent research in the field and effectively promoting technological innovation and industrial application in the multimodal detection of building facade hollowness. Simultaneously, this dataset also provides reliable support for the training and validation of the TIHSNet model, ensuring the stability and superiority of the model's performance, further enhancing the academic research value and engineering application feasibility of this invention.

[0034] This invention utilizes deep learning networks to design automated non-destructive testing technology, fully leveraging the inherent advantages of thermal infrared non-destructive testing. Employing a multimodal fusion approach combining thermal infrared and visible light, it achieves a non-contact, visualized testing process. This allows for the detection of voids, boundary segmentation, location classification, and reliability assessment without physical contact with the building facade. This avoids damage to the building structure while significantly improving testing efficiency, enabling rapid screening of large areas of building facades for voids. It is suitable for various building ceramic facade inspection scenarios. Furthermore, the scientifically designed model structure, with its fusion of thermal gradient attention modules and encoding / decoding architecture, fully leverages the temperature characteristics of thermal infrared images, effectively solving the segmentation challenges caused by heat diffusion. The introduction of visible light boundary information accurately fills the gaps in void location classification, matching the actual needs of construction personnel. An uncertainty quantification mechanism balances detection accuracy and reliability, providing early warning of false detection risks and assisting in optimizing boundary extraction. Compared to traditional methods and existing deep learning methods, this invention significantly improves detection accuracy, efficiency, and practicality, effectively promoting the standardization and automation of building health inspection. Attached Figure Description

[0035] Figure 1 This is a design drawing for the experimental wall.

[0036] Figure 2 For the experimental wall 1.

[0037] Figure 3 For the experimental wall 2.

[0038] Figure 4 It is a FOTRIC 288+ infrared thermal imager.

[0039] Figure 5 Three types of hollow locations are identified: center hollow, edge hollow, and corner hollow. The images show the on-site conditions and their thermal infrared images.

[0040] Figure 6 This is the overall framework diagram of the multimodal detection model for hollow areas on building facades.

[0041] Figure 7 The diagram shows the encoding / decoding semantic segmentation network that incorporates the thermal gradient attention module.

[0042] Figure 8 This is a schematic diagram of a cavity location-assisted classification network guided by visible light image boundary information.

[0043] Figure 9 A schematic diagram of an uncertainty quantification mechanism that introduces a dual output of predicted value and confidence level.

[0044] Figure 10 It serves as an indicator for evaluating uncertainty.

[0045] Figure 11 A radar chart comparing various indicators of the model.

[0046] Figure 12 This section compares image processing methods with deep learning methods.

[0047] Figure 13 Visualize ablation experiments for fuzzy, hollow thermal images with blurred boundaries.

[0048] Figure 14 Visualize ablation experiments for small-sized hollow targets.

[0049] Figure 15 Visualize complex thermogram ablation experiments with multi-scale characteristics.

[0050] Figure 16The visualization of the multimodal classification results (partial) shows six samples, each containing: (a) thermal infrared image (IRT), (b) hollow area segmentation mask (Precision), (c) infrared and visible light boundary fusion map (Fusion), (d) visible light image (RGB), (e) extracted tile boundary map (Edges), and (f) hollow spatial location classification result (Classify), where yellow represents central hollow areas, blue represents edge hollow areas, and red represents corner hollow areas. This invention achieves spatial localization and fine-grained location classification of hollow areas by fusing thermal infrared and visible light images, enhancing the engineering usability of the detection results.

[0051] Figure 17 This is a map showing the prediction results and uncertainty distribution for layered defects of different sizes and edge sharpness in infrared thermal imaging images.

[0052] Figure 18 This is a statistical analysis chart of pixel-level uncertainty under three detection scenarios. Detailed Implementation

[0053] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0054] This invention provides a multimodal detection method for hollow areas in building facades based on multimodal semantic segmentation and uncertainty quantification. The specific steps are as follows: Step 1: Construct a multimodal dataset of building facade hollowness containing thermal infrared and visible light image pairs. Currently, publicly available datasets on building facade hollowness are extremely scarce, especially in the fields of infrared thermal imaging and visible light fusion, where almost no publicly available data resources are available. Therefore, creating dedicated datasets has become an urgent need to promote research and application in this field.

[0055] Preliminary research revealed that reinforced concrete walls and sintered clay brick walls are currently the main types of building walls. Based on this, this invention constructed two types of experimental walls: Experimental Wall 1: a 100cm×80cm×15cm reinforced concrete wall and Experimental Wall 2: a 100cm×100cm×5cm sintered clay brick wall. Before applying tiles to the experimental walls, hollow defects were artificially created. Thirty-two building facade tiles with known true values ​​of hollowness were designed and manufactured. The mixing ratio of concrete, mortar, and aggregate used in the walls was 2:6:1. The tiles used were 100mm×100mm matte porcelain tiles and 60mm×240mm matte ceramic tiles. Figures 1-3 As shown.

[0056] like Figure 4As shown, data acquisition was performed using the FOTRIC 288+ infrared thermal imager developed by FOTRIC Corporation. The infrared thermal imager uses an uncooled infrared focal plane detector with an infrared resolution of 1280*1024, a pixel pitch of 12μm, and a thermal sensitivity of 30mK. The thermal infrared images and visible light images are in .jpg format.

[0057] Based on the location of hollow spots on the tiles, they can be divided into three types: central hollow spots, edge hollow spots, and corner hollow spots. [x],[y] Field images and thermal infrared images of three types of hollow areas are shown below. Figure 5 As shown.

[0058] To prevent overfitting, standard data augmentation methods such as rotation, flipping, and random jitter were applied to the multimodal dataset for building hollow detection, resulting in 2000 sets of sample data (IRT+RGB). Professional inspectors then labeled the dataset according to the actual physical dimensions of the hollow areas to generate a label file. Pixels belonging to hollow areas were marked as white, while pixels belonging to complete areas were marked as black.

[0059] As shown in Table 1, the dataset is divided into a training set (Train): validation set (Val): test set (Test) ratio of 7:1:2. The training set consists of 1434 IRT and visible light images, while the validation and test sets consist of 222 and 446 IRT and visible light images, respectively. The training set is used to train the deep learning model, adjusting its parameters such as the weights and biases of the neural network to minimize the loss function. The validation set is used for model validation or hyperparameter tuning. Validation data is not used in model training but is used to monitor model performance for adjustments during training. The test set is used to finally evaluate the model's generalization ability.

[0060] Table 1 Dataset Partitioning

[0061] Step 2: Establish a multimodal detection model for building facade hollowness using TIHSNet (Thermal-Infrared Heterogeneous Segmentation Network). like Figure 6As shown, the multimodal detection model for building facade voids adopts an encoding and decoding structure including an encoding and decoding semantic segmentation network that integrates thermal gradient attention modules, a void location auxiliary classification network guided by visible light image boundary information, and a Bayesian probability-based uncertainty quantification method for thermal infrared void segmentation models. Specifically, it extracts temperature distribution features by integrating thermal gradient perception-guided attention network, combines visible light image boundary information to assist in void location classification, and introduces a model uncertainty quantification mechanism to guide a second manual review of the uncertainty prediction results, enhancing the interpretability and credibility of the results. This constructs a robust, accurate, and interpretable intelligent void detection process.

[0062] 1) Encoding-decoding semantic segmentation network integrating hot gradient attention modules The encoding / decoding semantic segmentation network with integrated hot gradient attention module mainly consists of two parts: the encoding / decoding semantic segmentation network and the hot gradient attention module. Its structure is as follows: Figure 7 As shown. The semantic segmentation network employs a deep convolutional encoder-decoder structure to extract features from thermal infrared (IR) images and ultimately generate a high-precision mask for hollow regions. The thermal gradient attention module, through the calculation of thermal gradients and an attention mechanism, helps the network focus on temperature changes in and around the hollow regions, suppressing environmental noise interference while improving the segmentation accuracy of the hollow regions. Specifically: 1.1) Encoding / Decoding Semantic Segmentation Network The encoder's task is to extract high-level semantic features from the input image while gradually reducing the image's spatial resolution. The encoder consists of multiple convolutional layers, batch normalization layers, and max pooling layers to progressively extract local temperature features and high-level semantic information. Let the input image be (in These are the height, width, and number of channels of the image, denoted as the 0th layer feature map. .

[0063] The encoder consists of multiple convolutional layers, the first... Layer convolution operations use convolution kernels and bias It acts on the feature map of the previous layer. The output is as follows:

[0064] Among the symbols Represents a two-dimensional convolution operation, outputting... Indicates the first The response after convolution. Batch normalization and non-linear activation are performed after convolution to obtain the feature map of the current layer. :

[0065] in They are The mean and standard deviation for the current batch. These are learnable scaling and translation parameters. Represents a nonlinear activation function (commonly ReLU, i.e.) After the above processing, the first... Layer feature map It includes a deeper level of feature representation of the input image.

[0066] To progressively reduce spatial resolution, a max-pooling layer is used immediately after several convolutional operations. The max-pooling function is defined. For feature maps Perform downsampling operation to generate output. :

[0067] in Indicates The pooling window region centered on (e.g.) (Region). If max pooling with a stride of 2 is used, each pooling operation reduces the feature map space size to half its original size in each dimension. The pooled feature map. It is used as input to the next layer of convolution, thereby enabling image feature extraction and spatial resolution reduction in the encoder stage.

[0068] The decoder is responsible for restoring the low-resolution, high-level semantic feature maps extracted by the encoder into an output image of the same size as the original input image. For the encoder's... Layered pooling output Upsampling (such as transpose convolution) is denoted as:

[0069] in This indicates an upsampling operation, causing the output to... Spatial dimensions and encoder number Layer feature map The same applies. To utilize the shallower, more detailed features in the encoder, the upsampled features are concatenated with the corresponding encoder features along the channel dimension (skip connection) to obtain the fused features:

[0070] in This indicates a splicing operation along the channel direction. This skip connection structure helps preserve boundary and spatial detail information, improving segmentation accuracy.

[0071] Features after splicing The input is fed into the convolutional layer of the decoder for further processing. Let the convolutional kernel of this layer be... (Bias is) Then the output after convolution, BN, and activation is:

[0072] in Indicates the decoder's first The feature map of each layer. By repeatedly upsampling, concatenating, and convolutional operations, the decoder expands the spatial resolution layer by layer, and finally outputs a feature map with the same height and width as the input image.

[0073] In the final layer of the network, the top-level features of the decoder are mapped onto the hollow region segmentation mask. Assume the highest-resolution feature map output by the decoder is... The output layer convolution kernel and bias are The class probability map is obtained by convolution and adding activation (such as Softmax):

[0074] in This indicates the predicted segmentation result. The number of categories is (typically "hollow areas" and "background" in this task). In this way, the model can accurately segment hollow areas in the image while preserving the overall structural information.

[0075] 1.2) Thermal Gradient Attention Module (TGAttentionModule TGAM) To effectively identify hollow areas in thermal infrared images of building surfaces, the key lies in accurately characterizing the abnormal heat diffusion caused by these hollow areas. As a structural defect, hollow areas exhibit significantly different thermal conductivity characteristics compared to surrounding normal areas, manifesting in infrared images as abrupt temperature changes at the edges and uneven internal temperature distribution. Traditional semantic segmentation methods struggle to fully utilize this detailed information about temperature changes, leading to area prediction bias in hollow area identification. Therefore, this paper proposes a thermal gradient-driven attention mechanism based on an encoding / decoding semantic segmentation network. This mechanism enhances the model's ability to perceive hollow areas by introducing a thermal gradient attention module. The core idea of ​​this module is to guide the neural network to focus on areas with significant abrupt changes in heat diffusion based on the significant temperature gradient information in the thermal infrared image, thereby improving the accurate identification of hollow area boundaries.

[0076] In terms of module design, this invention first extracts multi-scale feature pyramids at different levels of the encoder and then rearranges and unifies their features for subsequent fusion processing. Subsequently, the temperature gradient magnitude in the feature map is calculated using the Sobel operator to construct a thermal gradient map, which is then normalized to generate an attention mask. This mask dynamically adjusts the feature responses of different regions, guiding the decoder to focus on high-gradient areas, thereby more accurately recovering the spatial distribution of voids. Through this thermal gradient-driven feature extraction mechanism, the model not only improves its sensitivity to local thermal anomalies but also significantly enhances its ability to characterize the boundary structure of voids, effectively improving the segmentation quality of void regions. The specific process is as follows.

[0077] Step 1 Calculate the gradient information of the input features. Thermal gradient (G) T ( ) represents the rate of temperature change per unit length, used to describe how heat is distributed and transported in space, as shown in the following formula:

[0078] Among them, G T The thermal gradient vector (unit: K / m) represents the rate of change of temperature with respect to spatial coordinates. T represents the temperature gradient, i.e., the partial derivatives along different directions; x, y, z are the spatial coordinates (unit: m). In the scenario of heat conduction on the tile surface, mainly considering two-dimensional thermal infrared images, the rate of change of the temperature field in the space of the tile surface can be simplified as:

[0079] Assume the original thermal infrared image is T(x,y), a two-dimensional temperature field, where x and y represent the spatial location of the image. Calculate the gradients of the feature map F of each encoder layer in the horizontal (x-direction) and vertical (y-direction):

[0080]

[0081] Implementing the thermal gradient G using the Sobel operator T First-order partial derivative:

[0082] Calculate the gradient magnitude:

[0083] The magnitude of the thermal gradient G(x,y) essentially characterizes the region of drastic change in the temperature field, i.e., the possible cavitation boundary.

[0084] Step 2Generate thermal gradient attention mask An attention map is constructed based on the thermal gradient magnitude to guide the network to pay more attention to thermal anomaly regions. First, the thermal gradient magnitude map is normalized to the [0,1] interval to generate an attention mask.

[0085] Then, feature alignment and upsampling are performed to normalize the gradient map. Upsampling to the spatial dimensions of feature maps at various scales in the encoder yields the fused gradient map, denoted as:

[0086] in, Let be the features of the encoder at layer i.

[0087] Finally, a hot gradient attention mask is constructed, which expands the fused gradient map to the same channel dimension as the feature map, and then performs channel-by-channel multiplication:

[0088] in, It is the sigmoid activation function. This represents pixel-wise, channel-wise multiplication (Hadamard product). This is the weighted thermal gradient attention mask. The final result is a multi-scale thermal gradient attention mask set {A1, A2, ..., A...}. n}

[0089] Step 3 Add thermal gradient attention mask to the decoder layer During the decoding stage, the network can integrate semantic, boundary, and thermophysical information to improve the localization and segmentation accuracy of hollow areas. The following fusion method is introduced into the standard decoder:

[0090] in, These are the raw features transmitted from the corresponding layer of the encoder via skip connections. This is the thermal gradient attention mask for this layer. This serves as the input to the fused decoder. This fusion incorporates the semantic features of the encoder, thermal gradient-driven boundary enhancement information, and the decoder's own upsampling features.

[0091] 2) Multi-modalClassification, a cavity location-aided classification network guided by visible light image boundary information. In the detection of hollow tiles, it is not only necessary to detect "whether hollow tiles exist," but also to classify them according to their location on the tile (edge, corner, center). This is a crucial step in translating the detection results into engineering maintenance decisions (whether to replace or re-apply sealant). However, thermal infrared images are difficult to extract tile boundary information due to blurred textures and unclear boundaries. Therefore, fusing boundary information from visible light images to classify hollow tile locations is an effective solution. In the hollow tile location-assisted classification network, firstly, the tile boundary information is extracted using visible light images. Then, the hollow tile regions segmented from the thermal infrared images are mapped onto the tile boundary boxes. Finally, based on the spatial relationship between the location of the hollow tile region and the tile boundary, it is classified into corner hollow tiles, edge hollow tiles, and center hollow tiles, as shown below. Figure 8 As shown, the specific steps are as follows: Step 1 Tile boundary extraction Visible light images Gaussian filtering followed by Canny edge detection yields edge map E:

[0092] Then, Hough line segment detection is performed based on the edge map E to extract all tile bounding boxes {B}. k}:

[0093] Step 2 Boundary information and segmentation mask fusion The mask for segmenting hollow regions can be represented as: Perform connected component analysis on M to extract all hollow regions. Each region Defined as a set of pixels:

[0094] For each hollow area Find the tile bounding box corresponding to its position. And calculate its relative position coordinate center point:

[0095] Map it to the current tile bounding box Normalized coordinates are used to generate a fused image:

[0096] Step 3 Hollow area location classification Based on the spatial relationship between the location of the hollow area and the tile boundary, it is classified into corner hollow, edge hollow, and center hollow for the purpose of classifying the location of hollow areas: Corner: If

[0097] Edge: If and or vice versa Center hollow (Center): If

[0098] It can be formalized as a classification function:

[0099] 3) Uncertainty Quantification (UQ) Method for Thermal Infrared Hollow Drum Segmentation Model Based on Bayesian Probability While deep learning excels in image recognition and defect detection tasks, it generally suffers from the "black box effect," where the model's internal decision-making mechanisms lack interpretability and transparency. This problem is particularly pronounced in scenarios like hollow drum detection, where safety and reliability are paramount. Adding to the complexity, infrared detection results are highly sensitive to changes in the external environment; factors such as ambient temperature, wind speed, and solar radiation can all interfere with image features. When the model's output lacks interpretability or cannot be effectively verified, the risk of incorrect judgments inevitably increases, reducing the overall credibility of the system. Therefore, this invention proposes an uncertainty quantification method based on Bayesian probability, aiming to interpret and model the credibility of hollow drum segmentation results at the model level. Figure 9 As shown, this method can evaluate the uncertainty of each pixel in the model's predicted hollow region segmentation mask. The uncertainty map can visually indicate the uncertainty of the deep learning model's prediction. Combined with the uncertainty quantification indices p(a|c) and p(u|i), we can understand the reliability of the model's prediction results (hollow region segmentation mask) and its indicative role in erroneous results. p(a|c) represents the probability of correct prediction among deterministically predicted pixels, directly revealing the model's performance; p(u|i) represents the probability of uncertainty among erroneously predicted pixels. The higher this value, the better the model's uncertainty quantification can perceive erroneous predictions, thereby improving the overall reliability of the prediction results. The uncertainty map is generated for manual verification, guiding secondary checks and improving both prediction reliability and efficiency.

[0100] Bayesian deep neural networks provide a framework that incorporates uncertainty by modeling the posterior distribution of mask pixels in segmenting hollow regions. Given a dataset:

[0101] Where, x n Represents a certain input variable, y nThis represents the mask for segmenting the hollow region corresponding to a certain input variable. New sample pairs { , The prediction distribution of a Bayesian neural network can be modeled as follows:

[0102] Where W represents the model weights. The softmax function applied to the model output is denoted as f. W ( ), is the posterior distribution of the weights, used to capture the set of possible model parameters given the data.

[0103] The limiting part of the predicted distribution of a Bayesian neural network is the computationally intractable posterior distribution. This is because the dimension of W is typically large. However, this problem becomes tractable through variational reasoning. Approximating the posterior with a tractable but sufficiently close approximation q is called the variational approximation. To measure the closeness between q and p(w|Dn), the inverse KL divergence is typically used, as shown below:

[0104] It can be proven that minimizing the KL divergence KL This is equivalent to maximizing the lower bound of evidence (ELBO), which can be easily optimized using common loss functions (such as cross-entropy loss) and stochastic optimizers. This invention uses a Bernoulli distribution q(W), where nodes in the dropout layer are distributed with a given probability p. d The dropout layer is kept on or off randomly. This is achieved by adding dropout layers using MCdropout. Adding these dropout layers also serves as a regularization measure, helping to stabilize the estimation. The dropout layer remains on during model testing. During inference, the predicted probability distribution is obtained through multiple forward passes and the addition of dropout layers.

[0105] Uncertainty maps visually indicate the uncertainty of deep learning model predictions. Combined with uncertainty quantification metrics p(a|c) and p(u|i), they reveal the model's reliability in predicting results (segmentation masks) and its ability to indicate erroneous outcomes. p(a|c) represents the probability of a correct prediction among deterministically predicted pixels, directly revealing the model's performance. p(u|i) represents the probability of uncertainty among erroneously predicted pixels; a higher p(u|i) value indicates a stronger ability of the model to detect erroneous predictions, thus improving the overall reliability of the prediction results. The generated uncertainty map is intended for manual verification, guiding secondary validation and improving both reliability and efficiency.

[0106] Step 3, Experiment Setup To achieve the experimental objectives, this invention was based on an Ubuntu 20.04.6 system, using a dual-card NVIDIA GeForce RTX 4090 GPU and an AMD Ryzen 9 7950X CPU (with 32GB of RAM). The deep learning framework used was PyTorch-2.2.2 + cu121, and the experimental environment was configured with CUDA 12.2 + Python 3.10.0. During testing, 5 random forward passes were performed (i.e., K = 5). To obtain better generalization ability, after appropriate preliminary experiments, the training parameters batch_size were set to 32, epochs to 200, and the initial learning rate to 1×10⁻⁶. -4 To obtain the optimal model that produces both uncertainty estimation and avoids overfitting the training data, the dropout rate is set between [0.01, 0.1]. The model uses the Adam optimizer for network optimization, and the learning decay strategy employs cosine annealing. Training uses the binary cross-entropy loss function, as shown in the following formula:

[0107] Where C is the number of samples, y i p(y) is the true value of the i-th sample. i ) is the probability that the i-th sample belongs to a specific class.

[0108] 1) Segmentation evaluation indicators The segmentation accuracy of the model is evaluated using precision, recall, F1 score, and intersection-over-union (IoU). Model complexity is evaluated using floating-point operations (Flops) and the number of parameters.

[0109] Precision measures the proportion of samples that the model predicts as positive (or the target class) when they are actually positive. In other words, it reflects the accuracy of the model in predicting positive samples. Precision is calculated as follows:

[0110] In this context, TP (True Positive) represents the true positive class, and FP (False Positive) represents the false positive class.

[0111] Recall measures the proportion of all samples that are actually positive that are correctly predicted as positive by the model. It reflects the model's coverage of positive samples. The recall rate is calculated as follows:

[0112] In this context, TP (True Positive) represents the true class, and FN (False Negative) represents the false negative class.

[0113] The F1 score is the harmonic mean of precision and recall, taking into account the combined effects of both. It seeks a balance between precision and recall, and is particularly suitable for imbalanced class scenarios. The F1 score is calculated as follows:

[0114] The intersection-to-union ratio (IoU) measures the degree of overlap between two regions (predicted and ground truth regions), specifically the overlap between the model's predicted image for hollow areas and its ground truth label image. IoU is the ratio of the area of ​​intersection to the area of ​​union of the predicted and ground truth regions, and it is calculated as follows:

[0115] Where area-overlap represents the area of ​​intersection between the predicted region and the real region, and area-union represents the area of ​​union between the predicted region and the real region.

[0116] 2) Uncertainty evaluation indicators This invention aims to achieve a model with low uncertainty and high accuracy, thereby improving model reliability; or a model with low accuracy and high uncertainty, allowing for timely detection of model errors and their resolution by professionals, thus reducing risk. Based on this, the following two conditional probabilities are defined: 1. The probability that an output is accurate given that the model has low uncertainty about it.

[0117] 2. The probability that the model's output is uncertain when it makes a prediction error (i.e., is inaccurate).

[0118] like Figure 10 The image shows the calculated metrics and the selected window size. The algorithm performs a sliding analysis on the prediction results, ground truth labels, and uncertainty graph, similar to a convolution operation. It calculates the accuracy of each window by iterating through the predicted and actual labels; if the accuracy exceeds a set threshold, the window is marked as accurate. Similarly, it calculates the mean uncertainty for each window from the uncertainty graph; if the mean uncertainty exceeds a set threshold, the window is marked as uncertain. The uncertainty threshold `uth` is derived from the minimum and maximum uncertainty values.

[0119] Where t ranges from [0,1]. After all dimensions of the image have been traversed, a confusion matrix is ​​constructed, containing the number of the following category windows: accurate and certain (nac), accurate but uncertain (nau), inaccurate but certain (nic), and inaccurate and uncertain (niu). Subsequently, the formula for calculating the conditional probability can be derived as follows:

[0120]

[0121] Step 4: Model Testing 1) Comparison of experimental results with other methods 1.1) Quantitative Results Table 2 presents the comprehensive comparison results of various methods on the self-built infrared thermal imaging hollow drum dataset. The experiment used four core indicators to evaluate the models: mean intersection-over-union (mIoU), precision, recall, and F1 score. The comparison methods cover traditional image processing techniques, deep learning models based on convolutional neural networks, deep learning models based on Transformers, and the TIHSNet model proposed in this paper. The specific analysis is as follows: Table 2 Test results of the self-made dataset

[0122] Note: Flops are in units of M, and Params are in units of M.

[0123] like Figure 11 As shown in the quantitative comparison results, the accuracy of manual methods is lower, but slightly higher than that of image processing methods, and the accuracy of hollow drum hammering is higher than that of manual hammering. Global-TS (Global Thresholding Segmentation) and Canny-ED (Canny Edge Detection), as traditional image processing methods, achieved mIoU of 0.647 and 0.631 respectively, indicating that traditional methods have weak segmentation performance on thermal infrared data. Both Precision and Recall metrics are below 0.75, especially Canny-ED with a Recall of only 0.716, indicating that image processing methods lack the ability to model the context of complex thermal infrared images, making it difficult to accurately capture hollow drum regions in complex backgrounds. The segmentation results are easily affected by noise and detail loss, making them unsuitable for hollow drum detection tasks.

[0124] Deep learning models significantly outperform image processing methods in mIoU and other metrics, demonstrating the advantages of deep learning in feature extraction and hollow region identification. Classic models such as U-Net, U-Net++, and PSPNet all achieved mIoUs exceeding 0.87. U-Net stands out with an mIoU of 0.872 and an F1-Score of 0.930, showcasing its powerful feature extraction capabilities. Models like DeepLabv3 and SegNet have a more balanced Precision and Recall (both close to or exceeding 0.92), but are slightly inferior to the U-Net series in terms of mIoU. Therefore, this invention selects the U-Net model as the baseline network for model innovation.

[0125] Models like D-LinkNet and TransUNet further improved performance, with TransUNet demonstrating the advantages of the Transformer architecture with an mIoU of 0.881, Precision of 0.941, and Recall of 0.944. However, the improvements of these models are relatively small compared to classic models, with F1-Scores hovering between 0.930 and 0.940, indicating limited gains on this dataset. More recent models such as MANet and DSCNet_Pro showed stronger detail capture capabilities, especially in Precision, which reached new heights (above 0.95). However, the Recall metrics of these models fluctuated slightly, possibly due to an overemphasis on edge information while neglecting some detailed regions.

[0126] The TIHSNet model proposed in this invention outperforms other models in all metrics, achieving an mIoU of 0.896, a precision of 0.961, a recall of 0.949, and an F1-Score as high as 0.944. Compared to the second-highest performing model (TransUNet with an mIoU of 0.881), TIHSNet improves mIoU by 1.5 percentage points and also shows improvements in precision and recall, fully validating the effectiveness of the model architecture design.

[0127] 1.2) Qualitative Results like Figure 12As shown, the qualitative visualization effects of different detection methods on a self-built hollow tile thermal infrared dataset are presented. The first column shows the results of manual tapping, the second column shows the results of manual tapping with a hollow tile hammer, and the third column shows the thermal infrared image (input image), corresponding to the thermal infrared features of the hollow tile area. This invention generates different segmentation results (the columns on the right) for ordinary, complex background, blurred edge, small size, and multi-scale hollow tiles, and compares them with the ground truth labels to evaluate the effectiveness of different methods in segmenting hollow tile areas.

[0128] Manual tapping methods are generally subjective and have low accuracy, while the hollow-drum hammer tapping method has high sensitivity and relatively accurate detection. Threshold segmentation has a high false positive rate in complex backgrounds and a high false negative rate in multi-scale hollow conditions. Edge detection performs poorly in cases of blurred edges and complex backgrounds. Deep learning methods can segment hollow areas well, but most prediction results have poor boundaries and even false negatives in multi-scale hollow conditions.

[0129] TIHSNet demonstrates significant advantages across all scenarios, particularly in complex, edge-blurred, small-sized, and multi-scale scenes, greatly enhancing boundary extraction and multi-scale feature capture capabilities. Traditional methods (such as Adaptive-TS and Canny-ED) struggle to adapt to complex scenes and detail requirements, while deep learning methods (such as U-Net, TransUnet, and DSCNet_Pro) lag behind TIHSNet in detail extraction and multi-scale adaptability. This fully demonstrates the superiority and practicality of TIHSNet in thermal infrared hollow drum segmentation tasks, providing an effective solution for non-destructive testing of complex architectural scenes.

[0130] 2) Results of thermal gradient attention module ablation experiments To verify the effectiveness of the proposed thermal gradient attention module in enhancing the semantic segmentation capability of thermal infrared images, this paper designs and implements a series of ablation experiments to gradually remove the key components of the TGAM module and analyze their actual contribution to the segmentation performance.

[0131] The experiment used the full model incorporating TGAM (denoted as Baseline+TGAM) as a reference, setting up the following three sets of control models: Baseline, the original encoder-decoder network architecture, without any added thermal gradient information; Baseline+GradOnly, adding thermal gradient maps as additional channel inputs at the encoder output stage, but without using an attention mechanism, i.e., the thermal gradient is only used as a set of static enhancement information; and Baseline+TGAM, fully utilizing the thermal gradient attention module, including thermal gradient extraction, attention mask generation, and skip connections to the encoder. Furthermore, to further verify the role of TGAM in the feature pyramid fusion process, this invention constructed another variant, Baseline+TGAM w / o Pyramid, which only introduces TGAM on a single-scale feature map without using a multi-scale feature fusion strategy. The experiment used a self-built thermal infrared image dataset of hollow building tiles, and the evaluation metrics were mIoU, Precision, Recall, and F1-Score. The performance of each model is shown in Table 3.

[0132] Table 3 Quantitative results of ablation experiments

[0133] The results show that without introducing an attention mechanism and using only the thermal gradient map as a static supplementary channel, the model improves across all four metrics, indicating that the thermal gradient map itself has a certain discriminative ability and can provide auxiliary information for segmentation tasks. Introducing the TGAM module without using a multi-scale pyramid structure further improves all metrics, demonstrating that TGAM can effectively mine important regions in the thermal gradient information and enhance semantic representation capabilities. When TGAM is further introduced into the multi-scale feature fusion structure, the model performance reaches its optimal state, with mIoU increasing to 0.896, and Precision, Recall, and F1-Score improving to 0.961, 0.949, and 0.944, respectively. Compared to the baseline, the overall F1-Score improves by 4.3%, indicating that the application of TGAM in the multi-scale fusion structure further enhances the model's ability to identify boundaries and local hollow regions.

[0134] Figure 13This paper demonstrates the segmentation performance of different model configurations when processing thermal infrared images with blurred boundaries. In the example images, "Ordinary" represents a typical hollow thermal image with clear boundaries, while "Edge Fog1," "Edge Fog2," and "Edge Fog3" correspond to three hollow thermal images with blurred boundaries, respectively. For the "Ordinary" sample, due to the obvious thermal gradient transition in the hollow region and the clear boundaries of the thermal image, all models can accurately identify and completely segment the hollow region, resulting in generally good segmentation results. However, in the "Edge Fog" series of samples, due to the significant thermal diffusion effect, the hollow boundaries are interfered with by background noise, and the thermal image texture is complex, posing a greater challenge to the segmentation task. The Baseline model, lacking thermal gradient information, exhibits weak perception of low-contrast regions, resulting in fragmented segmented regions, significant boundary errors, and near-unrecoverable hollow contours. The Baseline+GradOnly model, while activating some high-response regions after introducing a static thermal gradient map at the encoder output, still suffers from unstable overall predictions, blurred boundaries, and spurious responses and morphological deviations, indicating that thermal gradients, as supplementary channels, cannot effectively guide feature focusing. The Baseline+TGAM w / o Pyramid model, with its thermal gradient attention module, significantly suppresses background noise and generates more continuous segmented regions; however, due to the lack of multi-scale context fusion, boundary jumps and irregular contours persist. In contrast, the Baseline+TGAM model effectively extracts structurally complete and naturally bounded hollow masks in various blurred boundary scenarios, effectively suppressing background interference and demonstrating sensitivity and robustness to weak gradient targets, with segmented contours highly consistent with the ground truth labels. This performance improvement is attributed to the thermal gradient perception mechanism introduced by the TGAM module during multi-scale feature fusion, which dynamically emphasizes the semantic and edge information of hollow regions, thereby enhancing the model's segmentation capabilities under complex heatmap conditions.

[0135] exist Figure 14 In small hollow areas, due to their small heat capacity, the heat spreads rapidly to the surrounding structure after being heated, which easily produces the phenomenon of "thermal offset". That is, the spatial position of the target thermal response deviates from the actual structure, which weakens or even submerges the temperature difference characteristics of the hollow area in the thermal image, increasing the difficulty of detection. Figure 14Four typical thermal infrared samples are presented, including one standard-sized hollow target (Ordinary) and three smaller hollow targets with different shapes (Tiny-Defect1, Tiny-Defect2, and Tiny-Defect3). For the standard-sized hollow target, its thermal anomaly features are stable, its edges are clear, and the impact of thermal diffusion is small. All models can segment it relatively accurately, and the overall prediction results are good. However, in the three small-scale hollow cases, due to the significant effect of thermal diffusion, the target boundaries are blurred and the thermal contrast is low, which significantly increases the segmentation difficulty. The Baseline model generally suffers from serious missed detections or false detections in these samples, showing insufficient response to low-intensity targets. Although Baseline+GradOnly introduces thermal gradient information, it does not use an attention mechanism, so its enhancement effect on weak gradient regions is limited, and the segmentation effect is not significantly improved. Baseline+TGAM w / o Pyramid can capture some hollow regions at the main scale, but it still suffers from incomplete segmentation when the thermal field gradient is complex or the target size is smaller. In contrast, the complete model Baseline+TGAM significantly suppresses the response shift caused by thermal diffusion by modeling the spatial variation characteristics of thermal gradients. It can adaptively focus on the temperature difference change region at the edge of the hollow area. Even under conditions of small target size or irregular boundaries, it can still achieve accurate segmentation with clear boundaries and complete regions, demonstrating superior robustness and generalization ability.

[0136] Figure 15The visualization of segmentation results for four typical hollow samples under different ablation models is presented, including one standard-sized hollow sample (Ordinary) and three hollow samples with complex morphology and multi-scale features (Multi-Scale1, Multi-Scale2, and Multi-Scale3). From the first row of the standard-sized sample, it can be seen that all models can basically complete the identification of the target region, with clear segmentation boundaries and complete region contours. Among them, the Baseline+TGAM model performs best in edge fit, showing stronger boundary preservation and morphological reconstruction capabilities. However, in the latter three rows of samples with complex multi-scale features, the segmentation performance of different models varies significantly. The Baseline model, due to the lack of thermal gradient information, struggles to extract effective features from low-contrast heatmaps, resulting in significant false negatives. While the Baseline+GradOnly model incorporates thermal gradients as an additional channel, it lacks spatial saliency modeling, leading to frequent false positives, blurred boundaries, and poor target consistency. The Baseline+TGAM w / o Pyramid model, applying an attention mechanism at a single scale, can initially focus on regions with abrupt temperature changes. However, when dealing with targets with drastic size changes or rich structural details, its segmentation results still suffer from issues such as missing regions and incomplete contours. This indicates that the lack of multi-scale fusion limits the adaptability and generalization ability of the attention mechanism to targets of different scales. In contrast, the complete model Baseline+TGAM introduces thermal gradient attention guidance at each scale of the feature pyramid. This not only effectively enhances the expressive power of local gradient information but also improves the model's overall perception level of multi-scale structures. It significantly improves issues such as weak response and blurred edges for small targets, ultimately achieving accurate localization and boundary restoration of small-sized or complex hollow regions.

[0137] Visualization results from four representative samples show that TGAM not only performs well in scenes with clear boundaries, but also demonstrates significant advantages in challenging segmentation conditions such as blurred boundaries, small-sized voids, multi-scale or complex targets. By introducing thermal gradient information to model temperature changes and utilizing an attention mechanism to dynamically enhance salient regions, it effectively improves the accuracy and robustness of semantic segmentation.

[0138] Figure 16The visualization results of six typical samples in the multimodal hollow area classification module are presented. Each group includes six processing stages, corresponding to: (a) thermal infrared image (IRT), (b) hollow area segmentation mask (Precision), (c) multimodal fusion image (Fusion), (d) visible light image (RGB), (e) visible light boundary extraction results (Edges), and (f) hollow area spatial location classification results (Classify). As can be seen from (a) and (b), thermal infrared images can effectively reveal temperature anomaly areas, but the boundaries are blurred, making it impossible to accurately locate the spatial position of hollow areas in the tile structure. In contrast, (e) shows the tile boundary structure extracted based on the visible light image, clearly presenting the tile arrangement and providing a necessary foundation for subsequent spatial relationship modeling.

[0139] In the fusion result of (c), the thermal anomaly mask region segmented in (b) is mapped and superimposed onto the structural mesh extracted in (e), completing the alignment and fusion of infrared information and spatial geometric information. Based on this, by analyzing the relative position of the hollow region within the tile boundary frame, fine-grained location classification as shown in (f) is achieved. It can be observed that the classification results accurately classify hollow areas into "corner hollows (red)," "edge hollows (blue)," and "center hollows (yellow)," demonstrating high location classification accuracy and clear boundary determination.

[0140] Furthermore, the complex backgrounds and varied hot spot morphologies in samples 4 and 6 further validate the robustness of the fusion mechanism in diverse scenarios. Compared to traditional infrared segmentation, which can only perform coarse-grained identification of "whether there is a hollow area," the method presented in this paper achieves fine-grained classification of the spatial location of hollow areas for the first time. This effectively supports the formulation of differentiated subsequent repair strategies. For example, corner hollow areas often involve structural stability and should be replaced first, while central hollow areas are suitable for glue injection reinforcement, which has higher engineering application value.

[0141] 3) Quantification results of uncertainty Figure 17 This is a map showing the prediction results and uncertainty distribution for layered defects of different sizes and edge sharpness in infrared thermal imaging images.

[0142] Figure 17 (a) presents a case of a layered defect with clear edges. All models accurately segmented the defect, but the uncertainty was mainly concentrated in the defect edge region—a region with a significant temperature gradient, which is the main cause of prediction ambiguity. The U-Net and TransUNet models exhibited wider uncertainty distribution bands, indicating that they had lower sensitivity to defect edges. These results confirm that the boundary region is the main source of model prediction uncertainty and provide a basis for the development of uncertainty perception and detection methods.

[0143] Figure 17 (b) is a case study of detecting small-scale layered defects. These defects have weak thermal signals and are difficult to detect. Some models missed or misidentified these defects, exposing a key limitation of the models in handling subtle anomalies in practical applications. It is noteworthy that although the model failed to accurately locate the defect, its uncertainty distribution map still exhibited high uncertainty in the defect area. This reflects both the model's low confidence in predicting this area and its implicit ability to identify potential anomalies. In high-security applications such as curtain wall inspection, integrating uncertainty quantification into the inspection process can effectively improve the system's risk perception capability and overall reliability.

[0144] Figure 17 The results in (c) demonstrate that the TIHSNet model outperforms other comparative models in the hierarchical defect detection task with blurred edges, effectively segmenting defect regions with precise shapes and clear edges. The U-Net and TransUNet models suffer from distortion and blurred contours in their segmentation results. While the DSCNetPro model shows some improvement in defect boundary preservation, its performance still lags behind the TIHSNet model. The uncertainty distribution plot reveals that only the TIHSNet model maintains a consistently low level of uncertainty, concentrated only in the defect edge region, highlighting its superior prediction confidence and stability. This result also proves that, in addition to standard evaluation metrics, the uncertainty distribution plot is of significant value in assessing the detection reliability of a model.

[0145] Figure 18 The results present the pixel-level uncertainty statistical analysis results for three detection scenarios. The results show that the TIHSNet model exhibits high clustering and compactness in its uncertainty distribution across all scenarios, indicating that its uncertainty estimation results are stable and reliable. In contrast, the uncertainty distributions of the U-Net and TransUNet models show higher dispersion and a significant long-tail distribution characteristic, which is particularly pronounced in small target and blurred boundary detection scenarios, implying that these two models are at risk of overconfidence or underconfidence. The TIHSNet model has superior uncertainty localization capabilities, a significant advantage that is even more pronounced in challenging detection scenarios, enabling it to more accurately capture potential detection risks corresponding to boundary regions and small-scale defects.

[0146] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0147] In another embodiment of the present invention, a multimodal detection method system for hollow building facades is provided, comprising the steps of performing the above-described multimodal detection method for hollow building facades, and the system including: The hollow detection module is used to input thermal infrared images and visible light images into the multimodal detection model of building facade hollowness to obtain the location and classification of hollowness in the building facade. The multimodal detection model for building facade hollowness includes an encoding / decoding semantic segmentation network that integrates a thermal gradient attention module and a hollowness location auxiliary classification network guided by visible light image boundary information. The encoding / decoding semantic segmentation network is used to extract the temperature distribution features of the thermal infrared image to obtain the hollowness region segmentation mask; the hollowness location auxiliary classification network is used to extract the boundary information of the visible light image and combine it with the hollowness region segmentation mask to obtain the hollowness location classification.

[0148] In another embodiment of the present invention, a terminal device is also provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve corresponding method flows or corresponding functions; the processor described in this embodiment of the present invention can perform the following operations.

[0149] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). This computer-readable storage medium is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the above embodiments.

[0150] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0151] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0152] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0153] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A multimodal detection method for hollow areas in building facades, characterized in that, The specific steps are as follows: By inputting thermal infrared images and visible light images into the multimodal detection model for building facade voids, the location and classification of void locations on the building facade are obtained. The multimodal detection model for building facade hollowness includes an encoding / decoding semantic segmentation network that integrates a thermal gradient attention module and a hollowness location auxiliary classification network guided by visible light image boundary information. The encoding / decoding semantic segmentation network is used to extract the temperature distribution features of the thermal infrared image to obtain the hollowness region segmentation mask; the hollowness location auxiliary classification network is used to extract the boundary information of the visible light image and combine it with the hollowness region segmentation mask to obtain the hollowness location classification.

2. The multimodal detection method for hollow areas in building facades according to claim 1, characterized in that, The encoding / decoding semantic segmentation network with thermal gradient attention module includes an encoding / decoding semantic segmentation network and a thermal gradient attention module. The encoding / decoding semantic segmentation network adopts a deep convolutional encoder-decoder structure, wherein the encoder extracts features from the thermal infrared image, and the thermal gradient attention module obtains feature maps of different levels of the encoder of the encoding / decoding semantic segmentation network. The Sobel operator is used to calculate the temperature gradient magnitude in the feature map and normalizes it to obtain the thermal gradient attention mask. The decoder fuses the thermal gradient attention mask with the original features of each decoder layer to obtain the hollow region segmentation mask.

3. The multimodal detection method for hollow areas in building facades according to claim 2, characterized in that, The specific steps for obtaining the thermal gradient attention mask in the thermal gradient attention module are as follows: 1) Calculate gradient information Assuming the original thermal infrared image is T(x,y), representing a two-dimensional temperature field, where x and y represent the spatial location of the image; calculate the gradients of the feature map F of each encoder layer in the horizontal and vertical directions: Among them, G T The thermal gradient vector represents the rate of change of the temperature field in the space of the tile surface; x and y are the spatial coordinates, respectively. 2) Calculate the gradient magnitude to characterize the hollow boundary: Implementing the thermal gradient G using the Sobel operator T The first-order partial derivative yields the gradient magnitude G: 3) Constructing an attention map based on the thermal gradient magnitude map The thermal gradient magnitude is normalized to the [0,1] interval to obtain the normalized gradient map. , denoted as: Normalized gradient map Upsampled to the spatial dimensions of feature maps at various scales in the encoder, resulting in a fused gradient map. , denoted as: in, The features of the i-th layer encoder; fused gradient maps Expand to the same channel dimension as the feature map and perform channel-by-channel multiplication: in, It is the sigmoid activation function. This indicates pixel-wise and channel-wise multiplication. This is the weighted thermal gradient attention mask; thermal gradient attention mask Original features of the decoder layer The input to the decoder is obtained by fusion. .

4. The multimodal detection method for hollow areas in building facades according to claim 1, characterized in that, The specific steps of the cavity location classification network guided by visible light image boundary information to obtain cavity location classification are as follows: Visible light images Gaussian filtering followed by Canny edge detection yields edge map E: Then, Hough line segment detection is performed based on the edge map E to extract all tile bounding boxes {B}. k }: Hollow area segmentation mask is represented as Perform connected component analysis on M to extract all hollow regions. Each region Defined as a set of pixels: For each hollow area Find the tile bounding box corresponding to its position. And calculate its relative position coordinate center point: Map it to the current tile bounding box Normalized coordinates are used to generate a fused image: Based on the spatial relationship between the location of the hollow area and the tile boundary, the hollow areas are divided into corner hollow areas, edge hollow areas, and center hollow areas, as detailed below:

5. The multimodal detection method for hollow areas in building facades according to claim 1, characterized in that, The multimodal detection model for building facade hollowness also includes an uncertainty quantification method for thermal infrared hollowness segmentation model based on Bayesian probability. This method uses a Bayesian deep neural network to evaluate the uncertainty of each pixel in the hollowness region segmentation mask and outputs an uncertainty map to guide manual secondary verification.

6. The multimodal detection method for hollow areas in building facades according to claim 1, characterized in that, Thermal infrared and visible light images were acquired using a FOTRIC 288+ infrared thermal imager.

7. A multimodal detection method system for hollow areas in building facades, characterized in that, The system comprises the steps of performing the multimodal detection method for hollow building facades according to any one of claims 1 to 6, wherein the system includes: The hollow detection module is used to input thermal infrared images and visible light images into the multimodal detection model of building facade hollowness to obtain the location and classification of hollowness in the building facade. The multimodal detection model for building facade hollowness includes an encoding / decoding semantic segmentation network that integrates a thermal gradient attention module and a hollowness location auxiliary classification network guided by visible light image boundary information. The encoding / decoding semantic segmentation network is used to extract the temperature distribution features of the thermal infrared image to obtain the hollowness region segmentation mask; the hollowness location auxiliary classification network is used to extract the boundary information of the visible light image and combine it with the hollowness region segmentation mask to obtain the hollowness location classification.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal detection method for hollow building facades as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal detection method for hollow building facades as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal detection method for hollow building facades as described in any one of claims 1 to 6.

Images