Building exterior wall defect detection method based on visible light and infrared image fusion

Abstract

The invention provides a building outer wall defect detection method based on visible light and infrared image fusion. The method comprises the following steps: synchronously acquiring a visible light image and an infrared thermogram, which have a spatial registration relationship, of a building outer wall; inputting the visible light image into a visual Transform semantic segmentation network for processing to obtain a multi-category semantic segmentation result; wherein the semantic segmentation network calculates a first attention weight of a standard context for each feature unit divided by the image, determines a symmetric position of the feature unit based on row and column coordinates of the feature unit, calculates a second attention weight of the feature of the symmetric position, and performs fusion; carrying out geometric correction on a multi-category semantic segmentation result by utilizing a line segment detection algorithm to generate a pure wall body region mask; and according to the spatial registration relationship, mapping the pure wall body region mask to the infrared thermogram, extracting temperature data of the pure wall body region, determining a thermal anomaly threshold value based on the multi-peak characteristics of a temperature histogram, and marking a thermal anomaly defect region in the pure wall body region according to the threshold value.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and architectural engineering technology, specifically relating to a method for detecting defects in building exterior walls based on the fusion of visible light and infrared images. Background Technology

[0002] With the increasing demand for operation and maintenance and energy-saving diagnosis of existing urban buildings, hidden wall defects such as damaged insulation layers, delamination, structural thermal bridging, and abnormal heat conduction caused by wall hollowing or internal cavities are becoming increasingly prominent. These defects typically manifest as abnormal local heat flow and temperature distribution within the wall, collectively referred to as thermal anomaly defects. These defects not only increase building energy consumption but may also lead to safety hazards such as wall durability issues and exterior wall peeling. Therefore, conducting efficient and accurate detection of exterior wall thermal anomaly defects is of great significance for building energy efficiency assessment and defect remediation.

[0003] Infrared thermal imaging is commonly used in engineering to inspect building facades, offering advantages such as non-contact operation and the ability to reflect temperature field distribution. However, infrared thermal images suffer from relatively low resolution and insufficient texture detail. Furthermore, real-world facade scenes often include non-wall targets such as windows, doors, trees, air conditioner units, and obstructions. The significant differences in thermal response and reflection characteristics of different materials make it easy for temperature disturbances unrelated to defects to appear in the thermal images. Additionally, the background of the thermal image fluctuates significantly due to factors such as shooting angle, environmental radiation, wind speed, and sunlight. Therefore, relying solely on thermal images often makes it difficult to reliably distinguish between actual wall thermal anomalies and non-wall interference.

[0004] To improve detection reliability, one type of existing technology directly segments abnormal regions based on thermal imaging temperature thresholds or statistical distributions. Although simple to implement, it is prone to misjudging windows or equipment areas as abnormal under conditions of multiple targets, multiple materials, and occlusion. Another type of method attempts to combine visible light information to limit the target region or manually remove it according to rules. However, there are differences in scale and perspective between visible light and thermal imaging, and the boundaries of regular structures such as facade doors and windows are complex at oblique perspectives, resulting in coarse limiting results. It is still difficult to obtain accurate pure wall detection areas, thus affecting the accuracy and stability of subsequent thermal anomaly identification.

[0005] With the development of deep learning and multimodal vision technology, visible light facade semantic segmentation and structural geometric constraints have provided new ideas for thermal image anomaly detection. However, existing solutions still have problems such as inaccurate wall area extraction, difficulty in maintaining regular boundaries of doors and windows, and insufficient removal of non-wall target interference in complex facade environments. Summary of the Invention

[0006] To address the shortcomings and deficiencies of existing technologies, such as the susceptibility of infrared thermal imaging to interference from non-wall targets, inaccurate extraction of pure wall areas, and insufficient accuracy and stability in identifying thermal anomaly defects, this invention provides a method and system for detecting building exterior wall defects based on the fusion of visible light and infrared images, aiming to achieve accurate and reliable detection of thermal anomaly defects in building exterior walls.

[0007] The scheme first uses a UAV-borne multimodal sensor to simultaneously acquire visible light images and infrared thermal images of the building's exterior walls. The sensor includes a visible light camera and an infrared camera, placed horizontally side-by-side with horizontally aligned axes and largely overlapping fields of view to ensure the two images have a pre-defined spatial registration relationship. Then, the visible light images are input into a visual Transformer semantic segmentation network with a built-in symmetry perception attention module. This module calculates the attention weight of each feature unit in the image segmentation with its standard context, then determines the mirror position based on the row and column coordinates of the feature unit and calculates the corresponding attention weight. The two weights are fused using a learnable parameter with a value range of [0,1], thereby enhancing the building facade symmetry structure modeling capability and outputting multi-category semantic segmentation results including walls, doors, windows, and other facade elements. Next... The segmentation result is geometrically corrected based on the line segment detection algorithm. First, the visible light image is enhanced by Gaussian blurring twice. Then, morphological opening operation and connected component analysis are performed on the door and window areas. Rectangular door and window boundaries are formed by line segment matching and combination, and finally a geometrically consistent pure wall area mask is generated. Then, the pure wall mask is mapped to the infrared thermal image according to the spatial registration relationship. The corresponding temperature data is extracted and 3×3 neighborhood local smoothing is performed. After statistical analysis, a temperature distribution histogram (temperature bandwidth of 0.1℃) is drawn, and the extreme temperature samples of the highest 5% are removed to avoid peak misjudgment. Finally, the thermal anomaly detection threshold is determined based on the multi-peak feature of the histogram. When the histogram has a bimodal distribution, the local minimum point between the two main peaks is taken as the threshold. When it has a unimodal distribution, it is determined that there is no significant thermal anomaly. Thus, the thermal anomaly defect area is accurately marked in the pure wall area.

[0008] This invention, through the collaborative design of multimodal image fusion, symmetric perception semantic segmentation, and geometric correction, fully leverages the advantages of rich texture details in visible light images and temperature perception in infrared thermal imaging. It effectively eliminates irrelevant background interference such as doors and windows, significantly improving the extraction accuracy of pure wall areas and the accuracy and robustness of thermal anomaly defect identification. It can accurately detect thermal anomaly defects caused by insulation layer damage or debonding, structural thermal bridging, leakage, dampness, and hollowness, providing strong support for building energy conservation assessment and defect treatment.

[0009] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0010] A method for detecting defects in building exterior walls based on visible light and infrared image fusion includes:

[0011] Visible light images and infrared thermal images of the building's exterior walls are acquired simultaneously, and the visible light images and infrared thermal images have a spatial registration relationship.

[0012] The visible light image is input into a visual Transformer semantic segmentation network for processing to obtain multi-category semantic segmentation results including walls, doors, windows, and other facade elements. The semantic segmentation network has a built-in symmetry perception attention module, which is configured to: calculate a first attention weight with the standard context for each feature unit in the image segmentation, determine its symmetrical position based on the row and column coordinates of the feature unit, calculate a second attention weight with the symmetrical position feature, and fuse the first attention weight and the second attention weight.

[0013] The multi-category semantic segmentation results are geometrically corrected using a line segment detection algorithm to generate a pure wall region mask;

[0014] Based on the spatial registration relationship, the pure wall area mask is mapped onto the infrared thermal image, and the temperature data of the pure wall area is extracted.

[0015] Statistical analysis was performed on the temperature data of the pure wall area. Based on the multi-peak characteristics of the temperature histogram, the thermal anomaly threshold was determined, and thermal anomaly defect areas were marked in the pure wall area according to the threshold.

[0016] Furthermore, visible light images and infrared thermal images of the building's exterior walls are simultaneously acquired using a multimodal sensor mounted on an unmanned aerial vehicle. The multimodal sensor includes a visible light camera and an infrared camera, which are placed horizontally side by side with their axes aligned horizontally, and the overlapping area of ​​their fields of view covers the exterior wall area of ​​the building to be measured.

[0017] Furthermore, the specific configuration of the symmetry-aware attention module includes:

[0018] The symmetrical position is determined by calculating the mirror index using the row and column coordinates of the feature unit, and the key vector and value vector of the symmetrical position are extracted from the feature sequence based on the mirror index.

[0019] The first attention weight is calculated based on the query vector and key vector of the feature unit;

[0020] The second attention weight is calculated based on the query vector of the feature unit and the key vector at the symmetrical position;

[0021] The first attention weight and the second attention weight are fused by a learnable parameter γ, where the value of γ is in the range of [0,1], and is used to control the proportion of attention the model pays to local contextual relationships and symmetric structural relationships.

[0022] Furthermore, the geometric correction based on the line segment detection algorithm specifically includes:

[0023] The visible light image is subjected to denoising and line structure enhancement processing for line segment detection;

[0024] Morphological opening operations are performed on the door and window regions in the multi-class semantic segmentation results to eliminate noise and fine adhesion. Then, connected component analysis is performed to obtain independent door and window instance masks, and the minimum bounding rectangle of each instance mask is calculated as the geometric anchor edge.

[0025] Candidate straight lines in visible light images are extracted using a line segment detection algorithm. The candidate straight lines are matched one by one with the geometric anchor edges. Line segments with a minimum distance greater than 20 pixels or a difference in direction angle / slope greater than 0.1 are eliminated. The line segments with the minimum distance and the maximum length are retained on each anchor edge.

[0026] When all four anchor edges match valid line segments, they are combined to form a rectangular door / window boundary; otherwise, the door / window instance is not modified.

[0027] The rectangular door and window boundaries are used to perform mask correction on the multi-category semantic segmentation results to generate a geometrically consistent pure wall area mask.

[0028] Furthermore, the specific method for denoising and enhancing the line structure of the visible light image is as follows: two Gaussian blurs are applied. The Gaussian kernel for the first Gaussian blur is 5×5, and the Gaussian kernel for the second Gaussian blur is 3×3. The standard deviations of the two Gaussian blurs are the same.

[0029] Furthermore, the extraction of temperature data from the pure wall area includes:

[0030] The infrared thermal image is represented as a temperature matrix. This temperature matrix is ​​matched with a pure wall area mask according to pixel position. The temperature values ​​of the pixels in the wall area are retained, and the pixels in the non-wall area are assigned null values ​​to form the wall area temperature matrix.

[0031] The temperature matrix of the wall area is smoothed locally, and the temperature value of each pixel is updated to the average temperature of the effective pixels in its 3×3 neighborhood to reduce the influence of random noise and local outliers.

[0032] When calculating the smoothed temperature data, the temperature bandwidth of the temperature distribution histogram is 0.1℃, and the highest 5% of extreme temperature samples are removed to avoid misjudging peak values.

[0033] Furthermore, the specific method for determining the thermal anomaly threshold based on the multi-peak characteristics of the temperature histogram is as follows:

[0034] When the temperature histogram shows a bimodal distribution, the local minimum point between the two main peaks is taken as the thermal anomaly threshold, and the area where the temperature exceeds the threshold is classified as a thermal anomaly defect area.

[0035] When the temperature histogram shows a single-peak distribution, it is determined that there is no significant thermal anomaly.

[0036] Furthermore, a building exterior wall defect detection system based on visible light and infrared image fusion includes:

[0037] The multimodal acquisition module is configured to simultaneously acquire visible light images and infrared thermal images of the building's exterior walls using a UAV-borne multimodal sensor. The multimodal sensor includes a visible light camera and an infrared camera arranged horizontally side by side with overlapping fields of view. The visible light images and infrared thermal images have a spatial registration relationship.

[0038] The semantic segmentation module is configured to receive the visible light image, perform semantic segmentation through a visual Transformer semantic segmentation network with a built-in symmetry-aware attention module, and obtain multi-category semantic segmentation results including walls, doors, windows and other facade elements; the symmetry-aware attention module is configured to: for each feature unit of the image segmentation, calculate its first attention weight with the standard context, determine its symmetrical position based on the row and column coordinates of the feature unit, calculate its second attention weight with the symmetrical position feature, and fuse the first attention weight and the second attention weight;

[0039] The geometric correction module is configured to perform geometric correction on the multi-category semantic segmentation results using a line segment detection algorithm to generate a geometrically consistent pure wall region mask.

[0040] The cross-modal mapping module is configured to map the pure wall region mask onto the infrared thermal image according to the spatial registration relationship, and extract the temperature data of the pure wall region;

[0041] The thermal anomaly detection module is configured to perform statistical analysis on the temperature data of the pure wall area, determine the thermal anomaly threshold based on the multi-peak characteristics of the temperature histogram, and mark the thermal anomaly defect area in the pure wall area according to the threshold.

[0042] And a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described above.

[0043] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0044] Compared with the prior art, the present invention and its preferred embodiments have at least the following beneficial effects:

[0045] By using a multimodal fusion design of visible light and infrared images, the advantages of the rich texture details of visible light images and the accurate temperature perception of infrared thermal images are fully utilized, effectively making up for the limitations of single-modal detection in complex scenarios and providing more comprehensive information support for thermal anomaly defect detection.

[0046] By leveraging the visual Transformer semantic segmentation network with its built-in symmetry perception attention module and integrating standard contextual attention with symmetry position attention, the modeling capability for symmetrical building facade structures is significantly enhanced, greatly improving the semantic segmentation accuracy of facade elements such as walls, doors, and windows, and laying a solid foundation for subsequent extraction of pure wall areas.

[0047] By using a line segment detection algorithm to geometrically correct the semantic segmentation results, the boundary deviations of components such as doors and windows can be effectively corrected, generating a geometrically consistent pure wall area mask. This successfully eliminates interference from non-wall targets such as doors, windows, and obstructions, ensuring that thermal anomaly detection focuses only on the real wall area and avoiding misjudgments caused by irrelevant backgrounds.

[0048] The threshold for thermal anomaly detection is adaptively determined based on the multi-peak characteristics of the temperature histogram. Combined with the smoothing of temperature data and the removal of extreme values, the stability and reliability of thermal anomaly judgment are improved. It can accurately identify various thermal anomaly defects such as insulation layer damage, structural thermal bridging, leakage and dampness, and hollowness, providing accurate and reliable technical support for building exterior wall operation and maintenance diagnosis and energy-saving assessment.

[0049] The detection method is highly automated, requiring minimal human intervention, and is adaptable to complex building facade scenarios. It exhibits good robustness and practicality, effectively addressing issues such as inaccurate extraction of pure wall areas and insufficient accuracy in thermal anomaly identification in existing technologies. Attached Figure Description

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0051] Figure 1 This is a schematic diagram of the overall process of the method in an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of the overall framework for automatic detection of thermal anomalies and defects according to an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of the visual Transformer facade semantic segmentation network and line processing integrated framework based on a symmetric perception multi-head self-attention module according to an embodiment of the present invention.

[0054] Figure 4 This is a flowchart illustrating the structure of the symmetric sensing multi-head self-attention module in an embodiment of the present invention. Detailed Implementation

[0055] To make the features and advantages of the present invention more apparent and understandable, specific embodiments are described below in detail:

[0056] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0057] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0058] The purpose of this invention is to provide a method for detecting defects in building exterior walls based on the fusion of visible light and infrared images. This method helps to eliminate background interference unrelated to the wall and detect thermal anomalies only in the pure wall area. Furthermore, by improving and enhancing the facade symmetry and geometric structure perception module, the method improves the extraction accuracy of the pure wall area and the reliability of thermal anomaly defect identification.

[0059] The implementation of this invention includes: designing a UAV-borne multimodal sensor combination method using a visible light camera and an infrared camera to acquire visible light images and infrared thermal images of building exterior walls; inputting the visible light images into a visual Transformer facade semantic segmentation network containing a symmetric perception multi-head self-attention module to perform semantic segmentation on the images, obtaining semantic segmentation results for multiple categories of facade elements in the entire image, including wall, door and window, and other facade element regions; performing geometric correction on the semantic segmentation results of multiple categories of facade elements based on a line segment detection algorithm to obtain a pure wall region mask; mapping the wall mask to the infrared thermal image according to the spatial registration relationship between the visible light image and the infrared thermal image, and extracting the temperature distribution of the corresponding pure wall region; performing statistical analysis on the temperature distribution, determining the thermal anomaly detection threshold based on the multi-peak features of the temperature histogram, and marking the thermal anomaly region within the pure wall region.

[0060] This approach leverages the unique imaging advantages of both visible light and infrared cameras. Compared to traditional single-modal detection, this method significantly reduces the influence of irrelevant backgrounds such as doors and windows by accurately extracting a pure wall mask from the visible light image and mapping it to the infrared image. This improves the accuracy and robustness of thermal anomaly area identification, thereby enabling precise and reliable detection of thermal anomaly defect areas on building exterior walls.

[0061] like Figure 1As shown, the specific process of implementing this embodiment of the invention refers to the following steps:

[0062] Step S1: Design a method for combining visible light and infrared cameras on a UAV-borne multimodal sensor to acquire visible light images and infrared thermal images of building exterior walls.

[0063] Step S2: Input the visible light image into the visual Transformer facade semantic segmentation network containing a symmetry-aware multi-head self-attention module, perform semantic segmentation on the image, and obtain the semantic segmentation results of multi-class facade elements for the entire image, which includes walls, doors and windows and other facade element areas.

[0064] Step S3: Based on the line segment detection algorithm, perform geometric correction on the semantic segmentation results of multi-category facade elements and obtain a pure wall area mask;

[0065] Step S4: Based on the spatial registration relationship between the visible light image and the infrared thermal image, map the pure wall mask onto the infrared thermal image and extract the temperature distribution of the corresponding pure wall area;

[0066] Step S5: Perform statistical analysis on the temperature distribution, determine the thermal anomaly detection threshold based on the multi-peak characteristics of the temperature histogram, and mark the thermal anomaly defect area within the pure wall area.

[0067] The thermal anomaly defect area described in this invention refers to an abnormal area in the exterior wall and insulation system where the local temperature distribution deviates significantly from the surrounding normal wall surface due to structural, material aging, dampness, or construction quality issues. Thermal anomaly defect areas include, but are not limited to: abnormal heat loss due to insulation layer damage or delamination, abnormal heat transfer caused by structural thermal bridges, abnormal temperature caused by leakage / dampness, and abnormal heat conduction caused by hollow areas or internal cavities.

[0068] The integrated framework for automatic detection of thermal anomalies and defects in this invention is as follows: Figure 2 As shown, this framework uses multimodal image fusion as its core. It achieves accurate extraction of pure wall areas through visible light images and detects thermal anomalies by combining them with infrared thermal images. The specific process is as follows: First, visible light images and infrared thermal images of the building's exterior walls are acquired simultaneously using a UAV-borne multimodal sensor. Semantic segmentation and edge precision processing are performed on the visible light images to obtain segmentation results that include walls, doors, windows, and other facade elements. Then, a line segmentation algorithm is used to geometrically correct the segmentation results to extract pure wall areas. Finally, the registration relationship between the visible light and infrared images is used to map the pure wall areas onto the infrared thermal images. Thermal anomaly detection is achieved through threshold segmentation. The entire process requires no manual intervention, achieving automated and accurate detection.

[0069] As a preferred embodiment, the multimodal sensor setup in step S1 includes: one visible light camera and one infrared camera, placed horizontally side-by-side with their fields of view essentially overlapping; the visible light camera has a resolution of 1920x1080; the infrared camera has a resolution of 1280x1024; the axes of the two sensors are aligned horizontally, and their positions are adjusted to maximize the overlap of their fields of view, ensuring that the fields of view cover the target area to be measured; the visible light camera is used to acquire overall information about the wall surface for subsequent image segmentation of the pure wall area by the algorithm, and the infrared camera is used to detect thermal anomaly defects in the wall.

[0070] As a preferred implementation, in this embodiment, the spatial registration of the visible light image and the infrared thermal image is established in the following way: common building facade feature points (such as door and window corners, wall edge intersections) in the two modal images are selected, and feature points are extracted and matched using SIFT or ORB feature point detection algorithms. After removing mismatched points, the mapping relationship is solved by homography matrix or affine transformation model to achieve pixel-level precise alignment of the two images.

[0071] The semantic segmentation and line processing integrated framework of this invention is as follows: Figure 3 As shown, the framework consists of two parts: a semantic segmentation network and a line processing module. The semantic segmentation network adopts a visual Transformer architecture, which first divides, flattens, and maps the input visible light image into blocks, then embeds the features and inputs them into the encoder. After multi-layer encoding processing, the decoder, which includes symmetry-aware multi-head self-attention blocks, generates category-specific semantic features and finally outputs semantic segmentation results for multiple categories such as walls, windows, and doors. The line processing module extracts and filters lines from the semantic segmentation results, providing accurate straight line segments for subsequent geometric correction. The two work together to achieve accurate segmentation and boundary optimization of facade elements.

[0072] As a preferred embodiment, step S2 specifically includes the following steps:

[0073] Step S21: Divide the visible light image into several fixed-size image blocks, convert each image block into a corresponding feature vector through linear mapping, and superimpose its two-dimensional positional encoding to form the initial feature sequence input to the encoder. The dimension of the feature vector is set to... It is used to characterize the semantic and texture features of image patches.

[0074] Step S22: Input the feature sequence sequentially into a visual coding network consisting of multiple coding layers. Each coding layer performs self-attention calculation and feedforward network calculation sequentially according to the residual structure, forming a feature sequence passed between layers; where, the first... Layer coding calculation includes:

[0075]

[0076]

[0077] in, It is a multi-head self-attention module. The representation layer normalization operation, MLP(⋅) is a feedforward neural network, and They represent the first Layer and First The feature sequence of the layer, For the first The intermediate features obtained by adding the layer's self-attention output to the input residual.

[0078] In the self-attention module, for each image patch, its corresponding mirror position is determined based on its row and column coordinates in the image. A mirror index is constructed, and key vectors and value vectors corresponding to the mirror positions are selected from the feature sequence to simultaneously calculate standard attention weights and mirror attention weights. This self-attention module receives query vectors. Key vector AND value vector All of their dimensions are The standard attention weights are calculated as follows:

[0079]

[0080] in, The query matrix is ​​obtained from the mapping of the current sequence. The key matrix is ​​obtained by mapping the same sequence. The value matrix is ​​obtained by mapping the same sequence. This is a scaling factor used to stabilize the numerical range of attention weights.

[0081] To enhance the modeling capability of symmetrical building facades, the mirror position is calculated based on the row and column coordinates of image patches, a mirror index is constructed, and the mirror key vector is extracted from the feature sequence. With mirror value vector Thus, the mirror attention weights are obtained, and the mirror attention weights are calculated as follows:

[0082]

[0083] in, This is the mirror key matrix obtained by adjusting the key vector matrix based on the mirror index. This is the corresponding mirror value matrix; the meanings of the other parameters are the same as those in standard attention.

[0084] Standard attention and mirror attention are compared using learnable parameters. (Its value range is) The two processes are then fused to obtain the final attention output:

[0085]

[0086] in, This is used to control the proportion of attention the model pays to local contextual relationships and symmetric structural relationships. It can be set independently per layer or shared across multiple layers to achieve adaptive modeling of structural symmetry at different scales. This is to replace the final attention result output by the original self-attention module. Finally, the final attention is used to replace the standard self-attention, in order to enhance the ability to represent the symmetry of the facade structure.

[0087] The structure and flow of the self-attention module are as follows: Figure 4 As shown, for each image patch, the corresponding mirror position is determined based on its row and column coordinates in the image, a mirror index is constructed, and the key vector and value vector corresponding to the mirror position are selected from the feature sequence to simultaneously calculate the standard attention weight and the mirror attention weight.

[0088] Step S23: Input the encoded features output from step S22 into a semantic decoding network containing multiple decoders. Generate class-specific semantic features using the pre-defined cross-attention relationship between the category query vector and the encoded features. The decoding layer is calculated according to the following residual structure:

[0089]

[0090]

[0091] In this step, For decoder number The input features of the layer This represents the cross-attention mechanism. The remaining symbols are consistent with those defined in step S22. The decoding process is used to recover the semantic feature representation corresponding to the original image resolution.

[0092] Step S24: Perform pixel-level classification prediction on the decoded features using the classification head to obtain the semantic segmentation results of multi-class facade elements for the entire image, which includes wall, door and window and other facade element areas, and provide the segmentation results to step S3 for subsequent line segment geometry correction.

[0093] As a preferred embodiment, step S3 specifically includes the following steps:

[0094] Step S31: Extract candidate straight lines from the visible light image using the Line Segment Detection Operator (LSD);

[0095] The input visible light image is first subjected to denoising and line structure enhancement processing. A double Gaussian blurring technique is preferred to suppress noise and short pseudo-lines: the first blurring uses a 5×5 Gaussian kernel for strong blurring, and the second blurring uses a 3×3 Gaussian kernel for light blurring. The standard deviation of both blurring operations can be the same fixed value (e.g., σ=5). The LSD operator is then applied to the preprocessed image to obtain a set of candidate line segments.

[0096]

[0097] Where G is the set of candidate line segments obtained from LSD line segment detection. It is the first The starting coordinates of the candidate line segments, It is the first The endpoint coordinates of each candidate line segment are used to characterize the potential geometric boundaries of building facade components.

[0098] Step S32: Select and combine line segments based on their direction and relative position to form rectangular door and window boundaries;

[0099] In the semantic segmentation results of multi-class facade elements in the entire image, facade component regions such as doors and windows are preserved, and noise and fine adhesion are eliminated through morphological opening operations. The optimal erosion kernel and dilation kernel are both 5×5. Subsequently, connected component analysis is performed to decompose the door and window regions into B independent instance masks.

[0100]

[0101] Where F is the set of door and window instances obtained by connected component analysis from the door and window regions in the semantic segmentation results of multi-class facade elements in the entire image. It is the first The coordinates of the top left corner of each door and window instance (or a diagonal point of the smallest bounding box). It is the first The coordinates of the bottom right corner of each door and window instance (or the other opposite corner of the smallest outer frame).

[0102] Next, calculate the minimum bounding rectangle for each instance mask:

[0103]

[0104] in These are the ordinates of the upper and lower boundaries of the circumscribed rectangle, respectively. These are the x-coordinates of the left and right boundaries of the circumscribed rectangle. Its four sides are used as geometric anchor edges, and the candidate line segment set G is matched against each anchor edge one by one: Line segments are discarded when the minimum distance from the corresponding anchor edge is greater than a threshold δ (preferably about 20 pixels) or the angle / slope difference between the line segment direction and the anchor edge direction is greater than a threshold θ (preferably about 0.1). The retained line segments are assigned to one of the four sides of the anchor frame according to their direction, and only the line segment with the smallest distance and the largest length is retained as a boundary candidate on each side. When all four anchor edges of an instance are successfully matched with line segments, the four line segments are integrated to form a new circumscribed quadrilateral / rectangular door / window boundary; if there are missing edges, the instance is not corrected to avoid incorrect constraints.

[0105] Step S33: Apply mask correction to the semantic segmentation results of multi-class facade elements in the entire image using rectangular boundaries to generate a geometrically consistent pure wall mask.

[0106] For valid instances that meet the integration conditions in step S32, the original predicted regions are replaced with the integrated rectangular door and window boundaries in the semantic segmentation result mask of the multi-class facade elements in the entire image: first, the original regions of the door and window instances are set to the wall category, and then the pixels inside the rectangular boundaries obtained in S32 are assigned to the corresponding door and window categories, so that the boundaries of the door and window components satisfy the geometric priors of straight lines and quadrilaterals; after the replacement is completed for all valid instances, the corrected mask is obtained. ,Will Pixels marked as wall type are set to 1, and all other types are set to 0, generating a binary mask for pure wall areas. It serves as a geometrically consistent wall mask for subsequent wall extraction and infrared thermal anomaly detection.

[0107] As a preferred embodiment, step S4 specifically includes the following steps:

[0108] Step S41: Based on the pre-registration relationship between the visible light image and the infrared thermal image, map the wall mask in the visible light image domain to the infrared thermal image coordinate system, as follows:

[0109] The wall mask obtained through steps S2 and S3 is represented as a mask matrix. ,in For the first Line number The wall label values ​​of the column pixels. Since the visible light image and the infrared thermal image have been spatially registered in advance in this embodiment, the mask matrix can be used. It directly aligns with and applies to the pixel coordinate system of the infrared thermal image to achieve the corresponding positioning of the wall area in the infrared domain.

[0110] Step S42: Extract the temperature data corresponding to the wall mask from the infrared thermal image to form a temperature matrix of the wall area, as follows:

[0111] Infrared thermal images are represented as temperature matrices. ,in This represents the temperature value of the corresponding pixel. The temperature matrix... With mask matrix By matching pixel positions, the wall temperature matrix is ​​obtained: for those satisfying... Pixels belonging to the wall area retain their temperature values. ; Assign null values ​​to pixels in non-wall areas to extract temperature distribution only in wall areas, avoiding interference from non-target components such as windows and trees on temperature statistics.

[0112] Step S43: Perform local smoothing on the temperature matrix, updating the temperature value of each pixel to the average temperature of the effective pixels in its 3×3 neighborhood, in order to reduce the influence of random noise and local outliers, as follows:

[0113] To reduce the interference of pixel-level extreme temperatures and noise on subsequent threshold determination, local mean filtering is applied to the wall temperature matrix to obtain the target temperature matrix. Each target pixel Defined as centered on this pixel The average value of non-empty temperature values ​​within the neighborhood is smoothed only within the wall area, thereby improving the continuity and statistical stability of the wall temperature distribution.

[0114] Step S44: Statistically analyze the smoothed temperature data and plot a histogram of temperature distribution in the wall area to provide a basis for subsequent determination of thermal anomaly thresholds, as detailed below:

[0115] For the target temperature matrix The effective temperature values ​​of the wall are statistically analyzed, and a temperature distribution histogram of the wall is constructed according to temperature ranges. The temperature bandwidth of the histogram is preferably 0.1℃ to finely characterize the temperature frequency distribution characteristics of the wall area. At the same time, in order to avoid false peaks caused by a small number of extreme high-temperature pixels, the global maximum temperature can be identified during the statistical analysis, and the highest 5% of temperature samples can be treated as extreme values. This ensures that the peak-valley structure of the histogram more realistically reflects the normal / abnormal thermal distribution of the wall material, providing a reliable basis for the threshold determination in step S5.

[0116] As a preferred implementation, in this embodiment, the extreme temperature samples are identified as follows: the smoothed wall area temperature data are sorted in descending order, and the top 5% of high-temperature samples are taken as extreme values; the processing method is to directly remove these samples and draw a temperature distribution histogram based only on the remaining valid temperature data.

[0117] As a preferred embodiment, in step S5, a histogram is drawn for the temperature distribution of the pure wall area in the infrared thermal image. When the histogram has a bimodal distribution, the local minimum point between the two main peaks is taken as the abnormal threshold. Threshold segmentation is performed in this way. Areas with temperatures exceeding this threshold are classified as abnormal, and areas with temperatures below this threshold are considered normal. Thus, abnormal temperature areas are segmented and located. When the histogram has a unimodal distribution, it is determined that there are no significant thermal abnormality defects.

[0118] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0119] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0120] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0121] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0122] This invention is not limited to the preferred embodiment described above. Anyone inspired by this invention can derive various other forms of building exterior wall defect detection methods based on the fusion of visible light and infrared images. All equivalent variations and modifications made within the scope of the claims of this invention shall fall within the scope of this invention.

Claims

1. A method for detecting defects in building exterior walls based on the fusion of visible light and infrared images, characterized in that, include: Visible light images and infrared thermal images of the building's exterior walls are acquired simultaneously, and the visible light images and infrared thermal images have a spatial registration relationship. The visible light image is input into a visual Transformer semantic segmentation network for processing to obtain multi-category semantic segmentation results including walls, doors, windows, and other facade elements. The semantic segmentation network has a built-in symmetry perception attention module, which is configured to: calculate a first attention weight with the standard context for each feature unit in the image segmentation, determine its symmetrical position based on the row and column coordinates of the feature unit, calculate a second attention weight with the symmetrical position feature, and fuse the first attention weight and the second attention weight. The multi-category semantic segmentation results are geometrically corrected using a line segment detection algorithm to generate a pure wall region mask; Based on the spatial registration relationship, the pure wall area mask is mapped onto the infrared thermal image, and the temperature data of the pure wall area is extracted. Statistical analysis was performed on the temperature data of the pure wall area. Based on the multi-peak characteristics of the temperature histogram, the thermal anomaly threshold was determined, and thermal anomaly defect areas were marked in the pure wall area according to the threshold.

2. The method for detecting defects in building exterior walls based on visible light and infrared image fusion according to claim 1, characterized in that: Visible light images and infrared thermal images of the building's exterior walls are simultaneously acquired by a UAV-borne multimodal sensor. The multimodal sensor includes a visible light camera and an infrared camera, which are placed horizontally side by side with their axes aligned horizontally. The overlapping area of ​​their fields of view covers the exterior wall area of ​​the building to be measured.

3. The method for detecting defects in building exterior walls based on visible light and infrared image fusion according to claim 1, characterized in that: The specific configuration of the symmetry perception attention module includes: The symmetrical position is determined by calculating the mirror index using the row and column coordinates of the feature unit, and the key vector and value vector of the symmetrical position are extracted from the feature sequence based on the mirror index. The first attention weight is calculated based on the query vector and key vector of the feature unit; The second attention weight is calculated based on the query vector of the feature unit and the key vector at the symmetrical position; The first attention weight and the second attention weight are fused by a learnable parameter γ, where the value of γ is in the range of [0,1], and is used to control the proportion of attention the model pays to local contextual relationships and symmetric structural relationships.

4. The method for detecting defects in building exterior walls based on visible light and infrared image fusion according to claim 1, characterized in that: The geometric correction based on the line segment detection algorithm specifically includes: The visible light image is subjected to denoising and line structure enhancement processing for line segment detection; Morphological opening operations are performed on the door and window regions in the multi-class semantic segmentation results to eliminate noise and fine adhesion. Then, connected component analysis is performed to obtain independent door and window instance masks, and the minimum bounding rectangle of each instance mask is calculated as the geometric anchor edge. Candidate straight lines in visible light images are extracted using a line segment detection algorithm. The candidate straight lines are matched one by one with the geometric anchor edges. Line segments with a minimum distance greater than 20 pixels or a difference in direction angle / slope greater than 0.1 are eliminated. The line segments with the minimum distance and the maximum length are retained on each anchor edge. When all four anchor edges match valid line segments, they are combined to form a rectangular door / window boundary; otherwise, the door / window instance is not modified. The rectangular door and window boundaries are used to perform mask correction on the multi-category semantic segmentation results to generate a geometrically consistent pure wall area mask.

5. The method for detecting defects in building exterior walls based on visible light and infrared image fusion according to claim 4, characterized in that: The specific method for denoising and enhancing the line structure of the visible light image is as follows: two Gaussian blurs are applied. The Gaussian kernel for the first Gaussian blur is 5×5, and the Gaussian kernel for the second Gaussian blur is 3×3. The standard deviations of the two Gaussian blurs are the same.

6. The method for detecting defects in building exterior walls based on visible light and infrared image fusion according to claim 1, characterized in that: The extraction of temperature data from the pure wall area includes: The infrared thermal image is represented as a temperature matrix. This temperature matrix is ​​matched with a pure wall area mask according to pixel position. The temperature values ​​of the pixels in the wall area are retained, and the pixels in the non-wall area are assigned null values ​​to form the wall area temperature matrix. The temperature matrix of the wall area is locally smoothed, and the temperature value of each pixel is updated to the average temperature of the effective pixels in its 3×3 neighborhood to reduce the influence of random noise and local outliers. When calculating the smoothed temperature data, the temperature bandwidth of the temperature distribution histogram is 0.1℃, and the highest 5% of extreme temperature samples are removed to avoid misjudging peak values.

7. The method for detecting defects in building exterior walls based on visible light and infrared image fusion according to claim 1, characterized in that: The specific method for determining the thermal anomaly threshold based on the multi-peak features of the temperature histogram is as follows: When the temperature histogram shows a bimodal distribution, the local minimum point between the two main peaks is taken as the thermal anomaly threshold, and the area where the temperature exceeds the threshold is classified as a thermal anomaly defect area. When the temperature histogram shows a single-peak distribution, it is determined that there is no significant thermal anomaly.

8. A building exterior wall defect detection system based on visible light and infrared image fusion, characterized in that, include: The multimodal acquisition module is configured to simultaneously acquire visible light images and infrared thermal images of the building's exterior walls using a UAV-borne multimodal sensor. The multimodal sensor includes a visible light camera and an infrared camera arranged horizontally side by side with overlapping fields of view. The visible light images and infrared thermal images have a spatial registration relationship. The semantic segmentation module is configured to receive the visible light image, perform semantic segmentation through a visual Transformer semantic segmentation network with a built-in symmetry-aware attention module, and obtain multi-category semantic segmentation results including walls, doors, windows and other facade elements; the symmetry-aware attention module is configured to: for each feature unit of the image segmentation, calculate its first attention weight with the standard context, determine its symmetrical position based on the row and column coordinates of the feature unit, calculate its second attention weight with the symmetrical position feature, and fuse the first attention weight and the second attention weight; The geometry correction module is configured to perform geometric correction on the multi-category semantic segmentation results using a line segment detection algorithm to generate a geometrically consistent pure wall region mask. The cross-modal mapping module is configured to map the pure wall region mask onto the infrared thermal image according to the spatial registration relationship, and extract the temperature data of the pure wall region; The thermal anomaly detection module is configured to perform statistical analysis on the temperature data of the pure wall area, determine the thermal anomaly threshold based on the multi-peak characteristics of the temperature histogram, and mark the thermal anomaly defect area in the pure wall area according to the threshold.

9. A computer device, characterized in that, It includes a processor and a memory, wherein a computer program is stored in the memory, and when the processor executes the computer program, it implements the method of any one of claims 1-7.

10. A non-transitory computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method of any one of claims 1-7.

Images