Metal roof node leakage risk intelligent diagnosis method and system based on image recognition

Abstract

This application discloses an intelligent diagnostic method and system for leakage risk of metal roof nodes based on image recognition. The method includes: S1, multimodal data acquisition and preprocessing; S2, node region localization and feature extraction; S3, multi-scale visual Transformer feature fusion; S4, leakage feature identification and classification; S5, leakage risk assessment and early warning; and S6, model iteration and optimization. This application utilizes simultaneous acquisition and pixel-level registration of visible light and infrared thermal imaging to simultaneously acquire the surface texture information and temperature distribution information of nodes. By fusing global context and local detail features using a multi-scale visual Transformer network, it can effectively correlate weakly correlated features across modalities, thereby overcoming the limitations of single-modal detection. It can detect both visible surface defects and hidden problems such as internal water accumulation, reducing false alarm and missed detection rates.

Description

Technical Field

[0001] This application belongs to the field of building inspection and maintenance technology, specifically involving an intelligent diagnostic method and system for leakage risk of metal roof nodes based on image recognition. Background Technology

[0002] Metal roofing is widely used in industrial plants, large public buildings, and other fields due to its advantages such as lightweight, high strength, and quick construction. However, many of its structural nodes (such as lap joints, fasteners, and penetrating parts) are high-risk areas for leakage. Moisture intrusion not only corrodes the metal substrate and reduces its insulation performance, but long-term accumulation may also lead to structural safety hazards. Therefore, regular and effective inspection of the condition of roof nodes is crucial.

[0003] Currently, the inspection of metal roofs mainly relies on manual inspections and localized instrument assistance. Traditional manual inspection methods are highly dependent on the experience of the inspectors, resulting in inherent drawbacks such as low efficiency, limited coverage, strong subjectivity, and high risks associated with high-altitude operations. In recent years, the technology of using drones equipped with visible light or infrared thermal imagers for aerial inspection has begun to be applied, improving inspection efficiency and coverage to some extent. However, existing technical solutions still have many shortcomings:

[0004] First, the detection methods are limited and the analytical dimensions are insufficient. Relying solely on visible light images makes it difficult to detect hidden or early leaks (such as when internal water accumulation does not cause obvious surface discoloration), while relying solely on infrared thermal imaging is easily affected by factors such as environmental radiation, weather conditions, and the emissivity of roofing materials, and it is difficult to distinguish the specific causes of thermal anomalies (such as the cooling effect of leakage and structural thermal bridging).

[0005] Secondly, the level of automation and intelligence is limited. Existing methods mostly remain at the image acquisition level, with low accuracy in automatic node identification and positioning, especially for targets such as fasteners that occupy a small proportion of the image, which are prone to missed detection. Feature extraction mostly relies on traditional algorithms designed manually, which are insufficient in representing complex and subtle leakage-related features (such as micro-cracks and early corrosion), and the information from visible light and infrared modes has not been effectively integrated and complemented.

[0006] Therefore, there is an urgent need for an intelligent diagnostic method and system that can automatically, accurately, and comprehensively assess the risk of leakage at metal roof nodes and continuously optimize itself. Summary of the Invention

[0007] This application provides an intelligent diagnostic method and system for leakage risks at metal roof nodes based on image recognition, aiming to solve the problems of existing technologies having limited detection methods, insufficient analytical dimensions, and limited automation and intelligence.

[0008] Firstly, an intelligent diagnostic method for leakage risk at metal roof nodes based on image recognition, the method comprising:

[0009] S1. Multimodal data acquisition and preprocessing: The flight platform equipped with an imaging unit and a positioning and attitude determination system acquires synchronous or near-synchronous visible light and infrared thermal images of the metal roof. The acquired raw images are then subjected to geometric correction, radiometric equalization, noise filtering, and image registration processing to output the registered visible light-infrared image pairs.

[0010] S2. Node region localization and feature extraction: Receive the registered visible light-infrared image pair, automatically detect and locate the metal roof node region in the image based on a deep learning model, extract the appearance texture features and temperature distribution features from the visible light image block and infrared image block corresponding to the located node region, and output the multimodal feature vector of each node.

[0011] S3. Multi-scale Visual Transformer Feature Fusion: Construct a two-branch visual Transformer network with a global branch and a local branch; input the downsampled panoramic visible light image of the roof into the global branch to obtain a global feature map representing global semantics; input the high-resolution visible light and infrared image patches of the node regions located in S2 into the local branch to obtain a local feature map representing local details; fuse the global feature map and the local feature map through a saliency-guided feature fusion module to output the enhanced multi-scale fused features;

[0012] S4. Leakage Feature Identification and Classification: Receives the multi-scale fusion features output from S3, identifies leakage-related feature patterns through a classification network, and outputs the probability distribution and corresponding confidence score of each node belonging to the normal, suspected, or confirmed leakage categories.

[0013] S5. Leakage Risk Assessment and Early Warning: Based on the node classification results, confidence scores, node attribute characteristics and environmental correlation characteristics output by S4, the risk level of each node is calculated through the risk assessment model, and a visual report and early warning work order are generated according to the risk level.

[0014] S6. Model Iteration and Optimization: Based on on-site verification feedback data and newly added detection data, dynamically update the model parameters involved in S2, S3, S4 and S5 to achieve continuous optimization of diagnostic and evaluation performance.

[0015] Optionally, in S1, the flight platform is a multi-rotor unmanned aerial vehicle, the imaging unit includes a high-resolution visible light camera and an uncooled infrared thermal imager, both with their optical axes parallel and fixed on the same gimbal; the positioning and attitude determination system includes a GNSS receiver and an inertial measurement unit.

[0016] Optionally, in S2, the automatic detection and localization of metal roof node regions in the image based on a deep learning model specifically includes: using an improved YOLOv7 target detection network, introducing an attention module in the backbone network, and adopting a weighted bidirectional feature pyramid structure in the neck network to perform node detection on the preprocessed visible light image and output the bounding box, category, and confidence of the nodes.

[0017] The detection results are correlated with the building information model and their locations are refined to obtain the three-dimensional coordinates of the nodes in the real world.

[0018] Optionally, in step S3, the saliency-guided feature fusion module performs the following operations: channel compression and activation of the global feature map to generate a spatial saliency weight map;

[0019] The local feature map is upsampled to align with the size of the saliency weight map;

[0020] Based on the saliency weight map, the global features and the upsampled local features are weighted and fused through a gating mechanism to obtain the multi-scale fused features.

[0021] Optionally, in step S4, the classification network is a multilayer perceptron or a lightweight convolutional network; the classification network also supports inputting feature sequences of the same node at different historical moments, and extracting dynamically changing features through a time series analysis module to improve classification accuracy.

[0022] Optionally, in S5, the risk assessment model is a three-layer feedforward neural network based on the error backpropagation algorithm; the environmental correlation features include the ambient temperature and humidity at the time of collection and the cumulative rainfall in a specific period before detection; the node attribute features include node type, sealing level, service life and constituent materials.

[0023] Optionally, in step S6, the model iteration and optimization specifically includes: establishing a multimodal image and feature database of metal roof node leakage to store original detection data and on-site verification feedback data;

[0024] When new data accumulates to a preset scale or is triggered periodically, incremental learning algorithms are used to fine-tune the models in S2, S3, S4 and S5, and the production environment model is updated after the validation set evaluation is passed.

[0025] Secondly, an intelligent diagnostic system for leakage risks at metal roof nodes based on image recognition, the system comprising:

[0026] The data acquisition module is used to acquire synchronous visible light and infrared thermal images of the metal roof through the imaging unit and positioning and attitude determination system mounted on the flight platform, and output raw image data and associated pose and environmental data.

[0027] The image preprocessing and registration module is connected to the data acquisition module, receives the original image data, performs geometric correction, radiometric equalization, noise filtering and multimodal image registration processing, and outputs the registered visible light-infrared image pair;

[0028] The node detection and feature extraction module is connected to the image preprocessing and registration module. It receives the registered image pairs, locates node regions through a deep learning detection model, and extracts multimodal feature vectors from the corresponding image blocks.

[0029] The leakage identification and fusion decision module is connected to the node detection and feature extraction module. It receives the multimodal feature vector, performs feature fusion through a dual-branch multi-scale visual Transformer network, and outputs node state classification results and confidence scores through a classification network.

[0030] The risk assessment and visualization module is connected to the leakage identification and fusion decision module. It receives the classification results, confidence scores, node attributes and environmental data, calculates the risk level through the risk assessment model, and generates a three-dimensional visualized risk heat map and detection report.

[0031] The cloud platform and data management module connects the risk assessment and visualization module and other modules to manage project data, user permissions, model versions and system logs, and provides data storage, task scheduling and service interfaces.

[0032] Optionally, the system further includes an IoT access and mobile application module, which is connected to the cloud platform and data management module to receive early warning work orders, support on-site RFID tag scanning and data verification, and upload on-site verification feedback data to the cloud platform and data management module, forming a data closed loop.

[0033] Optionally, the node detection and feature extraction module is deployed on a high-performance computing server or edge computing device equipped with a GPU; the cloud platform and data management module adopts a distributed microservice architecture, and each functional module communicates as an independent microservice through an application programming interface or message queue.

[0034] Compared with the prior art, this application has at least the following beneficial effects:

[0035] This application achieves simultaneous acquisition and sub-pixel-level registration of visible light and infrared thermal imaging, enabling the system to simultaneously acquire the apparent texture information and temperature distribution information of nodes. By utilizing a multi-scale visual Transformer network to fuse global context and local detail features, it can effectively correlate weakly correlated features across modalities, thereby overcoming the limitations of single-modal detection. It can detect both visible surface defects and hidden problems such as internal water accumulation, reducing the false alarm and false alarm rates.

[0036] This application also completed the automatic detection and status classification of nodes. Furthermore, by integrating a risk assessment model with multi-dimensional features, it quantifies and classifies the structural risks that leakage may cause. The system deeply integrates the artificial intelligence recognition results with BIM / GIS data and historical maintenance records, and outputs a decision support report containing accurate three-dimensional positioning, risk level and specific maintenance suggestions. It realizes a complete workflow from problem discovery to guidance on handling, and improves the scientificity and efficiency of operation and maintenance decision-making.

[0037] This application establishes a dedicated database to collect on-site verification feedback and diverse new data, and uses incremental learning and other technologies to continuously fine-tune the detection, classification, and evaluation models. The system can continuously absorb new engineering experience and adapt to new changes brought about by different scenarios and material aging. This enables the diagnostic system to have a model continuous optimization mechanism based on incremental learning, which can become more accurate with use. It effectively overcomes the shortcomings of traditional AI models whose performance becomes fixed after deployment, and provides sustainable technical support for the long-term health management of assets. Attached Figure Description

[0038] Figure 1 A flowchart illustrating an image recognition-based intelligent diagnostic method for leakage risk at metal roof nodes, provided in one embodiment of this application;

[0039] Figure 2 This is a schematic diagram of the module connection of an intelligent diagnostic system for leakage risk of metal roof nodes based on image recognition, provided in one embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments.

[0041] The intelligent diagnostic method for leakage risk of metal roof nodes based on image recognition provided in this application includes the following steps:

[0042] S1: Multimodal Data Acquisition and Preprocessing. First, an aerial data acquisition system is constructed, consisting of a flight platform, an imaging unit, and a positioning and attitude determination system. The flight platform is preferably a multi-rotor UAV with stable hovering and automatic flight path capabilities. The imaging unit includes a high-resolution visible light digital camera and an uncooled focal plane infrared thermal imager, both rigidly connected to the same gimbal to ensure parallel optical axes, enabling synchronous or near-synchronous imaging of the same scene. The visible light camera has a resolution of no less than 20 megapixels, and the infrared thermal imager has a thermal sensitivity (NETD) of no more than 50 mK and a spatial resolution of no less than 640 × 512 pixels. The positioning and attitude determination system integrates a high-precision GNSS receiver and an inertial measurement unit (IMU) to record the latitude and longitude coordinates, altitude, and gimbal pitch, roll, and yaw angles at the moment of acquisition for each frame of image.

[0043] Before data collection, an automated flight path for the UAV is planned based on the building information model or 2D plan of the metal roof to be inspected. The flight path design must ensure comprehensive coverage of the roof, with focused and intensive imaging of known nodal areas, including but not limited to longitudinal and transverse lap joints of panels, areas with fasteners (such as self-tapping screws and rivets), eaves, ridge caps, skylight perimeter, roots of protruding components, and drainage gutters. The flight altitude is calculated based on the camera focal length and the required ground resolution, typically controlled within 5 to 20 meters above the roof. During data collection, the infrared thermal imager needs to be calibrated for radiation parameters based on ambient temperature, humidity, and the emissivity of the roof being measured. Priority is given to data collection within a time window of 2 hours after sunset to 2 hours before sunrise to reduce interference from solar radiation on the roof surface temperature distribution and enhance the contrast of temperature anomalies caused by potential leaks.

[0044] After acquiring the raw visible light and infrared image sequences, the following preprocessing steps are performed:

[0045] (1) Geometric correction: Based on the camera's internal parameters (focal length, principal point, distortion coefficient) and the pose data recorded during acquisition, the original image is subjected to distortion removal and geometric coarse correction to eliminate image deformation caused by lens optical distortion and UAV attitude changes.

[0046] (2) Brightness / Radiance Equalization: For visible light images, algorithms based on histogram matching or Retinex theory are used to correct illumination non-uniformity in order to reduce the influence of shadow and highlight areas. For infrared images, scene-based non-uniformity correction is performed, and the original radiance values ​​can be converted into temperature values ​​or normalized gray values ​​as needed;

[0047] (3) Noise filtering: Apply Gaussian filtering, median filtering or nonlocal mean filtering algorithms to suppress random noise and impulse noise in the image, while preserving edge and texture details as much as possible;

[0048] (4) Image registration: Due to the differences in physical location and imaging characteristics between visible light and infrared imaging sensors, precise spatial alignment of the two modal images acquired for the same target is required. A registration algorithm based on feature points (such as features extracted by SIFT, ORB, or deep learning methods) is adopted to calculate the homography transformation matrix and map the infrared image to the coordinate system of the visible light image to achieve high-precision spatial alignment, providing a foundation for subsequent feature fusion;

[0049] (5) Data Augmentation and Sample Labeling: To expand the training dataset and improve the robustness of the model, online or offline data augmentation operations are performed on the registered image pairs, including random horizontal and vertical flipping, rotation within a certain angle range (e.g., ±15°), scaling (e.g., 0.8 to 1.2 times), brightness and contrast fine-tuning, and adding random Gaussian noise. At the same time, professionals use annotation tools to perform bounding box or pixel-level semantic segmentation annotation on node positions and leakage anomaly areas based on the preprocessed images, forming a training sample library.

[0050] Furthermore, to optimize imaging of highly reflective metal roofs, the following targeted optimization measures were implemented during the data acquisition phase:

[0051] In visible light imaging, an adjustable linear polarization filter or circular polarization filter is installed in front of the camera lens. By rotating the filter angle, specular reflection light (glare) from a specific polarization direction from the metal surface is selectively filtered out, while allowing diffuse reflection light that characterizes surface texture and defects to pass through. This effectively suppresses highlight areas and enhances the contrast and recognizability of key features such as rust, water stains, and coating peeling in the image.

[0052] In infrared thermal imaging, given the low emissivity of metal surfaces and their susceptibility to environmental radiation, precise calibration of temperature measurements is necessary. In practice, the emissivity value of the specific coating material on the roof must first be obtained or measured on-site. This can be done using a portable emissivity meter in a typical area of ​​the roof, or by determining the typical emissivity range based on technical data provided by the coating supplier. The obtained emissivity parameters are then input into the settings menu of the infrared thermal imager. For on-site verification and compensation, a known intact area of ​​the roof with uniform temperature can be selected as a reference point. Its actual surface temperature is measured using a calibrated contact thermometer and compared with the temperature value read by the infrared thermal imager under the same emissivity settings. If a systematic deviation exists, the measurement results of the thermal imager are corrected based on this deviation, or the emissivity parameters used are modified to improve the absolute accuracy of the infrared temperature data and the consistency of temperature comparisons between different images, ensuring that subtle temperature anomalies caused by leaks can be reliably identified.

[0053] S2: Node region localization and feature extraction. This involves automatically identifying key nodes of the roof structure from the preprocessed multimodal image and extracting their visual and thermal features. The specific implementation process is as follows:

[0054] First, deep learning-based node target detection and localization are performed. Using preprocessed and enhanced high-resolution visible light images as input, a target detection neural network model based on an improved YOLOv7 architecture is employed. This improved model introduces a convolutional block attention module into the original backbone network to enhance the model's ability to focus on subtle texture features of nodes. Simultaneously, a weighted bidirectional feature pyramid structure is used in the neck network to more effectively fuse multi-scale features, improving detection performance for nodes of different sizes, especially for targets such as bolts that occupy a small portion of the image. The model is pre-trained end-to-end on a self-built dataset containing annotations for various metal roof nodes, capable of outputting bounding boxes and their confidence scores. This enables rapid and accurate identification and preliminary localization of nodes in preset categories, such as panel lap joints, fasteners (bolts, rivets), eaves edges, ridges, skylight edges, and pipe roots penetrating the roof.

[0055] To improve the robustness and localization accuracy of the model in detecting metal roof nodes, several targeted strategies were employed during the model training phase. For data augmentation, the Mosaic method was introduced, randomly scaling and cropping four training images and then stitching them together into one. This allows the model to learn multi-scale context and partially occluded node targets in a single forward propagation, enhancing its ability to recognize small fasteners and other targets. During model initialization, the K-means clustering algorithm was used to perform adaptive anchor box calculations on the labeled boxes in the training set, generating preset anchor box sizes that best match the actual size distribution of metal roof nodes, thus improving initial regression efficiency. The bounding box regression loss function adopted the Complete-IoU loss function. This function, based on the traditional IoU metric, adds a penalty for the distance between the center points of the predicted box and the ground truth box, as well as consideration of aspect ratio consistency, thereby achieving more accurate bounding box regression when nodes are closely arranged or partially occluded. To address false negatives or false positives caused by difficult samples such as "nodes in shadow" and "highly reflective nodes," a difficult sample mining strategy is implemented during training. This involves selecting samples with classification or localization difficulties based on their loss values ​​for targeted retraining in the later stages of training. Furthermore, during the data augmentation phase, these challenging scenarios are specifically simulated. For example, strong reflections are simulated by adjusting the image gamma value, or random polygonal shadow blocks are added to simulate structural occlusion, thereby improving the model's generalization performance under complex lighting and reflection conditions.

[0056] Secondly, node classification and location refinement based on prior construction knowledge are implemented. The 2D image detection results obtained in the previous step are associated with the imported Building Information Model (BIM) of the system. By solving the PnP problem or using feature matching, the coordinates of the node bounding box center point in the image coordinate system are spatially mapped and aligned with the corresponding 3D node entity coordinates in the BIM model. This process not only utilizes BIM information to verify and correct the category of the detection results (e.g., distinguishing between structural joints and decorative joints), but also improves the image positioning accuracy of nodes to the sub-pixel level and directly obtains their absolute position in the real world. For cases where the BIM model is missing or cannot be directly matched, the system will label the node as "undefined type" and record its image features for subsequent manual verification and model iteration updates.

[0057] Finally, multimodal feature extraction is performed. For each localized and classified node region, descriptive feature vectors are extracted from its registered visible light image patch and infrared image patch, respectively.

[0058] Visible light texture feature extraction: The gray-level co-occurrence matrix of the image within the nodal region is calculated, and statistical measures such as contrast, energy, homogeneity, and correlation are extracted to quantify texture roughness and uniformity. Simultaneously, edge detection is performed using Sobel or Canny operators, calculating the density and gradient direction consistency of edge pixels to assess the integrity of the sealing strip or the physical deformation of the board edge. Color histogram features can also be extracted to identify anomalous color regions, such as reddish-brown spots caused by rust.

[0059] Infrared temperature feature extraction: Statistical analysis is performed on the infrared temperature matrix (or grayscale matrix) corresponding to the node region. The average temperature, standard deviation of temperature, and the difference between the region's temperature and the average temperature of the surrounding intact roof area are calculated. Furthermore, by using adaptive threshold segmentation or clustering algorithms, local low-temperature anomaly blocks within the region are identified, and parameters such as their area, lowest temperature, and distance from key node locations (e.g., bolt centers) are extracted. These thermal characteristics are closely related to the evaporative cooling effect caused by leakage or the heat capacity effect of internal water accumulation.

[0060] Furthermore, in order to improve the comprehensive ability to identify the causes of node leakage risks, while extracting node features, the roof background features of the local area where the node is located are simultaneously collected and analyzed. Specifically, a square buffer zone of a fixed size (e.g., the side length is 2 to 3 times the size of the node frame) is extended outward from the node detection frame. This buffer zone is defined as the "background area" of the current node after excluding all other identified node areas.

[0061] The system calculates the overall texture uniformity index of the background area in the visible light image (such as the entropy value based on the gray-level co-occurrence matrix or the local binary mode variance), and the temperature distribution statistics in the infrared image (such as the average temperature and the standard deviation of the temperature gradient). By comparing the anomalous features of a single node (such as local low temperature or rust spots) with the feature baseline of its background area, the system can help determine whether the anomaly is an isolated node failure or a local manifestation of a systemic problem such as aging of the roof coating or dampness of the insulation layer. For example, if multiple nodes exhibit anomalies and the texture uniformity of their background areas decreases significantly or the overall temperature distribution is abnormal, the system can indicate a risk of large-area roof deterioration, thus providing a more comprehensive contextual basis for maintenance decisions.

[0062] S3: Multi-scale visual Transformer feature fusion. A dual-branch multi-scale visual Transformer feature extraction network is constructed, and a saliency-guided feature fusion mechanism is introduced.

[0063] The dual-branch network consists of a global branch and a local branch operating in parallel. The input to the global branch is a downsampled panoramic visible light image of the roof (e.g., scaled to 512×512 pixels). This image is divided into a sequence of fixed-size, non-overlapping image patches. Each patch is linearly projected and then a learnable positional code is added, forming a one-dimensional token sequence. This sequence is fed into a standard visual Transformer encoder, which consists of L1 layers (e.g., 6 layers), each containing a multi-head self-attention mechanism and a feedforward neural network. The global branch aims to capture high-level semantic contextual information such as the overall structural layout of the roof, lighting distribution, and large-area material consistency. Its output is a feature map representing the global environment. .

[0064] The local branch focuses on high-resolution node regions. Based on the bounding boxes of each node located in the previous step, an image region containing the node and its adjacent background is cropped from the original resolution visible light and registered infrared images (e.g., cropped and scaled to 224×224 pixels). This region image is also segmented, projected, and positionally encoded, and then fed into a separate visual Transformer encoder with L2 layers (e.g., 4 layers). The local branch is designed with smaller image patch sizes, allowing it to focus on the fine texture of the node itself, micro-cracks in the sealant, corrosion details around bolts, and subtle temperature gradient changes in the infrared image, outputting feature maps characterizing local details. ;

[0065] To further optimize network efficiency and enhance feature consistency, an interaction and sharing mechanism is introduced in the design and training of the dual-branch Transformer encoder. In one implementation, the encoders of the global and local branches share weights in the first N layers (e.g., the first two layers). These shared layers learn a general representation of basic low-level features of visible light images (such as edges and corners), thereby reducing the number of parameters and improving training efficiency. The subsequent M layers remain independent, learning high-level semantics oriented towards global context and local details, respectively. In another implementation, this method introduces a cross-branch attention module in the middle layer of the local branch encoder. Specifically, this module uses the feature token of the current layer of the local branch as the query vector and the feature token of the corresponding layer of the global branch as the key and value vectors. By calculating cross-attention, the local branch can actively retrieve and incorporate relevant contextual information from the global panoramic image (e.g., a local bolt node can obtain its relative position in the overall roof and surrounding ambient lighting information), thereby achieving more accurate semantic perception and feature alignment. Network training can employ a two-stage strategy: in the first stage, the two branches are pre-trained independently; in the second stage, after fixing the shared underlying weights or introducing cross-branch attention modules, the entire network is jointly fine-tuned to optimize the final fusion performance.

[0066] To effectively integrate global context and local details, a saliency-guided feature fusion module needs to be designed. This module uses the features output from the global branch. As a semantic guidance source. First, for A 1×1 convolution operation is applied for channel compression and integration, followed by a sigmoid activation function to generate a spatial saliency weight map S, where each pixel value represents the importance of its corresponding spatial location in the final decision. This weight map reflects the prior distribution of suspected anomalous regions in the global context. Subsequently, the local features F_l are spatially interpolated to align their dimensions with S. The fusion process is implemented through a gating mechanism. ,in This represents element-wise multiplication. This operation allows the network to dynamically adjust the contribution weights of local detail features based on global semantic saliency, enhancing detailed information in suspected leakage areas and relying on more robust global features in background areas. Finally, the fused features are refined by a context enhancement module consisting of a few convolutional layers and normalization layers, outputting enhanced multi-scale fused features. For subsequent classification and evaluation.

[0067] S4: Leakage Feature Identification and Classification. Based on the multi-scale fusion features output from S3, the status of each roof node is automatically diagnosed and classified. This process is accomplished by a lightweight classification network, preferably a multilayer perceptron containing several fully connected layers, batch normalization layers, and nonlinear activation functions (such as ReLU), with a Softmax classifier connected at the end. Fusion Features First, the data passes through a global average pooling layer, compressing it into a fixed-dimensional feature vector, which is then input into the classification network.

[0068] The classification network was trained to identify multimodal feature patterns directly or indirectly related to leakage. These feature patterns were derived from fused features and mainly include:

[0069] Infrared thermal anomaly features: The network learns to identify persistent low-temperature patterns related to leakage evaporative cooling or internal water accumulation thermal inertia. This is reflected in the activation of specific dimensions in the feature vector, corresponding to the significant negative temperature difference between the node region and the background, the continuous spatial distribution of the low-temperature region, and the anomalous variance of the temperature distribution;

[0070] Visible light appearance anomalies: The network learns to identify visual degradation of materials caused by moisture intrusion, including:

[0071] Water stains: characterized by patterns associated with specific color distributions (such as dark patches) and infiltrated textures without clear boundaries;

[0072] Rust characteristics: Characterized by activation of features associated with reddish-brown hues and localized textural roughening (oxidation products);

[0073] Coating failure manifests as patterns related to high-frequency texture features in areas of coating blistering, cracking, and peeling, as well as changes in the reflective properties of the exposed base metal.

[0074] Deformation features are constructed: the network learns to identify anomalies in the physical state of nodes. This is characterized by fusing geometric and structural information contained in the features, such as discontinuities at the edges of lap joints (corresponding to warping or detachment), linear crack features of sealant, and non-concentric ring texture anomalies around fasteners;

[0075] The output of the classification network is the probability distribution of each node belonging to each predefined category. This invention defines a three-level classification result:

[0076] Normal: No leakage-related characteristic patterns were detected, or the intensity of the characteristic patterns was below the empirical threshold.

[0077] Suspected leak: One or more weakly correlated anomalous features are detected (e.g., only slight color difference without obvious low temperature), or the feature pattern is present but with moderate confidence, which may be affected by temporary condensation, shadows or surface contamination.

[0078] Leakage confirmed: Two or more strongly correlated anomalous feature patterns are detected simultaneously (such as significant persistent low temperature areas and spatially overlapping water stain textures), and the model has a high confidence in this judgment;

[0079] The highest class probability output by the network is used as the confidence score for that classification result. This score reflects the model's confidence in making judgments based on learned feature patterns and will be used in subsequent risk assessment weight calculations.

[0080] Furthermore, in addition to multilayer perceptrons, a lightweight convolutional network with global average pooling (such as the Squeeze-and-Excitation module) can be used as the classification head. For applications that require outputting pixel-level leakage regions, the feature map can be input into a decoder network for semantic segmentation.

[0081] The implementation of the classification network is not limited to a specific architecture. In a preferred embodiment, the classification network consists of a lightweight convolutional module as the classification head. Specifically, the fused feature map output from step S3 is first input into a 1×1 convolutional layer for channel adjustment, and then a global average pooling layer is applied to compress the spatial features into channel feature vectors. To further improve classification performance, a Squeeze-and-Excitation network module can be introduced before the global average pooling layer. This module learns the weights of each channel through fully connected layers and performs recalibration, thereby adaptively emphasizing feature channels related to leakage and suppressing irrelevant channels. Finally, the feature vector is processed by one or more fully connected layers, and the final classification probability is output by the Softmax function, effectively improving classification accuracy while maintaining low computational complexity.

[0082] In another embodiment, when the application scenario requires not only node-level classification but also precise location of specific pixel regions leaking within the node image, the classification network can be replaced by a semantic segmentation decoder. This decoder takes the fused feature map output from step S3 as input and gradually recovers spatial resolution through a series of upsampling layers (e.g., transposed convolution or pixel shuffling operations). To preserve detail, the decoder can fuse high-resolution feature maps from corresponding layers during the encoding process (from different stages of the network in step S3), i.e., employing a skip connection structure. The final layer of the decoder outputs a probability map of the same size as the input node region image, where the value of each pixel represents the probability of it belonging to the "leakage" or "background" category. Training of this structure typically employs a pixel-wise cross-entropy loss function, or combines it with the Dice loss function to improve class imbalance, thereby achieving pixel-level precise segmentation of leaking regions and providing accurate spatial localization guidance for subsequent repairs.

[0083] Furthermore, to enhance the diagnostic capability for node state evolution, this method supports the use of multimodal feature sequences collected and processed from the same node at multiple different historical time points (denoted as T1, T2, ..., Tn). During the classification phase, the system first performs time alignment and normalization on the feature vectors at each time point. For the features at the current detection time Tn, in addition to their absolute feature values, the system also calculates the difference vector between them and the corresponding features at one or more previous historical time points (such as Tn-1), forming a temporal difference feature. Subsequently, the absolute features at the current time point are concatenated with the aforementioned temporal difference features along the channel dimension to form an extended temporal enhancement feature vector, which serves as a new input to the classification network. Alternatively, the absolute features at multiple time points can be arranged chronologically to form a feature sequence, which is then input into a lightweight temporal analysis module (e.g., a gated recurrent unit based on a one-dimensional convolutional or recurrent neural network) for encoding to extract dynamic features that characterize changing trends. This time-series enhancement feature or dynamic trend feature enables the classification network to identify persistent deterioration patterns such as increased low temperature in the leakage area, expansion of corrosion area, or extension of crack length. This allows for a more reliable distinction between transient anomalies caused by occasional environmental disturbances (such as temporary water accumulation or local shadows) in a single inspection and persistent anomalies reflecting structural deterioration, significantly improving the long-term accuracy of diagnosis and providing a data foundation for predictive maintenance.

[0084] S5: Leakage Risk Assessment and Early Warning. Based on the aforementioned identification and classification results, a quantitative assessment is conducted on the structural risks and maintenance urgency that may be caused by node leakage, and corresponding early warning and handling procedures are triggered, specifically including:

[0085] Risk assessment model construction and inference: Risk level assessment is implemented using a three-layer feedforward neural network model (i.e., BP neural network) based on the error backpropagation algorithm. The input layer of this model is designed to receive multi-dimensional feature vectors, specifically including:

[0086] Image recognition quantification features include the classification category output in step S4 (converted by one-hot encoding), the confidence score of the corresponding category, and the area ratio of abnormal features in the node region when the classification is "confirmed leakage" or "suspected leakage".

[0087] Environmental characteristics include: ambient temperature and relative humidity during the data collection operation; cumulative rainfall data for a specific time period prior to the detection (such as the past 24 hours or 72 hours) obtained from meteorological data services via the system interface; and optionally, ambient light intensity during the detection period.

[0088] Node attribute characteristics: Inherent attributes of nodes extracted from BIM models or asset databases, including node type (such as lap joints, through joints), sealing level of the node in the design, service life, and main constituent materials (such as steel plate type, coating type, and sealant type). These characteristics all need to be digitized or coded.

[0089] The BP neural network comprises an input layer, at least one hidden layer, and an output layer. The number of neurons in the hidden layer can be determined through trial and error or by an empirical formula based on the input dimension, with the ReLU activation function being the preferred choice. The output layer is configured according to the task: one approach is to directly output a continuous risk index value; a more efficient approach is to set three output neurons, corresponding to the predicted probabilities of "low risk," "medium risk," and "high risk," respectively, and normalize them using the Softmax function, with the final risk level determined by the highest probability. Model training requires a labeled sample set (labeled as the actual risk consequences or the risk level assessed by experts) constructed based on historical detection data and subsequent maintenance records, and supervised learning is performed by minimizing the cross-entropy loss function using the backpropagation algorithm.

[0090] The risk level output by the model will be matched with a predefined set of disposal rules to automatically generate recommended disposal measures. For example, "low risk" may correspond to "included in the routine observation plan and re-inspected after 6 months"; "medium risk" corresponds to "recommended to carry out local repairs or reinforced sealing within 3 months"; and "high risk" corresponds to "immediately arrange special maintenance and assess structural impact".

[0091] Visualization, report generation, and early warning work orders: The assessment results will be integrated with the Geographic Information System (GIS) and the 3D visualization engine. The system reads the GIS coordinate information of the building and the 3D model generated by oblique photography or BIM, maps the assessment results of each node (including location coordinates, risk level, and feature image) to the corresponding position on the 3D roof, and highlights them with different colors (such as green, yellow, and red) to form a risk heat map;

[0092] The system automatically generates structured detection and analysis reports, covering a detection overview, risk node statistics and distribution, a detailed node diagnostic list (including image evidence), overall risk assessment conclusions, and maintenance recommendations. Simultaneously, for nodes assessed as "medium risk" or higher, the system automatically creates an early warning work order in the management backend. The work order includes a node location description, risk level, recommended measures, and links to associated image evidence, and can be assigned to designated maintenance personnel or contractors. The work order status (pending, processing, reviewed) can be tracked throughout the system and synchronized with the mobile application, forming a closed-loop management system from discovery to handling.

[0093] Furthermore, to ensure continuous adaptation of the model's evaluation capabilities to the actual engineering conditions, this system incorporates a dynamic update mechanism for the risk assessment model. This mechanism operates in an incremental learning manner, triggered by the following conditions: For any node that has completed risk diagnosis, when the system confirms through a mobile application or maintenance report interface that the node has undergone corresponding on-site repair and the final treatment conclusion (such as "repair completed," "sealing replaced," "verified as a false positive") has been entered, the system will automatically construct a new training sample. The input feature vector of this sample consists of all image recognition quantization features, environmental association features, and node attribute features extracted from the node in the most recent detection. Its target label is generated based on the entered treatment conclusion, either corrected or confirmed. For example, if on-site inspection confirms a serious leak and repairs have been carried out, the label of this sample can be set to "high risk (confirmed)." The newly generated sample will be merged with the historical training sample library. The system periodically (e.g., after accumulating N new samples) initiates a model fine-tuning training, using a small learning rate, and iteratively updates the weights of the deployed risk assessment model based on the merged dataset. This process can employ algorithms such as elastic weight consolidation or gradient contextual memory to effectively retain the memory of historical data patterns while incorporating new knowledge, preventing catastrophic amnesia. Through this dynamic update mechanism, the model can continuously learn the deterioration and maintenance feedback patterns of specific buildings or even specific types of nodes, making risk prediction increasingly accurate and forming a self-optimizing closed loop.

[0094] Furthermore, to ensure the risk assessment model can fully consider and utilize the reliability information from the preceding identification stages, the classification confidence score output by S4 is input as a key feature into the risk assessment model. This confidence score, after normalization, directly constitutes an independent dimension in the risk assessment model's input vector. During model training, through learning from a large number of samples, the model can autonomously establish the intrinsic relationship between this confidence feature and the final risk level label. For example, the model will learn to analyze the following input pattern: when other input features (such as a certain low-temperature difference or slight corrosion) all suggest potential leakage, if the accompanying classification confidence score is low (e.g., below 0.7), it indicates that the image recognition model doubts this judgment and may be subject to temporary interference; therefore, the true risk label corresponding to this sample is more likely to be "low risk." Conversely, if accompanied by a high confidence score, it is more likely to correspond to "medium risk" or "high risk." In this way, the uncertainty of the preceding identification stage is explicitly encoded and transmitted to the risk assessment stage, so that the final risk assessment is not only based on the presence or absence of abnormal features, but also incorporates a quantitative consideration of the credibility of the judgment of these features. This significantly improves the robustness and rationality of risk classification decisions and reduces over-warning or under-reporting caused by misjudgment in a single step.

[0095] S6: Model iteration and optimization form the core closed loop of system self-evolution and continuous performance improvement. Its purpose is to expand and optimize the underlying data foundation through continuous feedback from engineering practice, and thereby continuously refine the diagnostic and evaluation model. Specifically, this includes:

[0096] Dynamic Construction and Enhancement of a Dedicated Dataset: A dedicated multimodal image and feature database for metal roof node leakage was established. This database not only stores the original detection task data (images, recognition results, evaluation reports), but more importantly, it systematically includes two key types of data:

[0097] On-site verification feedback data: When processing early warning work orders or during routine inspections, maintenance personnel can upload on-site verification images, text descriptions, and final confirmation status (such as "leakage confirmed - repaired", "false alarm - surface stains", "missed alarm - newly discovered cracks") of the system-identified nodes through the mobile application.

[0098] Actively collected diverse data: The system systematically retests existing buildings under different seasons and weather conditions (such as continuous monitoring before and after rain), and actively incorporates detection data of new types of buildings (such as different roofing materials and different climate zones) to proactively enrich the diversity and coverage of the dataset;

[0099] All incoming data undergoes S1-based standardization preprocessing and, together with corresponding node attributes and environmental parameters, forms a continuously growing, high-quality training sample library.

[0100] The system periodically (e.g., quarterly) or after a certain amount of new data has accumulated, initiates an incremental learning and optimization process for the model. This process targets two core models:

[0101] Optimization of the node detection and feature extraction models (S2 / S3): The YOLO and multi-scale ViT models were fine-tuned using newly added image data with precise node location and type annotations. To prevent catastrophic forgetting of old knowledge, algorithms such as elastic weight consolidation were employed, and protective penalty terms for important old parameters were introduced into the loss function.

[0102] Optimization of the leakage risk assessment model (S5): The BP neural network is retrained using new samples with final field verification labels (i.e., real risk consequences). This process can utilize the idea of ​​transfer learning, fine-tuning only the last few layers of the network to adapt to the risk association patterns that may be slightly different implied in the new data;

[0103] After optimization, the new model will be evaluated on a separate validation set (derived from historical data). Evaluation metrics include recognition accuracy, recall, and risk assessment accuracy. Only new models with performance superior to or equivalent to the current production model will be approved and deployed to the production environment to replace the old model. The system retains all historical model versions and corresponding data snapshots for performance tracking and necessary version rollbacks.

[0104] In one embodiment, an image recognition-based intelligent diagnostic system for leakage risk at metal roof nodes is provided, comprising:

[0105] The data acquisition module is responsible for performing on-site multimodal data acquisition tasks on the metal roof, serving as the data source for subsequent intelligent diagnostic processes. Its core hardware includes a flight platform, imaging unit, positioning and attitude determination system, and data transmission unit.

[0106] The hardware components of the data acquisition module include:

[0107] Flight platform: Employs an industry-grade multi-rotor drone, featuring stable flight control, sufficient payload capacity, and endurance. Models with wind and water resistance are preferred to adapt to complex outdoor weather conditions.

[0108] The imaging unit integrates a dual-spectrum (visible and thermal infrared) pod. The visible light camera is a high-resolution CMOS sensor used to capture surface texture; the infrared thermal imager is an uncooled microbolometer used to capture temperature distribution. Both are aligned on their optical axes through an internal mechanical structure and can be controlled by a gimbal to achieve synchronized pitch, roll, and yaw movements, ensuring simultaneous imaging of the same target area.

[0109] The positioning and attitude determination unit integrates a high-precision real-time dynamic carrier phase differential GNSS receiver and an inertial measurement unit to provide centimeter-level positioning accuracy and high-frequency attitude data, giving each frame of image precise geographical location and spatial attitude information.

[0110] The onboard computing and storage unit has a certain edge computing capability, which can process some data in real time and store it locally.

[0111] The software system of this module can automatically or semi-automatically generate optimized flight data collection routes based on the input target building information.

[0112] Basic flight path: Based on the building outline or imported BIM model shell, automatically generate a zigzag flight path to ensure full roof coverage;

[0113] Node-enhanced flight path: Based on a pre-defined node type library (e.g., panel seams, skylights) or node 3D coordinates extracted from imported BIM models, the system automatically overlays targeted shooting tasks onto the basic flight path for these nodes. For each identified node area, the flight control system controls the UAV to hover or fly along a specific path, adjusting the gimbal angle to ensure the node is centered in the image and captured at the optimal resolution and angle (e.g., orthogonal or a specific tilt angle). Shooting strategies (e.g., shooting distance, overlap rate) can be customized according to node type and pre-defined detection requirements.

[0114] This module integrates a meteorological data interface. Before, during, and after the task begins, the system automatically calls the application programming interface (API) of public or private meteorological data services via the mobile network to obtain accurate meteorological data for the task area within the data acquisition time window, including but not limited to ambient temperature, relative humidity, wind speed, wind direction, and recent (e.g., the past 24 hours) rainfall history. This data is encapsulated and timestamped with the acquired image data.

[0115] The acquired raw image data, pose data, and environmental data undergo initial packaging and time synchronization in the onboard unit. They are then transmitted via a secure data link (such as 4G / 5G or Wi-Fi after the mission) to the ground-based image preprocessing and registration module server or cloud storage. Data is encrypted during transmission to ensure security.

[0116] The image preprocessing and registration module receives the raw image stream and associated metadata from the data acquisition module, performs a series of automated processes, and provides high-quality, aligned multimodal image data for subsequent analysis. Its core functions include:

[0117] Image correction and enhancement: Lens distortion correction based on camera intrinsic parameters is performed on visible light images; non-uniformity correction and radiometric calibration based on a calibration plate are performed on infrared images. Adaptive histogram equalization and homomorphic filtering algorithms are used to enhance the texture details of visible light images and the temperature difference contrast of infrared images, respectively.

[0118] Multimodal image registration: For simultaneously acquired visible light and infrared image pairs, preliminary geographic alignment is first performed using their inherent pose data (from UAV GNSS / IMU). Subsequently, a fine registration algorithm based on scale-invariant feature transformation or phase correlation is used to calculate the perspective transformation matrix between the two images, achieving sub-pixel-level spatial alignment of pixels in the visible light image with corresponding temperature points in the infrared image, and outputting the registered image pair;

[0119] BIM / GIS Data Integration: This module provides standard interfaces (such as IFC and SHP file interfaces) to import BIM models or GIS outline data of buildings. By matching the geographic coordinates of images acquired by drones with the coordinate system of the BIM model, it automatically maps the roof area in the image to the corresponding 3D surface in the BIM, and provides predefined information from the BIM, such as type, number, and 3D location, for potential nodes in the image to guide and verify subsequent automatic detection.

[0120] The node detection and feature extraction module is one of the core analysis engines of intelligent diagnostics. It is deployed on high-performance computing servers or edge computing devices equipped with GPUs, and includes:

[0121] An improved YOLOv7 node detection model: This model is initialized with weights pre-trained on a large general dataset and fine-tuned on a self-built metal roof node dataset. As mentioned earlier, the improvements include the introduction of an attention mechanism and optimization of the loss function. The model takes the pre-processed visible light image as input and outputs the bounding boxes, class labels, and confidence scores of all roof nodes in the image in real time. Its inference speed needs to meet real-time or near-real-time processing requirements.

[0122] A dual-branch, multi-scale ViT feature extraction network receives node region images (visible and infrared channels after multimodal registration) localized and cropped by a detection model. The global and local branches of the Transformer encoder can be built based on the VisionTransformer or SwinTransformer architecture. This module utilizes GPUs for parallel computation acceleration, efficiently extracting deep fusion features of nodes. The module's output is a high-dimensional feature vector and corresponding feature map for each node.

[0123] The leakage identification and fusion decision module receives node features from the feature extraction module and performs the final status diagnosis.

[0124] Feature Fusion and Classification: This module embeds the saliency-guided feature fusion module described in S3 and the classification network (MLP or lightweight CNN classification head) described in S4. It maps the multi-scale fused features of nodes to a probability distribution of three categories: "normal," "suspected leakage," and "confirmed leakage," and outputs the category corresponding to the highest probability and its confidence score.

[0125] Saliency guidance mechanism: This mechanism is embedded in the feature fusion process. It uses software algorithms to calculate the spatial attention weight map and perform feature weighted fusion, ensuring that more attention is paid to potential abnormal regions prompted by global semantics when making decisions.

[0126] The risk assessment and visualization module is responsible for risk quantification and outcome presentation, specifically including:

[0127] BP Neural Network Risk Assessment Engine: This engine encapsulates a pre-trained risk assessment model. It receives node classification results, quantitative features, node attributes, and environmental data from upstream modules, calls the model to perform calculations, and outputs the risk level (low, medium, high) and specific risk index value for each node.

[0128] 3D Visualization Platform: This platform is developed based on a game engine or a professional GIS / BIM visualization library. It imports detailed 3D models of buildings and dynamically renders color-coded risk markers (such as red, yellow, and green dots) at corresponding locations on the 3D model based on the results of the risk assessment engine. The platform supports functions such as risk heatmap overlay, viewing from any angle, querying node information, and cross-sectional analysis. It also provides a one-click function to generate standard format inspection reports, with report content automatically aggregated and populated from the system database.

[0129] The IoT access and mobile application module expands the system's physical interaction and field application capabilities, and has the following functions:

[0130] RFID tag binding and data traceability: Anti-metal RFID tags are installed at key nodes on the physical roof. A mobile app with RFID read / write capabilities allows on-site personnel to scan the tags and immediately access all historical inspection records, images, risk assessment reports, and maintenance logs for that node, enabling digital asset management and full lifecycle traceability.

[0131] Mobile App: This app provides functions such as work order management, on-site verification, and data collection assistance. Inspection personnel can receive and process early warning work orders dispatched by the system, and upload verification photos, videos, or text records on-site via the app to confirm leakage conditions or close the work order after repairs are completed. The app also supports caching data in offline environments and automatically synchronizing upon network connection.

[0132] The cloud platform and data management module adopts a distributed microservice architecture to ensure the system's scalability, reliability, and security. The aforementioned functional modules (such as preprocessing services, detection services, evaluation services, and visualization services) are broken down into independent microservices that can be deployed and scaled independently, and communicate through RESTful APIs or message queues.

[0133] Multi-project collaborative management: Provides a project, user, and role permission management system, supporting multiple different construction projects to independently manage data and perform analysis tasks on the same platform;

[0134] Data dashboards and reports: Provide a graphical management dashboard that displays key performance indicators such as system operating status, detection task statistics, and risk distribution overview in real time;

[0135] Model version management and system logs: Centralized management of all AI model versions, training data, and performance metrics, with support for version rollback. The system records all user operations, service calls, and exception events, providing complete audit logs.

[0136] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. An intelligent diagnostic method for leakage risk at metal roof nodes based on image recognition, characterized in that, The method includes: S1. Multimodal data acquisition and preprocessing: The flight platform equipped with an imaging unit and a positioning and attitude determination system acquires synchronous or near-synchronous visible light and infrared thermal images of the metal roof. The acquired raw images are then subjected to geometric correction, radiometric equalization, noise filtering, and image registration processing to output the registered visible light-infrared image pairs. S2. Node region localization and feature extraction: Receive the registered visible light-infrared image pair, automatically detect and locate the metal roof node region in the image based on a deep learning model, extract the appearance texture features and temperature distribution features from the visible light image block and infrared image block corresponding to the located node region, and output the multimodal feature vector of each node. S3. Multi-scale Visual Transformer Feature Fusion: Construct a two-branch visual Transformer network with a global branch and a local branch; input the downsampled panoramic visible light image of the roof into the global branch to obtain a global feature map representing global semantics; input the high-resolution visible light and infrared image patches of the node regions located in S2 into the local branch to obtain a local feature map representing local details; fuse the global feature map and the local feature map through a saliency-guided feature fusion module to output the enhanced multi-scale fused features; S4. Leakage Feature Identification and Classification: Receives the multi-scale fusion features output from S3, identifies leakage-related feature patterns through a classification network, and outputs the probability distribution and corresponding confidence score of each node belonging to the normal, suspected, or confirmed leakage categories. S5. Leakage Risk Assessment and Early Warning: Based on the node classification results, confidence scores, node attribute characteristics and environmental correlation characteristics output by S4, the risk level of each node is calculated through the risk assessment model, and a visual report and early warning work order are generated according to the risk level. S6. Model Iteration and Optimization: Based on on-site verification feedback data and newly added detection data, dynamically update the model parameters involved in S2, S3, S4 and S5 to achieve continuous optimization of diagnostic and evaluation performance.

2. The intelligent diagnostic method for leakage risk of metal roof nodes based on image recognition according to claim 1, characterized in that, In S1, the flight platform is a multi-rotor unmanned aerial vehicle (UAV), the imaging unit includes a high-resolution visible light camera and an uncooled infrared thermal imager, both with their optical axes parallel and fixed on the same gimbal; the positioning and attitude determination system includes a GNSS receiver and an inertial measurement unit.

3. The intelligent diagnostic method for leakage risk of metal roof nodes based on image recognition according to claim 1, characterized in that, In S2, the automatic detection and localization of metal roof node regions in the image based on a deep learning model specifically includes: using an improved YOLOv7 target detection network, introducing an attention module in the backbone network, and adopting a weighted bidirectional feature pyramid structure in the neck network to perform node detection on the preprocessed visible light image and output the node's bounding box, category, and confidence score. The detection results are correlated with the building information model and their locations are refined to obtain the three-dimensional coordinates of the nodes in the real world.

4. The intelligent diagnostic method for leakage risk of metal roof nodes based on image recognition according to claim 1, characterized in that, In step S3, the saliency-guided feature fusion module performs the following operations: channel compression and activation of the global feature map to generate a spatial saliency weight map; The local feature map is upsampled to align with the size of the saliency weight map; Based on the saliency weight map, the global features and the upsampled local features are weighted and fused through a gating mechanism to obtain the multi-scale fused features.

5. The intelligent diagnostic method for leakage risk of metal roof nodes based on image recognition according to claim 1, characterized in that, In step S4, the classification network is a multilayer perceptron or a lightweight convolutional network; the classification network also supports inputting feature sequences of the same node at different historical moments, and extracting dynamically changing features through a time series analysis module to improve classification accuracy.

6. The intelligent diagnostic method for leakage risk of metal roof nodes based on image recognition according to claim 1, characterized in that, In S5, the risk assessment model is a three-layer feedforward neural network based on the error backpropagation algorithm; the environmental correlation features include the ambient temperature and humidity at the time of collection and the cumulative rainfall in a specific period before detection; the node attribute features include node type, sealing level, service life and constituent materials.

7. The intelligent diagnostic method for leakage risk of metal roof nodes based on image recognition according to claim 1, characterized in that, In S6, the model iteration and optimization specifically includes: establishing a multimodal image and feature database of metal roof node leakage, which is used to store the original detection data and on-site verification feedback data; When new data accumulates to a preset scale or is triggered periodically, incremental learning algorithms are used to fine-tune the models in S2, S3, S4 and S5, and the production environment model is updated after the validation set evaluation is passed.

8. An intelligent diagnostic system for leakage risk at metal roof nodes based on image recognition, characterized in that, The system includes: The data acquisition module is used to acquire synchronous visible light and infrared thermal images of the metal roof through the imaging unit and positioning and attitude determination system mounted on the flight platform, and output raw image data and associated pose and environmental data. The image preprocessing and registration module is connected to the data acquisition module, receives the original image data, performs geometric correction, radiometric equalization, noise filtering and multimodal image registration processing, and outputs the registered visible light-infrared image pair; The node detection and feature extraction module is connected to the image preprocessing and registration module. It receives the registered image pairs, locates node regions through a deep learning detection model, and extracts multimodal feature vectors from the corresponding image blocks. The leakage identification and fusion decision module is connected to the node detection and feature extraction module. It receives the multimodal feature vector, performs feature fusion through a dual-branch multi-scale visual Transformer network, and outputs node state classification results and confidence scores through a classification network. The risk assessment and visualization module is connected to the leakage identification and fusion decision module. It receives the classification results, confidence scores, node attributes and environmental data, calculates the risk level through the risk assessment model, and generates a three-dimensional visualized risk heat map and detection report. The cloud platform and data management module connects the risk assessment and visualization module and other modules to manage project data, user permissions, model versions and system logs, and provides data storage, task scheduling and service interfaces.

9. The intelligent diagnostic system for leakage risk of metal roof nodes based on image recognition according to claim 8, characterized in that, The system also includes an IoT access and mobile application module, which is connected to the cloud platform and data management module. This module is used to receive early warning work orders, support on-site RFID tag scanning and data verification, and upload on-site verification feedback data to the cloud platform and data management module, thus forming a data closed loop.

10. The intelligent diagnostic system for leakage risk of metal roof nodes based on image recognition according to claim 8, characterized in that, The node detection and feature extraction module is deployed on a high-performance computing server or edge computing device equipped with a GPU; the cloud platform and data management module adopts a distributed microservice architecture, and each functional module communicates as an independent microservice through an application programming interface or message queue.

Images