Real-time monitoring method and system for highway infrastructure based on UAV Internet of Things

Abstract

The present invention discloses a real-time monitoring method and system for highway infrastructure based on the UAV Internet of Things, which relates to the field of highway monitoring technology, including: using a group of drones equipped with laser radars, high-resolution cameras, infrared thermal imagers and vibration sensors to synchronously collect geometric, texture, temperature and vibration data of highway infrastructure, and combining the environmental data of weather stations and traffic cameras to construct a multimodal data stream. The deviation caused by the heterogeneity of drone posture and sensors is eliminated by a spatiotemporal alignment algorithm, and the interference of rain and fog is eliminated by using dark channel priors, atmospheric scattering models and adversarial generative networks, and then the geometric parameters of cracks are extracted. Based on a graph neural network, an association map is constructed, and node features and edge weights are integrated. The remaining probability of fatigue life of the infrastructure is calculated and the risk level is divided in combination with the damage index. Finally, a predictive maintenance decision is generated to achieve a full-link closed loop from data collection, processing to decision-making.

Description

Technical Field

[0001] The present invention relates to the field of highway monitoring technology, and in particular to a method and system for real-time monitoring of highway infrastructure based on an unmanned aerial vehicle (UAV) Internet of Things (IoT). Background Art

[0002] The long-term service life of highway infrastructure is susceptible to vehicle loads, environmental erosion, and natural disasters. Traditional manual inspections are subject to low efficiency, high costs, and limited coverage. Existing monitoring technologies often rely on single sensors (such as cameras or vibration sensors), making multimodal data fusion difficult. They are also severely affected by environmental interference such as rain, fog, and sunlight, resulting in insufficient crack detection accuracy.

[0003] Existing methods for assessing infrastructure health status are mostly based on static thresholds. They lack real-time correlation analysis of dynamic environmental parameters (such as traffic loads and weather changes), making it difficult to predict fatigue life and risk levels. While the rise of drone-based IoT technology has improved data collection capabilities, technical bottlenecks remain in the spatiotemporal alignment, noise removal, and semantic association of multi-source heterogeneous data, hindering the intelligent level of infrastructure health management.

[0004] Therefore, a real-time monitoring method and system for highway infrastructure based on UAV Internet of Things is urgently needed to solve the above-mentioned problems. Summary of the Invention

[0005] To address the aforementioned technical issues, a real-time monitoring method and system for highway infrastructure based on the UAV Internet of Things (IoT) is provided. This technical solution addresses the problem that existing methods proposed in the aforementioned background art primarily assess infrastructure health based on static thresholds, lacking real-time correlation analysis of dynamic environmental parameters (such as traffic loads and meteorological changes), making it difficult to predict fatigue life and risk levels. While the rise of UAV IoT technology has improved data collection capabilities, technical bottlenecks remain in the spatiotemporal alignment, noise removal, and semantic correlation of multi-source heterogeneous data, hindering the intelligent level of infrastructure health management.

[0006] In order to achieve the above objects, the technical solution adopted by the present invention is:

[0007] The real-time monitoring method of highway infrastructure based on UAV IoT includes:

[0008] S1. Obtain sensor data transmitted by the drone swarm, including lidar data, high-resolution image data, infrared image data, and vehicle load vibration data;

[0009] S2. Obtain environmental data from weather stations and traffic cameras, perform spatiotemporal alignment of lidar data, high-resolution image data, infrared image data, and vehicle load vibration data, remove rain and fog interference data, and extract crack geometry parameters;

[0010] S3. Construct an infrastructure-crack-environment correlation map based on lidar data, high-resolution image data, infrared image data, vehicle load vibration data, and environmental data.

[0011] S4. Based on the infrastructure-crack-environment correlation map, obtain the remaining probability of infrastructure fatigue life and infrastructure risk level;

[0012] S5. Generate predictive maintenance decisions using the remaining probability of infrastructure fatigue life and infrastructure risk level.

[0013] In an optional embodiment, the spatiotemporal alignment of the lidar data, the high-resolution image data, the infrared image data, and the vehicle load vibration data specifically includes:

[0014] Based on lidar data, high-resolution image data, infrared image data, and vehicle load vibration data, the standard timestamp corresponding to the high-precision clock module built into the drone swarm and the GPS timing system is obtained;

[0015] LiDAR data, high-resolution image data, infrared image data, and vehicle load vibration data aligned based on standard timestamps;

[0016] A cross-modal spatiotemporal registration network is constructed to perform spatiotemporal alignment of lidar data, high-resolution image data, infrared image data, and vehicle load vibration data using lidar data as the spatiotemporal reference.

[0017] In an optional embodiment, removing rain and fog interference data specifically includes:

[0018] Using the dark channel prior algorithm, the continuous RGB image frames in the high-resolution image data are histogram matched to eliminate the uneven brightness caused by rain and fog;

[0019] Based on the atmospheric scattering model, the transmittance map of continuous RGB images in the high-resolution image data after eliminating the uneven brightness caused by rain and fog is obtained, and the contrast of the RGB image is restored;

[0020] Set the temperature gradient threshold, synchronously calibrate the dynamic range of the infrared temperature map in the infrared image data, and eliminate abnormal temperature fluctuations caused by rain and fog;

[0021] Obtain environmental depth information from the lidar data to assist in defogging the infrared temperature map in the infrared image data; obtain the point cloud density in the lidar data to determine the corresponding infrared temperature map in the infrared image data.

[0022] Spatially related positions, and eliminating rain and fog scattering noise in the infrared temperature map corresponding to the spatially related positions;

[0023] The continuous frame RGB images in the high-resolution image data and the infrared temperature map in the infrared image data are sequentially input into the multimodal U-Net, and the enhanced continuous frame RGB images and infrared temperature map are output;

[0024] An equivalent infrared temperature map under clear weather conditions is generated through a generative adversarial network, and the temperature maps corresponding to locations with abnormal temperature fluctuations caused by rain and fog are removed from the enhanced infrared temperature map.

[0025] In an optional embodiment, extracting the fracture geometric parameters specifically includes:

[0026] Use the CBAM attention module to fuse the texture features of continuous frame RGB images in high-resolution image data and the thermal features of infrared temperature maps in infrared image data to generate crack candidate areas;

[0027] PointNet++ is used to extract geometric features from the lidar data, including normal vectors and curvature, and the crack candidate area is corrected to obtain the crack candidate correction area.

[0028] Mask R-CNN is used to perform instance segmentation on the crack candidate correction area to obtain the precise boundary of the crack;

[0029] Morphological operations are performed on the precise boundaries of the cracks using closing operations and erosion and dilation to close small crack gaps and remove isolated noise points. The crack area corresponding to the remaining precise boundaries of the cracks is used as the standard crack area.

[0030] Align the RGB image in the high-resolution image data corresponding to the crack standard area with the point cloud data in the lidar data, and generate a crack sparse depth map through a stereo matching algorithm;

[0031] Obtaining gradient information in an infrared temperature map in the infrared image data, and determining the first crack depth in combination with a crack standard area;

[0032] In the crack standard area, the pixel-level length is calculated along the main direction of the crack to obtain the crack standard length;

[0033] Obtain the standard crack width corresponding to the standard crack area through the minimum circumscribed rectangle;

[0034] Obtaining the point cloud density in the laser radar data corresponding to the crack standard area to obtain the second crack depth;

[0035] The average of the first crack depth and the second crack depth is obtained as the standard crack depth.

[0036] In an optional embodiment, the construction of an infrastructure-crack-environment correlation map based on lidar data, high-resolution image data, infrared image data, vehicle load vibration data, and environmental data specifically includes:

[0037] Define the graph structure, including infrastructure nodes, crack nodes, and environment nodes;

[0038] Determine the standard area of ​​cracks based on lidar data and high-resolution image data;

[0039] Obtaining the crack geometric parameters corresponding to the standard crack area, including the standard crack depth, standard crack length and standard crack width;

[0040] Determine the crack nodes based on the standard crack depth, standard crack length and standard crack width;

[0041] Obtain natural and traffic environment information, including weather station temperature, rainfall, wind speed, traffic volume, vehicle axle weight, and vehicle speed;

[0042] Determine environmental nodes based on weather station temperature, rainfall, wind speed, traffic volume, vehicle axle weight and vehicle speed;

[0043] Capture physical entities, including bridge piers, pavement slabs, guardrails, and drainage systems, based on LiDAR data and high-resolution imagery data;

[0044] Obtain the vibration frequency corresponding to the physical entity, and determine the infrastructure nodes by combining the bridge piers, pavement slabs, guardrails and drainage systems with the vibration frequency corresponding to the physical entity;

[0045] The infrastructure nodes, crack nodes and environment nodes are connected by pre-set connection rules to obtain the infrastructure-crack-environment correlation map.

[0046] In an optional embodiment, obtaining the infrastructure fatigue life remaining probability and infrastructure risk level based on the infrastructure-crack-environment correlation map specifically includes:

[0047] Based on the infrastructure-crack-environment correlation map, the infrastructure node feature vector, crack node feature vector and environment node vector are obtained;

[0048] Obtain the edge weights between infrastructure nodes and crack nodes, the edge weights between environment nodes and crack nodes, and the edge weights between environment nodes and infrastructure nodes respectively;

[0049] The edge weights are sequentially input into the graph neural network to obtain the remaining probability of infrastructure fatigue life corresponding to the infrastructure nodes;

[0050] Determine the damage index corresponding to the infrastructure node based on the standard crack depth, standard crack length, and standard crack width corresponding to the crack node connected to the infrastructure node;

[0051] Determine the infrastructure risk level through the damage index corresponding to the infrastructure node;

[0052] The calculation formula for the remaining probability of fatigue life of the infrastructure corresponding to the infrastructure node is:

[0053]

[0054] Where, is the remaining probability of infrastructure fatigue life corresponding to the i-th infrastructure node, is the Sigmoid function, is the L-th layer weight matrix, is the node feature matrix corresponding to the i-th infrastructure node, is the bias term of the Lth layer;

[0055] The calculation formula of the damage index corresponding to the infrastructure node is:

[0056]

[0057] Where, is the damage index corresponding to the i-th infrastructure node, is the edge weight between infrastructure node i and crack node m, Environment Node The edge weight of the target area corresponding to the crack node m at time t, Environment Node The edge weight of the target area corresponding to infrastructure node i at time t, are the standard crack depth, standard crack length and standard crack width corresponding to the crack node m, is the mean of the maximum crack standard depth, maximum crack standard length and maximum crack standard width allowed for infrastructure node i, is the total number of crack nodes corresponding to the i-th infrastructure node.

[0058] Furthermore, a real-time monitoring system for highway infrastructure based on the UAV Internet of Things is proposed, which is used to implement any of the above monitoring methods, including:

[0059] An acquisition module, which is used to acquire lidar data, high-resolution image data, infrared image data, and vehicle load vibration data, as well as environmental data from weather stations and traffic cameras;

[0060] a data processing module for performing spatiotemporal alignment on the lidar data, the high-resolution image data, the infrared image data, and the vehicle load vibration data, for removing rain and fog interference data from the lidar data, the high-resolution image data, and the infrared image data, and for extracting crack geometric parameters, including standard crack depth, standard crack length, and standard crack width;

[0061] a risk assessment module, the risk assessment module being used to construct an infrastructure-crack-environment correlation map based on lidar data, high-resolution image data, infrared image data, vehicle load vibration data, and environmental data, and to obtain the infrastructure fatigue life remaining probability and infrastructure risk level based on the infrastructure-crack-environment correlation map;

[0062] A maintenance decision module, wherein the maintenance decision module is used to generate a predictive maintenance decision using the remaining probability of fatigue life of the infrastructure and the infrastructure risk level;

[0063] The display module is used to show the real-time monitoring process and results of highway infrastructure based on the drone Internet of Things to users.

[0064] In an optional embodiment, the acquisition module includes:

[0065] a first acquisition unit, the first acquisition unit being used to acquire lidar data, high-resolution image data, infrared image data, and vehicle load vibration data;

[0066] The second acquisition unit is used to obtain environmental data from weather stations and traffic cameras.

[0067] In an optional embodiment, the data processing module includes:

[0068] a spatiotemporal alignment unit, configured to perform spatiotemporal alignment on the lidar data, the high-resolution image data, the infrared image data, and the vehicle load vibration data;

[0069] A denoising unit, configured to remove rain and fog interference data from the lidar data, the high-resolution image data, and the infrared image data;

[0070] The extraction unit is used to extract fracture geometric parameters, including standard fracture depth, standard fracture length and standard fracture width.

[0071] In an optional embodiment, the risk assessment module includes:

[0072] A correlation map construction unit, configured to construct an infrastructure-crack-environment correlation map based on lidar data, high-resolution image data, infrared image data, vehicle load vibration data, and environmental data;

[0073] A data evaluation unit is used to obtain the remaining probability of infrastructure fatigue life and the infrastructure risk level based on the infrastructure-crack-environment correlation map.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] This proposal proposes a real-time highway infrastructure monitoring method based on the UAV Internet of Things. Through multimodal data fusion and spatiotemporal alignment, it achieves precise synchronization of data from drone swarms equipped with lidar, high-resolution imagery, infrared thermal imaging, and vibration sensors. This method is less susceptible to the strong heterogeneity and large spatiotemporal deviations of multi-source data, significantly improving the integrity and consistency of data collection and providing a reliable foundation for subsequent analysis.

[0076] This proposal proposes a real-time monitoring method for highway infrastructure based on the UAV Internet of Things. Through the CBAM attention mechanism and graph neural network (GNN), it constructs an infrastructure-crack-environment correlation map, realizes the dynamic correlation analysis between crack parameters (depth, length, width) and dynamic environmental parameters (traffic flow, temperature, vibration frequency), and improves the comprehensiveness and accuracy of crack risk assessment.

[0077] This proposal proposes a real-time monitoring method for highway infrastructure based on the UAV Internet of Things. By using a generative adversarial network (GAN) and a multimodal U-Net, it eliminates rain and fog interference and enhances the quality of infrared and visible light images. This addresses the issue of low data availability in inclement weather and enables all-weather, high-precision crack detection, especially in low-visibility environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 This is a flow chart of the real-time monitoring method for highway infrastructure based on UAV Internet of Things proposed by the present invention;

[0079] Figure 2 This is a flow chart for removing rain and fog interference data in the present invention;

[0080] Figure 3 This is a flow chart for extracting crack geometric parameters in the present invention;

[0081] Figure 4 This is a system framework diagram of the real-time monitoring system for highway infrastructure based on drone Internet of Things proposed in the present invention. DETAILED DESCRIPTION

[0082] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are merely examples, and those skilled in the art may conceive of other obvious variations.

[0083] Reference Figure 1 - Figure 4 As shown in FIG, the real-time monitoring method of highway infrastructure based on UAV Internet of Things includes:

[0084] S1. Obtain sensor data transmitted by the drone swarm, including lidar data, high-resolution image data, infrared image data, and vehicle load vibration data;

[0085] The LiDAR records LiDAR data at 100Hz, the high-resolution camera captures high-resolution image data at 200Hz, the infrared camera outputs infrared image data at 60Hz, and the vibration sensor samples vehicle load vibration data at 500Hz.

[0086] S2. Obtain environmental data from weather stations and traffic cameras, perform spatiotemporal alignment of lidar data, high-resolution image data, infrared image data, and vehicle load vibration data, remove rain and fog interference data, and extract crack geometry parameters;

[0087] S3. Construct an infrastructure-crack-environment correlation map based on lidar data, high-resolution image data, infrared image data, vehicle load vibration data, and environmental data.

[0088] S4. Based on the infrastructure-crack-environment correlation map, obtain the remaining probability of infrastructure fatigue life and infrastructure risk level;

[0089] S5. Generate predictive maintenance decisions using the remaining probability of infrastructure fatigue life and infrastructure risk level.

[0090] Furthermore, the spatiotemporal alignment of the lidar data, high-resolution image data, infrared image data, and vehicle load vibration data specifically includes:

[0091] Based on lidar data, high-resolution image data, infrared image data, and vehicle load vibration data, the standard timestamp corresponding to the high-precision clock module built into the drone swarm and the GPS timing system is obtained;

[0092] LiDAR data, high-resolution image data, infrared image data, and vehicle load vibration data aligned based on standard timestamps;

[0093] A cross-modal spatiotemporal registration network is constructed to perform spatiotemporal alignment of lidar data, high-resolution image data, infrared image data, and vehicle load vibration data using lidar data as the spatiotemporal reference.

[0094] In order to achieve spatiotemporal alignment of lidar data, high-resolution image data, infrared image data and vehicle load vibration data, a complete process from raw data acquisition to multimodal collaborative registration needs to be constructed: first, the four types of data are synchronously collected through the high-precision clock module built into the drone and the GPS timing system to ensure that the timestamp error of the lidar data, visible light image, infrared thermal imaging frame and vibration signal is less than 1 millisecond. The lidar data records environmental depth information at a frequency of 100Hz, the high-resolution camera captures road surface images at 200Hz, the infrared camera outputs temperature distribution at 60Hz, and the vibration sensor samples the dynamic load caused by the vehicle axle weight at 500Hz; then, based on the timestamp-aligned quaternary data group, a motion compensation algorithm is used to eliminate the spatial deviation caused by the change of the drone's flight attitude. Specifically, the attitude angles (pitch, roll, yaw) calculated by the IMU are used to perform a rigid body transformation on the lidar data, and the point clouds of adjacent frames are aligned to the global coordinate system. At the same time, the pixel displacement field between the visible light image and the infrared image is calculated by the optical flow method to compensate for the vibration caused by the drone in the infrared thermal imaging. Motion compensation is performed on the generated blurred areas, and a spatial mapping relationship is then established based on the LiDAR data. SIFT feature points are extracted from the visible light image and infrared image through feature matching. The RANSAC algorithm is used to eliminate mismatched points and then the perspective transformation matrix is ​​calculated. The 2D image coordinates are mapped to the 3D LiDAR data space, ensuring that the position error of the crack area in the image and point cloud is less than 3 cm. On this basis, a temporal alignment algorithm is designed to address the heterogeneity of vibration signals and multimodal visual data. The sampling frequency of the vibration signal (500Hz) is matched to the frame rate of the LiDAR data (100Hz) through cubic spline interpolation. At the same time, the vehicle axle load mutation event is used as the time anchor point to synchronize the load peak in the vibration signal with the road surface area scanned by the LiDAR, ensuring that the vibration frequency domain energy spectrum and the road surface texture features of the LiDAR data strictly correspond on the time axis. Finally, a cross-modal spatiotemporal registration network is constructed. Using the LiDAR data as the spatiotemporal reference, a graph convolutional network (GCN) is used to establish a correlation map between the frequency domain features of the vibration signal and the thermal features of the image. The attention mechanism is used to dynamically weight different modes.

[0095] Furthermore, the removal of rain and fog interference data specifically includes:

[0096] Using the dark channel prior algorithm, the continuous RGB image frames in the high-resolution image data are histogram matched to eliminate the uneven brightness caused by rain and fog;

[0097] Based on the atmospheric scattering model, the transmittance map of continuous RGB images in the high-resolution image data after eliminating the uneven brightness caused by rain and fog is obtained, and the contrast of the RGB image is restored;

[0098] Set the temperature gradient threshold, synchronously calibrate the dynamic range of the infrared temperature map in the infrared image data, and eliminate abnormal temperature fluctuations caused by rain and fog;

[0099] Obtain environmental depth information from the lidar data to assist in defogging the infrared temperature map in the infrared image data; obtain the point cloud density in the lidar data to determine the corresponding infrared temperature map in the infrared image data.

[0100] Spatially related positions, and eliminating rain and fog scattering noise in the infrared temperature map corresponding to the spatially related positions;

[0101] The continuous frame RGB images in the high-resolution image data and the infrared temperature map in the infrared image data are sequentially input into the multimodal U-Net, and the enhanced continuous frame RGB images and infrared temperature map are output;

[0102] An equivalent infrared temperature map under clear weather conditions is generated through a generative adversarial network, and the temperature maps corresponding to locations with abnormal temperature fluctuations caused by rain and fog are removed from the enhanced infrared temperature map.

[0103] Specifically, multimodal data fusion and physical model-driven enhancement technology are used to eliminate the interference of rain, fog, and temperature anomalies on infrared and visible light data: first, a dark channel prior algorithm is used to perform histogram matching on continuous frame RGB images, and the contrast attenuated by rain and fog is restored by estimating the atmospheric light curtain and transmittance map (based on the atmospheric scattering model I=J·t+A·(1−t)). The temperature gradient threshold ∇Tth is simultaneously set to eliminate abnormal temperature fluctuations (regions with ΔT>∇Tth) caused by rain and fog scattering in the infrared temperature map. The spatial correlation constraint is constructed by combining the LiDAR point cloud density ρ and depth d: the spatial resolution of the infrared temperature map is adjusted by weighting the point cloud density, and the geometric projection matrix T is used to calculate the spatial resolution of the infrared temperature map. The spatial coordinates of the LiDAR point cloud are mapped to an infrared temperature map to eliminate noise in non-physically relevant areas. The preprocessed RGB and infrared data are fed into a multimodal U-Net. The encoder extracts cross-modal features (such as RGB texture and infrared thermal information), and the decoder generates an enhanced fog-free image (I^RGB) and a temperature map (T^IR). Finally, a CycleGAN is used to generate the equivalent infrared temperature for clear weather (Tclear). Anomalous regions are replaced using an adversarial loss function (LGAN) = E[logD(T^IR)] + E[log(1−D(Tclear))], resulting in the final output, a rain- and fog-free RGB image (I^RGBclean) and a temperature map (T^IRclear). This process, through the dual enhancement of a physical model (atmospheric scattering) and a data-driven approach (U-Net + GAN), significantly improves the robustness of subsequent crack detection.

[0104] Furthermore, the extraction of crack geometric parameters specifically includes:

[0105] Use the CBAM attention module to fuse the texture features of continuous frame RGB images in high-resolution image data and the thermal features of infrared temperature maps in infrared image data to generate crack candidate areas;

[0106] PointNet++ is used to extract geometric features from the lidar data, including normal vectors and curvature, and the crack candidate area is corrected to obtain the crack candidate correction area.

[0107] Mask R-CNN is used to perform instance segmentation on the crack candidate correction area to obtain the precise boundary of the crack;

[0108] Morphological operations are performed on the precise boundaries of the cracks using closing operations and erosion and dilation to close small crack gaps and remove isolated noise points. The crack area corresponding to the remaining precise boundaries of the cracks is used as the standard crack area.

[0109] Align the RGB image in the high-resolution image data corresponding to the crack standard area with the point cloud data in the lidar data, and generate a crack sparse depth map through a stereo matching algorithm;

[0110] Obtaining gradient information in an infrared temperature map in the infrared image data, and determining the first crack depth in combination with a crack standard area;

[0111] In the crack standard area, the pixel-level length is calculated along the main direction of the crack to obtain the crack standard length;

[0112] Obtain the standard crack width corresponding to the standard crack area through the minimum circumscribed rectangle;

[0113] Obtaining the point cloud density in the laser radar data corresponding to the crack standard area to obtain the second crack depth;

[0114] The average of the first crack depth and the second crack depth is obtained as the standard crack depth.

[0115] Specifically, this step first uses the CBAM attention module to synchronously fuse the texture features of the high-resolution RGB image with the thermal features of the infrared temperature map to generate initial crack candidate regions. This candidate region is then optimized using the normal vectors and curvature features of the LiDAR point cloud extracted using PointNet++. Mask R-CNN instance segmentation is then used to obtain precise boundaries. Closing operations and morphological operations are then used to eliminate noise interference and determine the standard crack region. A sparse depth map is generated based on stereo matching of LiDAR and RGB data. The multi-source information of infrared temperature gradients and point cloud density is integrated to calculate and average the crack depth. Length and width are then obtained through pixel projection and geometric analysis, ultimately outputting the standard crack depth, length, and width. This process significantly improves the accuracy and robustness of crack parameter extraction through multimodal complementarity (RGB texture + infrared thermal + LiDAR geometry) and multi-stage optimization (candidate correction → morphological denoising → multi-source depth fusion), providing reliable data support for infrastructure health assessments.

[0116] Furthermore, the infrastructure-crack-environment correlation map is constructed based on the lidar data, high-resolution image data, infrared image data, vehicle load vibration data, and environmental data, specifically including:

[0117] Define the graph structure, including infrastructure nodes, crack nodes, and environment nodes;

[0118] Determine the standard area of ​​cracks based on lidar data and high-resolution image data;

[0119] Obtaining the crack geometric parameters corresponding to the standard crack area, including the standard crack depth, standard crack length and standard crack width;

[0120] Determine the crack nodes based on the standard crack depth, standard crack length and standard crack width;

[0121] Obtain natural and traffic environment information, including weather station temperature, rainfall, wind speed, traffic volume, vehicle axle weight, and vehicle speed;

[0122] Determine environmental nodes based on weather station temperature, rainfall, wind speed, traffic volume, vehicle axle weight and vehicle speed;

[0123] Capture physical entities, including bridge piers, pavement slabs, guardrails, and drainage systems, based on LiDAR data and high-resolution imagery data;

[0124] Obtain the vibration frequency corresponding to the physical entity, and determine the infrastructure nodes by combining the bridge piers, pavement slabs, guardrails and drainage systems with the vibration frequency corresponding to the physical entity;

[0125] The infrastructure nodes, crack nodes and environment nodes are connected by pre-set connection rules to obtain the infrastructure-crack-environment correlation map.

[0126] Specifically, the preset connection rules include physical proximity (such as cracks in the same bridge component and adjacent piers), causal relationship (such as traffic load and bridge deck crack propagation), and spatiotemporal coupling (such as rainfall and pavement temperature).

[0127] Visualization form:

[0128] .

[0129] Furthermore, the acquisition of the infrastructure fatigue life remaining probability and infrastructure risk level based on the infrastructure-crack-environment correlation map specifically includes:

[0130] Based on the infrastructure-crack-environment correlation map, the infrastructure node feature vector, crack node feature vector and environment node vector are obtained;

[0131] Obtain the edge weights between infrastructure nodes and crack nodes, the edge weights between environment nodes and crack nodes, and the edge weights between environment nodes and infrastructure nodes respectively;

[0132] The edge weights are sequentially input into the graph neural network to obtain the remaining probability of infrastructure fatigue life corresponding to the infrastructure nodes;

[0133] Determine the damage index corresponding to the infrastructure node based on the standard crack depth, standard crack length, and standard crack width corresponding to the crack node connected to the infrastructure node;

[0134] Determine the infrastructure risk level through the damage index corresponding to the infrastructure node;

[0135] The calculation formula for the remaining probability of fatigue life of the infrastructure corresponding to the infrastructure node is:

[0136]

[0137] Where, is the remaining probability of infrastructure fatigue life corresponding to the i-th infrastructure node, is the Sigmoid function, is the L-th layer weight matrix, is the node feature matrix corresponding to the i-th infrastructure node, is the bias term of the Lth layer;

[0138] The calculation formula of the damage index corresponding to the infrastructure node is:

[0139]

[0140] Where, is the damage index corresponding to the i-th infrastructure node, is the edge weight between infrastructure node i and crack node m, For the environment node The edge weight of the target area corresponding to the crack node m at time t, For the environment node The edge weight of the target area corresponding to infrastructure node i at time t, are the standard crack depth, standard crack length and standard crack width corresponding to the crack node m, is the mean of the maximum crack standard depth, maximum crack standard length and maximum crack standard width allowed for infrastructure node i, is the total number of crack nodes corresponding to the i-th infrastructure node.

[0141] Specifically, the specific process steps of step S4 are:

[0142] S4.1. Defining Node Types

[0143] Infrastructure Node: Bridge Pier V i 、Pavement Slab j 、Guardrail H k , drainage system D l .

[0144] Crack node: Crack area C m (Associated with an infrastructure node).

[0145] Environmental node: weather temperature T t , rainfall R t , wind speed W t , traffic flow Q t 、Vehicle axle weight F t , vehicle speed V t .

[0146] S4.2. Node feature definition

[0147] Infrastructure node characteristics: f i infra =[v i ] T (v i : vibration frequency)

[0148] Crack node characteristics: f m crack =[l m , d m , w m ]T (l m : crack length; d m : crack depth; w m : crack width;)

[0149] Environmental node characteristics: f t env =[T t , R t , W t , Q t , F t , V t ] T

[0150] In step S4.2, the calculation f i infra 、f m crack and f t env The values ​​are [*] T The sum of all quantized values ​​of * in , where T The only dimension is represented.

[0151] S4.3. Calculating link weight

[0152] Edge weights between infrastructure nodes and crack nodes:

[0153]

[0154] is the weight coefficient;

[0155] Edge weights between environment nodes and crack nodes:

[0156]

[0157] Where, and denote the covariance and variance operations respectively;

[0158] Edge weights between environment nodes and infrastructure nodes:

[0159]

[0160] For the association operation, if the environment node is associated with the infrastructure node (same time and same place), then is 1, otherwise it is 0;

[0161] For example, the crack C3 of a bridge pier V1 has a depth of d=8mm, a length of 1m, a width of 1cm, an ambient temperature of 25°C, and a traffic volume of 2000 vehicles / hour (the calculation units of depth, length, and width are negligible, but when quantifying, the width needs to be in mm, the length in m, and the width in cm).

[0162] Graph link: The edge weight between the infrastructure node and the crack node corresponding to V1 and C3 is 0.82, the edge weight between the environment node and the infrastructure node is 0.67, and the edge weight between the environment node and the crack node is 0.4.

[0163] Lifespan prediction: P(V1)=σ(0.5[0.82811+0.672000+0.350.4])=0.90 (90% residual probability)

[0164] Risk level (assuming that the infrastructure node is only associated with one environment node and one crack node, and assuming that the average of the maximum crack standard depth, maximum crack standard length, and maximum crack standard width allowed for infrastructure node i is 4): Damage index D = 0.54 → Risk level 2 (medium risk).

[0165] Infrastructure risk level mapping: .

[0166] Furthermore, the system leverages the remaining probability of infrastructure fatigue life and its risk level to generate predictive maintenance decisions. High-risk facilities are directly subject to emergency maintenance orders. Medium-risk facilities are predicted to fail based on their remaining life curves and scheduled for periodic overhauls. Low-risk facilities are included in long-term monitoring lists. Furthermore, the system optimizes maintenance resource allocation through reinforcement learning, prioritizing cracks in high-traffic and temperature-sensitive areas. It also automatically generates decision-making reports containing repair plans, cost estimates, and schedules, enabling a shift from passive response to proactive prevention.

[0167] Furthermore, a real-time monitoring system for highway infrastructure based on the UAV Internet of Things is proposed, which is used to implement any of the above monitoring methods, including:

[0168] An acquisition module, which is used to acquire lidar data, high-resolution image data, infrared image data, and vehicle load vibration data, as well as environmental data from weather stations and traffic cameras;

[0169] a data processing module for performing spatiotemporal alignment on the lidar data, the high-resolution image data, the infrared image data, and the vehicle load vibration data, for removing rain and fog interference data from the lidar data, the high-resolution image data, and the infrared image data, and for extracting crack geometric parameters, including standard crack depth, standard crack length, and standard crack width;

[0170] a risk assessment module, the risk assessment module being used to construct an infrastructure-crack-environment correlation map based on lidar data, high-resolution image data, infrared image data, vehicle load vibration data, and environmental data, and to obtain the infrastructure fatigue life remaining probability and infrastructure risk level based on the infrastructure-crack-environment correlation map;

[0171] A maintenance decision module, wherein the maintenance decision module is used to generate a predictive maintenance decision using the remaining probability of fatigue life of the infrastructure and the infrastructure risk level;

[0172] The display module is used to show the real-time monitoring process and results of highway infrastructure based on the drone Internet of Things to users.

[0173] Furthermore, the acquisition module includes:

[0174] a first acquisition unit, the first acquisition unit being used to acquire lidar data, high-resolution image data, infrared image data, and vehicle load vibration data;

[0175] The second acquisition unit is used to obtain environmental data from weather stations and traffic cameras.

[0176] Furthermore, the data processing module includes:

[0177] a spatiotemporal alignment unit, configured to perform spatiotemporal alignment on the lidar data, the high-resolution image data, the infrared image data, and the vehicle load vibration data;

[0178] A denoising unit, configured to remove rain and fog interference data from the lidar data, the high-resolution image data, and the infrared image data;

[0179] The extraction unit is used to extract fracture geometric parameters, including standard fracture depth, standard fracture length and standard fracture width.

[0180] Furthermore, the risk assessment module includes:

[0181] A correlation map construction unit, configured to construct an infrastructure-crack-environment correlation map based on lidar data, high-resolution image data, infrared image data, vehicle load vibration data, and environmental data;

[0182] A data evaluation unit is used to obtain the remaining probability of infrastructure fatigue life and the infrastructure risk level based on the infrastructure-crack-environment correlation map.

[0183] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A real-time monitoring method for highway infrastructure based on UAV Internet of Things, characterized by: include: S1. Obtain sensor data transmitted by the drone swarm, including lidar data, high-resolution image data, infrared image data, and vehicle load vibration data; S2. Obtain environmental data from weather stations and traffic cameras, perform spatiotemporal alignment of lidar data, high-resolution image data, infrared image data, and vehicle load vibration data, remove rain and fog interference data, and extract crack geometry parameters; S3. Construct an infrastructure-crack-environment correlation map based on lidar data, high-resolution image data, infrared image data, vehicle load vibration data, and environmental data. S4. Based on the infrastructure-crack-environment correlation map, obtain the remaining probability of infrastructure fatigue life and infrastructure risk level; S5. Generate predictive maintenance decisions using the remaining probability of infrastructure fatigue life and infrastructure risk level; The infrastructure-crack-environment correlation map is constructed based on lidar data, high-resolution image data, infrared image data, vehicle load vibration data, and environmental data, specifically including: Define the graph structure, including infrastructure nodes, crack nodes, and environment nodes; Determine the standard area of ​​cracks based on lidar data and high-resolution image data; Obtaining the crack geometric parameters corresponding to the standard crack area, including the standard crack depth, standard crack length and standard crack width; Determine the crack nodes based on the standard crack depth, standard crack length and standard crack width; Obtain natural and traffic environment information, including weather station temperature, rainfall, wind speed, traffic volume, vehicle axle weight, and vehicle speed; Determine environmental nodes based on weather station temperature, rainfall, wind speed, traffic volume, vehicle axle weight and vehicle speed; Capture physical entities, including bridge piers, pavement slabs, guardrails, and drainage systems, based on LiDAR data and high-resolution imagery data; Obtain the vibration frequency corresponding to the physical entity, and determine the infrastructure nodes by combining the bridge piers, pavement slabs, guardrails and drainage systems with the vibration frequency corresponding to the physical entity; The infrastructure nodes, crack nodes and environment nodes are connected by pre-set connection rules to obtain the infrastructure-crack-environment correlation map.

2. The real-time monitoring method for highway infrastructure based on UAV Internet of Things according to claim 1 is characterized in that: The spatiotemporal alignment of the lidar data, the high-resolution image data, the infrared image data, and the vehicle load vibration data specifically includes: Based on lidar data, high-resolution image data, infrared image data, and vehicle load vibration data, the standard timestamp corresponding to the high-precision clock module built into the drone swarm and the GPS timing system is obtained; LiDAR data, high-resolution image data, infrared image data, and vehicle load vibration data aligned based on standard timestamps; A cross-modal spatiotemporal registration network is constructed to perform spatiotemporal alignment of lidar data, high-resolution image data, infrared image data, and vehicle load vibration data using lidar data as the spatiotemporal reference.

3. The real-time monitoring method for highway infrastructure based on UAV Internet of Things according to claim 1 is characterized in that: The removal of rain and fog interference data specifically includes: Using the dark channel prior algorithm, the continuous RGB image frames in the high-resolution image data are histogram matched to eliminate the uneven brightness caused by rain and fog; Based on the atmospheric scattering model, the transmittance map of continuous RGB images in the high-resolution image data after eliminating the uneven brightness caused by rain and fog is obtained, and the contrast of the RGB image is restored; Set the temperature gradient threshold, synchronously calibrate the dynamic range of the infrared temperature map in the infrared image data, and eliminate abnormal temperature fluctuations caused by rain and fog; Obtain environmental depth information from the lidar data to assist in defogging the infrared temperature map in the infrared image data; obtain the point cloud density in the lidar data to determine the corresponding infrared temperature map in the infrared image data. Spatially related positions, and eliminating rain and fog scattering noise in the infrared temperature map corresponding to the spatially related positions; The continuous frame RGB images in the high-resolution image data and the infrared temperature map in the infrared image data are sequentially input into the multimodal U-Net, and the enhanced continuous frame RGB images and infrared temperature map are output; An equivalent infrared temperature map under clear weather conditions is generated through a generative adversarial network, and the temperature maps corresponding to locations with abnormal temperature fluctuations caused by rain and fog are removed from the enhanced infrared temperature map.

4. The real-time monitoring method for highway infrastructure based on UAV Internet of Things according to claim 1 is characterized in that: The extraction of crack geometric parameters specifically includes: Use the CBAM attention module to fuse the texture features of continuous frame RGB images in high-resolution image data and the thermal features of infrared temperature maps in infrared image data to generate crack candidate areas; PointNet++ is used to extract geometric features from the lidar data, including normal vectors and curvature, and the crack candidate area is corrected to obtain the crack candidate correction area. Mask R-CNN is used to perform instance segmentation on the crack candidate correction area to obtain the precise boundary of the crack; Morphological operations are performed on the precise boundaries of the cracks using closing operations and erosion and dilation to close small crack gaps and remove isolated noise points. The crack area corresponding to the remaining precise boundaries of the cracks is used as the standard crack area. Align the RGB image in the high-resolution image data corresponding to the crack standard area with the point cloud data in the lidar data, and generate a crack sparse depth map through a stereo matching algorithm; Obtaining gradient information in an infrared temperature map in the infrared image data, and determining the first crack depth in combination with a crack standard area; In the crack standard area, the pixel-level length is calculated along the main direction of the crack to obtain the crack standard length; Obtain the standard crack width corresponding to the standard crack area through the minimum circumscribed rectangle; Obtaining the point cloud density in the laser radar data corresponding to the crack standard area to obtain the second crack depth; The average of the first crack depth and the second crack depth is obtained as the standard crack depth.

5. The real-time monitoring method for highway infrastructure based on UAV Internet of Things according to claim 1 is characterized in that: The method of obtaining the infrastructure fatigue life remaining probability and infrastructure risk level based on the infrastructure-crack-environment correlation map specifically includes: Based on the infrastructure-crack-environment correlation map, the infrastructure node feature vector, crack node feature vector and environment node vector are obtained; Obtain the edge weights between infrastructure nodes and crack nodes, the edge weights between environment nodes and crack nodes, and the edge weights between environment nodes and infrastructure nodes respectively; The edge weights are sequentially input into the graph neural network to obtain the remaining probability of infrastructure fatigue life corresponding to the infrastructure nodes; Determine the damage index corresponding to the infrastructure node based on the standard crack depth, standard crack length, and standard crack width corresponding to the crack node connected to the infrastructure node; Determine the infrastructure risk level through the damage index corresponding to the infrastructure node; The calculation formula for the remaining probability of fatigue life of the infrastructure corresponding to the infrastructure node is: Where, is the remaining probability of infrastructure fatigue life corresponding to the i-th infrastructure node, is the Sigmoid function, is the L-th layer weight matrix, is the node feature matrix corresponding to the i-th infrastructure node, is the bias term of the Lth layer; The calculation formula of the damage index corresponding to the infrastructure node is: Where, is the damage index corresponding to the i-th infrastructure node, is the edge weight between infrastructure node i and crack node m, For the environment node The edge weight of the target area corresponding to the crack node m at time t, For the environment node The edge weight of the target area corresponding to infrastructure node i in time period t, are the standard crack depth, standard crack length and standard crack width corresponding to the crack node m, is the mean of the maximum crack standard depth, maximum crack standard length and maximum crack standard width allowed for infrastructure node i, is the total number of crack nodes corresponding to the i-th infrastructure node.

6. A real-time monitoring system for highway infrastructure based on the UAV Internet of Things, used to implement the monitoring method according to any one of claims 1 to 5, characterized in that: include: An acquisition module, which is used to acquire lidar data, high-resolution image data, infrared image data, and vehicle load vibration data, as well as environmental data from weather stations and traffic cameras; a data processing module for performing spatiotemporal alignment on the lidar data, the high-resolution image data, the infrared image data, and the vehicle load vibration data, for removing rain and fog interference data from the lidar data, the high-resolution image data, and the infrared image data, and for extracting crack geometric parameters, including standard crack depth, standard crack length, and standard crack width; a risk assessment module, the risk assessment module being used to construct an infrastructure-crack-environment correlation map based on lidar data, high-resolution image data, infrared image data, vehicle load vibration data, and environmental data, and to obtain the infrastructure fatigue life remaining probability and infrastructure risk level based on the infrastructure-crack-environment correlation map; A maintenance decision module, wherein the maintenance decision module is used to generate a predictive maintenance decision using the remaining probability of fatigue life of the infrastructure and the infrastructure risk level; The display module is used to show the real-time monitoring process and results of highway infrastructure based on the drone Internet of Things to users.

7. The real-time monitoring system for highway infrastructure based on UAV Internet of Things according to claim 6 is characterized in that: The acquisition module includes: a first acquisition unit, the first acquisition unit being used to acquire lidar data, high-resolution image data, infrared image data, and vehicle load vibration data; The second acquisition unit is used to obtain environmental data from weather stations and traffic cameras.

8. The real-time monitoring system for highway infrastructure based on UAV Internet of Things according to claim 6 is characterized in that: The data processing module includes: a spatiotemporal alignment unit, configured to perform spatiotemporal alignment on the lidar data, the high-resolution image data, the infrared image data, and the vehicle load vibration data; A denoising unit, configured to remove rain and fog interference data from the lidar data, the high-resolution image data, and the infrared image data; The extraction unit is used to extract fracture geometric parameters, including standard fracture depth, standard fracture length and standard fracture width.

9. The real-time monitoring system for highway infrastructure based on UAV Internet of Things according to claim 6 is characterized in that: The risk assessment module includes: A correlation map construction unit, configured to construct an infrastructure-crack-environment correlation map based on lidar data, high-resolution image data, infrared image data, vehicle load vibration data, and environmental data; A data evaluation unit is used to evaluate the data based on the infrastructure-crack-environment correlation map. Obtain the remaining probability of infrastructure fatigue life and infrastructure risk level.

Images