Air-ground cooperative inspection and detection system and method for highway infrastructure diseases

Abstract

The invention discloses an air-ground cooperative inspection and detection system and method for highway infrastructure diseases, and relates to the technical field of road infrastructure safety monitoring. The system comprises a data acquisition module used for acquiring highway infrastructure data from different dimensions; the data processing module is used for aligning and extracting highway infrastructure data of different dimensions in time and space by using a multi-modal neural network, and fusing the extracted features in combination with a cross-modal attention mechanism and a multi-scale feature fusion strategy; and the three-dimensional imaging module is used for carrying out multi-physics field joint inversion according to the features fused by the data processing module, constructing a three-dimensional model of the highway infrastructure, and carrying out disease identification and early warning according to the three-dimensional model of the highway infrastructure. According to the invention, combined acquisition and multi-modal data cross-domain fusion processing are carried out on various devices, and comprehensive detection of shallow surface and deep structures along the highway is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of road infrastructure safety monitoring, and in particular to a system and method for collaborative air-ground inspection and detection of highway infrastructure defects. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid expansion of the highway network, infrastructure safety issues along expressways are becoming increasingly prominent. Expressways are particularly vulnerable to a variety of natural disasters in mountainous areas, high-hazard zones, and complex terrain. Changes in geological conditions or rainfall and snowmelt can easily trigger landslides, mudslides, and other disasters, which can damage the road surface and directly impact expressway safety, causing traffic disruptions at best and even casualties at worst.

[0004] Existing inspection and monitoring methods mostly rely on manual labor or single equipment, resulting in low detection efficiency, shallow detection depth, limited coverage, and high detection costs. While the rise of drone technology has improved inspection efficiency to a certain extent, existing technologies, when integrating drones with other equipment to achieve multimodal data collection, fail to further explore the interrelationships between multimodal data, hindering accurate disease detection results. Consequently, detection still relies on the inherent functionality of a single device.

[0005] Therefore, there is an urgent need for a comprehensive intelligent inspection and detection system that can collect multi-dimensional data from multiple devices and conduct comprehensive analysis and judgment to achieve comprehensive monitoring and status diagnosis of highway infrastructure. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a system and method for the collaborative air-ground inspection and detection of highway infrastructure defects, which can jointly collect data from multiple devices and perform cross-domain fusion processing of multimodal data, thereby realizing comprehensive detection of shallow and deep structures along the highway.

[0007] In order to achieve the above object, the present invention is implemented through the following technical solutions: A first aspect of the present invention provides a highway infrastructure defect collaborative air-ground inspection and detection system, comprising: Data collection module, used to collect highway infrastructure data from different dimensions; The data processing module is used to align and extract highway infrastructure data of different dimensions in time and space using a multimodal neural network, and to fuse the extracted features using a cross-modal attention mechanism and a multi-scale feature fusion strategy; The three-dimensional imaging module is used to perform multi-physics field joint inversion based on the features fused by the data processing module, construct a three-dimensional model of highway infrastructure, and perform disease identification and early warning based on the three-dimensional model of highway infrastructure.

[0008] Furthermore, the data acquisition module includes a UAV take-off and landing platform and an airborne detection module, a vehicle-mounted three-dimensional radar detection module and a vehicle-mounted seismic detection module. The UAV take-off and landing platform and the airborne detection module are used for the take-off and landing of the UAV and for carrying detection equipment for detection. The vehicle-mounted three-dimensional radar detection module is used to detect the structure below the shallow surface of the highway pavement. The vehicle-mounted seismic detection module is used to verify and further detect the detection results of the vehicle-mounted three-dimensional radar detection module.

[0009] Furthermore, the UAV take-off and landing platform and the airborne detection module include a UAV take-off and landing platform module and a UAV module. The UAV take-off and landing platform module is used for the take-off and landing of the UAV and the autonomous planning of the inspection path, and dynamically adjusts the flight path according to the real-time collected data. The UAV module includes a UAV and the detection equipment carried by the UAV. The UAV can achieve different detection purposes by carrying different types of detection equipment.

[0010] Furthermore, the vehicle-mounted 3D radar detection module uses ground-penetrating radars of different frequencies to analyze the reflection characteristics of underground structures, thereby generating a 3D image below the surface.

[0011] Furthermore, the vehicle-mounted seismic detection module accurately restores the spatial position of underground abnormal targets by arranging active excitation source devices and seismic detection devices at the front and bottom of the inspection vehicle, and adjusting the source frequency and energy output according to different strata.

[0012] Furthermore, it also includes a data transmission module for pre-processing the data collected by the data acquisition module and transmitting it to the data processing module in real time.

[0013] Furthermore, a pyramid structure and a skip connection mechanism are introduced into the multi-scale feature fusion strategy. The pyramid structure is used to dynamically adjust the downsampling rate, and the skip connection mechanism is embedded with electromagnetic filtering features, which can simultaneously retain the coarse-grained global features and fine-grained local features of the data, reduce cross-modal alignment errors, and enhance the model's target recognition capabilities at different scales.

[0014] Furthermore, based on the results of the multi-physics field joint inversion, a three-dimensional model of the highway infrastructure is obtained by cross-scale voxel modeling of the macro, meso and micro layers.

[0015] Furthermore, the three-dimensional imaging module includes a disaster warning module, which triggers a three-level warning mechanism based on the disease identification results, and provides prompts and warnings to the identified high-risk areas, medium-risk areas or low-risk areas respectively.

[0016] A second aspect of the present invention provides a highway infrastructure defect collaborative air-ground inspection and detection system method, comprising the following steps: Collect highway infrastructure data from different dimensions; A multimodal neural network is used to align and extract highway infrastructure data of different dimensions in time and space, and the extracted features are fused by combining a cross-modal attention mechanism and a multi-scale feature fusion strategy. Based on the characteristics of data processing module fusion, multi-physical field joint inversion is performed to construct a three-dimensional model of highway infrastructure, and disease identification and early warning are carried out based on the three-dimensional model of highway infrastructure.

[0017] One or more of the above technical solutions have the following beneficial effects: The present invention discloses a system and method for collaborative air-ground inspection and detection of highway infrastructure defects. Through the synergistic effects of multi-dimensional data collection, multi-modal data fusion, three-dimensional model construction and defect identification, intelligent inspection, and data verification, it achieves efficient, accurate, and intelligent inspection and early warning of highway infrastructure defects, effectively improving the operation and maintenance management level of highway infrastructure and ensuring the safe operation of highways.

[0018] The present invention uses an unmanned aerial vehicle (UAV) take-off and landing platform and an airborne detection module, a vehicle-mounted three-dimensional radar detection module, and a vehicle-mounted seismic detection module to collect data on highway infrastructure from multiple dimensions, both in the air and on the ground. This can comprehensively cover all levels of highway infrastructure, including the shallow surface layer and deep structure of the road surface, providing a rich and comprehensive data foundation for subsequent disease identification.

[0019] This paper utilizes a multimodal neural network to align and extract the collected data of different dimensions in time and space. Combined with a cross-modal attention mechanism and a multi-scale feature fusion strategy, it can effectively address the heterogeneity problem between multi-source data and achieve deep data fusion. The cross-modal attention mechanism can highlight key information in data of different modalities. The multi-scale feature fusion strategy, by introducing a pyramid structure and skip connection mechanism, retains both the coarse-grained global features and the fine-grained local features of the data, reducing cross-modal alignment errors, significantly enhancing the model's ability to recognize targets at different scales, and improving the accuracy of disease identification.

[0020] Based on the results of multi-physics field joint inversion, the present invention obtains a three-dimensional model of highway infrastructure through cross-scale voxel modeling of the macro, meso and micro layers. This model can more realistically and intuitively reflect the internal structure and disease conditions of highway infrastructure, providing strong support for the accurate diagnosis of diseases.

[0021] This invention uses a constructed 3D model to identify defects, accurately identifying the types and locations of various defects in highway infrastructure, such as cracks, voids, and subsidence. Simultaneously, the disaster warning module within the 3D imaging module triggers a three-level warning mechanism based on the defect identification results, providing prompts and warnings for high-risk, medium-risk, and low-risk areas, respectively. This enables early detection and timely treatment of defects, effectively reducing their impact on the safe operation of highways and improving their safety and reliability.

[0022] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0024] Figure 1 This is a schematic diagram of the workflow of the air-ground collaborative inspection and detection system for highway infrastructure defects in Example 1 of the present invention; Figure 2 Schematic diagram of a probe vehicle equipped with a highway infrastructure defect air-ground collaborative inspection and detection system in Example 1 of the present invention; Figure 3 This is a schematic diagram of the vehicle-machine collaborative workflow in Example 1 of the present invention; Among them, 1. Active excitation source device, 2. Detection vehicle, 3. Control center, 4. UAV take-off and landing platform, 5. UAV take-off and landing platform and airborne detection module, 6. UAV, 7. Vehicle-mounted three-dimensional radar carrying mechanism, 8. Vehicle-mounted three-dimensional radar detection module, 9. Seismic detector module. DETAILED DESCRIPTION

[0025] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0026] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations; Example 1: The first embodiment of the present invention provides a highway infrastructure disease air-ground collaborative inspection and detection system, such as Figure 1 As shown, the workflow includes: S1: Mission Planning and Equipment Preparation. The system generates an inspection task list based on multi-dimensional data such as the target inspection area's geographic information, historical disease records, and meteorological conditions. It utilizes geographic information systems and historical risk models to create a checklist. Based on the geological characteristics and disaster susceptibility of different regions, it develops coordinated inspection plans for drones and vehicle-mounted equipment, ensuring high-density scanning of key areas. It automatically generates optimal inspection routes, dynamically adjusts task priorities, and optimizes drone and vehicle-mounted equipment routes.

[0027] S2: Collect multi-source data based on the prepared and planned content. Under the task scheduling of control center 3, the drone flies along the planned route, collecting detection data from the target area. While traveling at a constant speed, the detection vehicle 2 uses its onboard 3D radar and seismic detection system to collect road detection data. Based on the pre-analysis results of the real-time data collection, the system identifies high-risk areas and automatically adjusts the drone's flight path and the detection strategy of the onboard equipment. In key areas, a multi-source collaborative detection mode is initiated.

[0028] S3: Data preprocessing and multimodal data fusion are then performed based on the collected data. By preprocessing the drone and vehicle-mounted detection data, underground structure reflection characteristics are extracted and key geological interfaces are identified. The data processing module utilizes a multimodal neural network architecture to align and fuse the drone and vehicle-mounted detection data in time and space. Data is standardized, denoised, and modally converted to ensure consistency and high quality across modal data. The 3D imaging module uses multi-physics field joint inversion to construct a cross-scale voxel model encompassing macro, meso, and micro layers.

[0029] S4: Defect identification and dynamic risk assessment based on multimodal data fusion results. Based on the 3D model and fused data, the system automatically identifies surface, shallow, and deep-seated defects. Defect areas are presented as heat maps, annotated with location, extent, and risk level. A machine-learning-based defect evolution model, combined with environmental variables such as rainfall and temperature fluctuations, predicts defect development trends and timing. The system systematically assesses the speed and direction of defect development, the impact of defects on highway safety, and the probability of potential disasters (such as landslides and debris flows).

[0030] S5: Disaster early warning and response based on identification and risk assessment results. A three-level early warning mechanism is set up to enable simultaneous early warning push, generate emergency response plans, and conduct disaster detection linkage with relevant departments.

[0031] S6: Generate a report based on the detection results and compare and evaluate it with historical data as a reference for the next mission planning and equipment preparation.

[0032] In this embodiment, the above steps are implemented by using a probe car 2 equipped with a highway infrastructure defect air-ground collaborative inspection and detection system. Figure 2 As shown, the system includes: a drone take-off and landing platform and airborne detection module 5, a vehicle-mounted 3D radar mounting mechanism 7, a vehicle-mounted 3D radar detection module 8, a vehicle-mounted seismic detection module, a data transmission module, a data processing module, and a 3D imaging module. By combining drones, airborne detection equipment, and vehicle-mounted detection equipment, the system achieves comprehensive detection of both shallow and deep structures along highways. It can effectively identify and diagnose potential risks of various geological hazards, particularly threats to highways from collapses, landslides, and debris flows. It also enables intelligent prediction of the evolution of highway foundation defects, providing real-time diagnosis of highway road conditions and intelligent early warning of defects.

[0033] The specific contents of each module are as follows: The data acquisition module is used to collect highway infrastructure data from different dimensions. The data acquisition module includes a drone landing platform and an airborne detection module, a vehicle-mounted 3D radar detection module 8, and a vehicle-mounted seismic detection module. The drone landing platform and the airborne detection module are used for drone 6 takeoff and landing and for carrying detection equipment for detection. The vehicle-mounted 3D radar detection module is used to detect structures below the shallow surface of the highway pavement. The vehicle-mounted seismic detection module is used to verify the detection results of the vehicle-mounted 3D radar detection module and conduct further detection.

[0034] The drone take-off and landing platform and airborne detection module include a drone take-off and landing platform module and a drone module. The drone take-off and landing platform module is used for the drone's take-off and landing, as well as autonomous planning of inspection routes, and dynamically adjusts the flight route based on real-time collected data. The drone module includes a drone and detection equipment carried by the drone. The drone can achieve different detection purposes by carrying different types of detection equipment.

[0035] In one specific embodiment, the drone landing module includes a drone landing platform 4, which is responsible for ensuring autonomous takeoff and landing of the drone. The drone landing platform 4 is controlled by an embedded control module. The drone can autonomously plan inspection routes based on set mission requirements and dynamically adjust its flight path based on real-time data collected.

[0036] Drone 6 in the drone module carries detection equipment for aerial inspections, combining it with the vehicle's 3D radar and seismic detection system to conduct shallow and deep structural surveys. The drone is equipped with different types of detection equipment to achieve different detection objectives. It features high-resolution cameras, infrared imagers, lidar, and ground-penetrating radar for aerial inspections and shallow surface detection. The high-resolution camera primarily captures high-definition images of road surfaces and slopes, capturing surface changes such as cracks and subsidence. The infrared imager measures temperature differences in the road surface to detect temperature anomalies caused by subsurface voids and water seepage, and in conjunction with ground-penetrating radar, identifies hidden road surface defects. The lidar is used for road surface smoothness measurement and 3D modeling. The airborne ground-penetrating radar enables detailed detection of underground targets within 15 meters, including efficient scanning and data collection of roadside slopes, and can detect impending highway collapse, landslides, mudslides, and collapses.

[0037] UAV 6 uses laser infrared joint calibration technology to establish a temperature-deformation correlation model to eliminate thermal expansion misjudgment.

[0038] Specifically, the temperature-deformation correlation model is: .

[0039] Among them, ΔL is the deformation, α is the thermal expansion coefficient of the material, L0 is the initial length, ΔT is the temperature change, and β is the laser ranging correction coefficient. The temperature gradient data collected by the infrared imager can eliminate the misjudgment of deformation caused by thermal expansion.

[0040] Specifically, the temperature-deformation correlation model introduces material thermal expansion parameters and temperature gradient compensation terms, and uses the material thermal expansion coefficient and temperature change to calculate the thermally induced deformation, thereby realizing the identification of non-structural deformation caused by temperature change, thereby effectively eliminating the misjudgment of deformation caused by thermal expansion.

[0041] The vehicle-mounted 3D radar detection module 8 is composed of ground-penetrating radars of different frequencies. It analyzes the reflective characteristics of underground structures using these frequencies to generate a 3D image of the subsurface. A vehicle-mounted 3D radar mounting mechanism 7 is located at the rear of the probe vehicle 2 to carry the vehicle-mounted 3D radar detection module 8.

[0042] In a specific embodiment, the vehicle-mounted 3D radar detection module of this embodiment utilizes a multi-channel antenna array and fully polarized radar technology to enhance its ability to identify different targets, enabling high-resolution and rapid detection within 3 meters below the road surface. The module also utilizes adaptive variable-frequency pulse signal modulation technology to achieve high-resolution detection, from shallow depths to deep depths over a wide range of frequencies. Combined with a high-precision inertial navigation system, it effectively compensates for errors caused by vehicle motion and can complete detection while maintaining a constant speed, achieving high-resolution detection at speeds of 60-80 km / h. The system achieves centimeter-level accuracy, enabling detection of subtle structural issues and providing early warnings. By emitting electromagnetic waves to the target area and receiving the reflected signals, the system analyzes the reflection characteristics of the underground structure, generating a precise 3D image of the subsurface and accurately detecting potential hazards such as cracks and cavities beneath the road surface.

[0043] Specifically: Adaptive variable frequency pulse signal modulation technology: The system dynamically adjusts the pulse signal transmission frequency (ranging from 100 MHz to 2 GHz) and adaptively selects the optimal operating frequency according to road type and detection depth to optimize detection depth and resolution.

[0044] High-precision inertial navigation compensation: The on-board 3D radar detection module integrates a high-precision inertial navigation system (INS). By monitoring the vehicle's acceleration and angular velocity in real time, it compensates for positioning deviations caused by vehicle motion during high-speed driving, ensuring the spatial alignment accuracy of radar data is better than 5cm.

[0045] Multi-channel full-polarization technology: The vehicle-mounted 3D radar detection module is equipped with a multi-channel antenna array and adopts full-polarization technology. By transmitting and receiving electromagnetic waves in different polarization directions, it enhances the recognition ability of underground targets such as cracks and cavities, thereby improving the accuracy of disease detection.

[0046] In terms of data processing, the vehicle-mounted 3D radar detection module introduces deep learning-driven radar signal adaptive noise reduction technology, combined with wavelet transform and Hilbert packet (Hilbert envelope) network analysis to effectively remove environmental noise, and uses an adaptive reverse time migration algorithm combined with intelligent filtering methods to improve underground target resolution.

[0047] Specifically, the vehicle-mounted radar system continuously detects underground structures, acquiring echo signal sequences and forming preliminary time-domain radar data. The original radar signal is subjected to multi-scale wavelet decomposition, extracting sub-signals in each frequency band to identify and separate noise and useful target features at different frequency components. A Hilbert transform is applied to the specific frequency band signal after the wavelet transform to obtain its envelope spectrum, and the significant energy position of the target feature is identified by changes in the envelope amplitude. The signal is reconstructed by adjusting the weights of each frequency band, suppressing low-amplitude noise and enhancing the boundary clarity of the target signal. An adaptive reverse time migration algorithm is used to invert and propagate the noise-reduced signal, focusing on the underground target and improving imaging accuracy. Based on the imaging results, intelligent filtering technology is used to further enhance image resolution and ensure a clear outline of the underground target.

[0048] The vehicle-mounted seismic detection module arranges an active excitation source device 1 and a seismic detector device at the front and bottom of the inspection vehicle, and adjusts the source frequency and energy output according to different strata to accurately restore the spatial position of underground abnormal targets.

[0049] In a specific embodiment, the vehicle-mounted seismic detection module integrates composite source technology, deep learning-driven seismic signal noise reduction algorithm, adaptive filtering algorithm and intelligent offset imaging algorithm, which can effectively improve the resolution of seismic reflection signals in a low signal-to-noise ratio environment and accurately restore the spatial position of underground abnormal targets.

[0050] Among them, the composite source technology uses the combined excitation of electromagnetic and mechanical sources. By adjusting the pressure curve of the hydraulic system and the pulse frequency of the electromagnetic coil, it can achieve multi-band excitation output in the range of 20 Hz to 500 Hz, thereby improving the adaptability of detection depth under different formation conditions.

[0051] The deep learning-driven seismic signal denoising algorithm first constructs a training dataset containing a variety of typical noises and target reflector signals, and uses a convolutional neural network with a U-Net architecture for feature extraction and noise discrimination training. The collected raw seismic waveforms are input into the trained model, which automatically identifies and adaptively filters out noise components, outputting a target reflection waveform with a high signal-to-noise ratio and clear boundaries, thereby maintaining the integrity of the target reflector's boundary features.

[0052] In the adaptive filtering process, the filter passband range is dynamically set according to the estimated target depth and seismic data sampling rate. Bandpass filtering and empirical mode decomposition are used for spectrum decomposition and signal reconstruction. Hilbert envelope analysis is further combined to achieve local contrast enhancement of waveform amplitude and background noise suppression.

[0053] In intelligent migration imaging processing, the initially established velocity model is used in combination with the finite difference method to simulate the seismic wave propagation path, and the reverse time migration method is used to perform time inversion and spatial focusing of the received signal to achieve accurate restoration of the wave field energy to the position of the real reflector; the sparse constrained inversion strategy is used to optimize the velocity model, and the bilateral filtering algorithm is combined to enhance the edge features and suppress noise of the imaging results, completing high-resolution visual reconstruction of complex underground structures.

[0054] The specific steps include: Step 1: Build a velocity model.

[0055] Step 1.1: Acquire required data: The active excitation source device in the vehicle-mounted seismic detection module generates multi-band excitation waves in the range of 20 Hz to 500 Hz, and the seismic detector receives seismic signals reflected and refracted by underground structures.

[0056] Step 1.2: Construct the initial velocity field. The system analyzes the travel time-distance characteristic curve of the collected seismic signal to calculate the apparent velocity distribution. It then combines the highway engineering geological data to establish an initial layered velocity model that includes the concrete layer, asphalt layer, base layer, and soil layer.

[0057] Step 1.3: Grid discretization: Divide the detection area into three-dimensional grid cells of size 0.1m × 0.1m × 0.05m, and assign an initial velocity value to each grid cell to form a discretized velocity field.

[0058] Step 1.4: Construct multi-source data constraints. The system combines the dielectric constant information obtained by the vehicle-mounted 3D radar detection module and the UAV infrared imaging data to constrain the initial velocity field by establishing a dielectric constant-velocity conversion relationship and a temperature-velocity correlation model.

[0059] Step 1.5: Traveltime inversion optimization. Use the traveltime equation and the least squares iterative method to perform preliminary inversion optimization on the velocity model until the traveltime residual is reduced to a preset threshold.

[0060] Step 1.6: Full waveform inversion refinement. Construct an objective function with L1-norm sparsity constraints and use a multi-scale strategy to gradually optimize the velocity model from low frequency to high frequency to obtain a high-precision velocity field model.

[0061] Step 2: Use the velocity model to perform intelligent migration imaging processing.

[0062] Step 2.1: Wavefield simulation calculation. Based on the optimized velocity model, the second-order acoustic wave equation is discretized using the finite difference method. A reasonable time step and spatial grid are set to meet stability conditions to simulate the propagation path of seismic waves in the underground medium. Step 2.2: Construct a bidirectional wavefield. Based on the velocity model, simulate the forward wavefield propagation and calculate the wavefield state from the earthquake source location. Then, based on the actual recorded data of the geophone, reconstruct the reverse wavefield. Step 2.3: Reverse time migration imaging: perform cross-correlation calculations on the forward wavefield and the reverse wavefield at each time step, focus the wavefield energy at the actual reflector position, and form the initial imaging result; Step 2.4: Model iterative optimization. The velocity model is modified using a sparse constraint inversion strategy. The sparse constraint controls the gradient sparsity of the model parameters through the L1 norm regularization term to improve the boundary recognition ability. Step 2.5: Image enhancement. Apply a bilateral filtering algorithm to the imaging results. This algorithm takes into account both spatial distance weights and range similarity weights, retaining structural boundary features while suppressing random noise. Step 2.6: Defect Feature Extraction: Based on the enhanced imaging results, cracks, voids, and loose areas in the highway infrastructure are identified by analyzing abnormal reflection characteristics, velocity anomaly areas, and their geometric characteristics. Defects are then located, classified, and risk assessed.

[0063] The data transmission module is used to pre-process the data collected by the data acquisition module and transmit it to the data processing module in real time.

[0064] In a specific embodiment, the data transmission module is responsible for transmitting the detection data of the airborne high-resolution camera, infrared imager, ground penetrating radar, etc. and the pre-processed data of the vehicle-mounted three-dimensional radar detection module and the vehicle-mounted seismic detection module to the data processing module in real time.

[0065] The data processing module is used to use multimodal neural networks to align and extract highway infrastructure data of different dimensions in time and space, and to fuse the extracted features by combining cross-modal attention mechanism and multi-scale feature fusion strategy.

[0066] In a specific embodiment, the data processing module adopts an adaptive data fusion architecture based on a multimodal neural network optimized for road linear features. The data of each modality is preprocessed, and the feature dimensions are matched and adjusted. Through the axial attention mechanism, differentiable morphological residual module and cross-modal fusion technology, the detection data from drones and vehicle-mounted detection data are aligned and fused in time and space.

[0067] The data processing module includes a preprocessing module, an axial attention road feature extraction module, a differentiable morphological residual module, and a multi-scale feature fusion module: The preprocessing module of the data processing module is responsible for standardizing, denoising, and modal conversion of data to ensure consistency and high quality across all modalities. The data is then fed into the corresponding feature extraction subnetworks to achieve deep feature learning. Specifically, drone imagery (visible light and infrared) uses a ResNet + Transformer fusion network, extracting local detail features through ResNet and modeling global semantic information using Transformer. Visible light imagery uses convolutional neural network feature extraction to detect cracks and road damage, while infrared imagery uses heatmap normalization and infrared temperature difference analysis to identify anomalous hotspots. Vehicle-mounted radar data uses a one-dimensional convolutional neural network to extract frequency features from radar echoes, combined with a bidirectional LSTM for temporal dependency modeling to analyze continuous changes in underground structures. Seismic waveform data uses a gated recurrent unit and an attention mechanism to extract key seismic wave features, and a joint time-frequency transform is used to analyze signal characteristics.

[0068] The data processing module's axial attention road feature extraction module employs an axial attention mechanism. Through attention submodules in two dimensions, horizontally and vertically, the horizontal axis focuses on transverse crack features, while the vertical axis focuses on longitudinal crack features. Each submodule is equipped with eight attention heads, which perform weighted calculations on the input image feature map in the horizontal and vertical directions, respectively, to extract transverse and longitudinal crack features from road images. A pre-trained crack morphology prior matrix is ​​based on 100,000 historical highway crack image data. After extracting typical crack morphological features using a ResNet-50 convolutional neural network, a 256×256 spatial weight matrix is ​​trained to perform point-wise weighted multiplication on the feature maps output by the attention. This strengthens the response strength of crack patterns, thereby improving the accuracy and robustness of crack detection.

[0069] The specific calculation method is: The feature map is input into the Transformer encoder, and the attention weights are generated by combining the query, key, and value matrices to ultimately output enhanced crack features. The pre-trained crack morphology prior matrix is ​​based on historical highway crack image data. After extracting typical crack morphology features through a ResNet-50 convolutional neural network, a 256×256 spatial weight matrix is ​​trained to perform point-wise multiplication weighting on the feature map output by the attention, thereby enhancing the response strength of the crack pattern.

[0070] The Differentiable Morphological Residual Module is a combined module designed to integrate differentiable dilation, erosion, and residual mechanisms. Its input is the output of the axial attention road feature extraction module, which serves as the input to the fusion module. The Differentiable Morphological Residual Module consists of parallel dilation and erosion submodules, optimized through backpropagation to achieve end-to-end joint training of morphological operations and deep learning.

[0071] The expansion kernel is initialized to a matrix of all 1s, simulating the traditional morphological expansion operation. The erosion kernel is initialized to a differential pattern of -2 at the center and 1 at the edge, ensuring structural integrity through the following constraints: .

[0072] Among them, i represents the position index of the i-th row in the convolution kernel, j represents the position index of the j-th column in the convolution kernel, is the initialization sum of the dilation kernel, is the initialization sum of the erosion kernel, is the total number of nuclear elements.

[0073] The 5×5 dilation kernel and 3×3 erosion kernel in the differentiable morphological residual module are optimized using the stochastic gradient descent (SGD) algorithm, with an initial learning rate of 0.01 and a momentum of 0.9. During training, the cross-entropy loss for crack detection is used as the objective function, and the kernel parameters are iteratively adjusted to enhance crack continuity and suppress noise. The dilation operator submodule is implemented using separable convolution, using a 5×5 learnable kernel to enhance the continuity of road cracks. The erosion operator submodule uses a 3×3 dynamic kernel, which automatically adjusts the kernel weights based on the local gradient information of the input feature map to suppress isolated noise points.

[0074] The forward propagation formula of the differentiable morphological residual module is: .

[0075] in: represents the module output features, represents the module input feature, D 5×5 is a 5×5 learnable expansion kernel, E 3×3 is a 3×3 dynamic erosion kernel, Conv 1×1 It is a 1×1 convolutional layer.

[0076] The multi-scale feature fusion module combines the cross-modal attention mechanism with a multi-scale feature fusion strategy to fuse the extracted features. During the data fusion process, the cross-modal attention mechanism is employed to enable the network to automatically learn the importance weights of features from different modalities, dynamically adjusting the contribution of each data source in the decision-making process, thereby enhancing the reliability of the overall information. Features from different modalities are aligned, establishing associations between various types of airborne and vehicle-based detection data. The attention mechanism strengthens information interaction between features from different modalities.

[0077] Specifically, the vehicle-mounted seismic detection module collects road vibration signals in real time, and calculates Young's modulus E and Poisson's ratio v based on the elastic wave inversion algorithm to achieve dynamic parameter inversion. Establish visible light image crack characteristics Dielectric anomaly characteristics of ground penetrating radar Elastic mechanics correlation model: .

[0078] in, is the radar characteristic impedance, which is calculated from the transmitting frequency f and the medium parameters. is the magnetic permeability, is the relative dielectric constant, obtained in real time through vehicle-mounted radar echo inversion. The material mechanics constitutive relationship is embedded in the deep learning loss function to achieve physical interpretability of cross-modal feature fusion.

[0079] The multi-scale feature fusion module employs a multi-scale feature fusion strategy that incorporates a pyramid structure and a skip connection mechanism. The pyramid structure dynamically adjusts the downsampling rate, while the skip connection mechanism embeds electromagnetic filtering features. This allows the system to simultaneously preserve coarse-grained global features such as the overall road structure and landslide distribution, as well as fine-grained local features such as crack morphology and underground anomalous echo signals. This reduces cross-modal alignment errors and enhances the model's ability to recognize objects at different scales. Multi-scale fusion ensures that the system can capture both macro-level disaster signs and accurately identify subtle structural changes, improving the accuracy of detecting road hazards.

[0080] The three-dimensional imaging module is used to perform multi-physics field joint inversion based on the features fused by the data processing module, construct a three-dimensional model of highway infrastructure, and perform disease identification and early warning based on the three-dimensional model of highway infrastructure.

[0081] In one specific embodiment, the 3D imaging module includes a joint inversion module, a model building module, and a disaster warning module. The 3D imaging module uses multi-physics joint inversion and cross-scale voxel modeling to generate a 3D model of highway infrastructure for disease evolution prediction. This module comprehensively detects the deep and shallow structures and slopes of the highway foundation, identifies potential structural threats, and enables disaster prediction and early warning.

[0082] The joint inversion module couples the electromagnetic wave equation, elastic wave equation, and heat conduction equation and uses the alternating direction multiplier method (ADMM) to iteratively solve them. Specifically: Coupling mode: The system uses the dielectric constant in the electromagnetic wave equation, the elastic modulus in the elastic wave equation, and the thermal conductivity in the heat conduction equation as parameters to be inverted, and realizes the joint optimization of multiple physical fields through a shared underground structure model.

[0083] Specifically: Construct a joint inversion model for coupled electromagnetic wave equations, elastic wave equations, and heat conduction equations: .

[0084] in, represents the electric field strength, represents the magnetic permeability, represents the dielectric constant, represents the displacement vector, represents density, represents the stiffness tensor, t represents time, Indicates temperature, represents the specific heat capacity, Indicates thermal conductivity, dielectric constant The material constitutive relationship is associated with the elastic modulus and Poisson's ratio to share the underground structure model during the inversion process, thereby ensuring model consistency.

[0085] ADMM iterative solution: The alternating direction multiplier method (ADMM) is used for iterative solution, breaking the original multiphysics inversion problem into three subproblems. The dielectric constant, elastic modulus, Poisson coefficient, and thermal conductivity are optimized alternately. The constraints are updated by introducing Lagrange multipliers, and the solution is gradually iterated.

[0086] Convergence threshold: Set the residual convergence threshold δ = 0.01, set the relative error of dielectric constant ≤ 5%, set the relative error of elastic modulus ≤ 3%, set the relative error of thermal conductivity ≤ 2%, and meet the requirements of relevant road detection specifications to ensure the accuracy of the inversion results.

[0087] The model building module, based on the results of multi-physics field joint inversion, generates a 3D model of highway infrastructure through cross-scale voxel modeling at the macro, meso, and micro levels. It also combines neural radiation fields with time series prediction models to generate a cloud map of the disease evolution.

[0088] First, a cross-scale voxel modeling including the macro layer (0.5m³), the meso layer (0.1m³), and the micro layer (5mm³) was constructed, and the three-level voxel grid system was defined as shown in Table 1: Table 1. Three-level voxel grid system

[0089] The macroscopic layer refers to the overall structural level of the highway infrastructure, mainly including: roadbed and foundation system, bridge foundation structure, large slope and protection engineering, with a resolution of 0.5m×0.5m×0.2m, constructed by data obtained by airborne ground penetrating radar and seismic refraction waves, and the physical field constraint is E≥1GPa. The mesoscopic layer refers to the intermediate structural level of the highway pavement and below, mainly including: asphalt surface layer, cement concrete surface layer, base layer and subbase layer in the pavement structure system, with a resolution of 0.1m×0.1m×0.05m, constructed by data obtained by vehicle-borne radar and direct wave seismic signals, and the physical field constraint is dielectric anomaly Δ >15%. The microscopic layer refers to the surface structure of the highway, primarily the surface and near-surface structures of the pavement, expansion joints, and other structures. It has a resolution of 5mm × 5mm × 2mm and is constructed using data obtained from laser point clouds and infrared thermal imaging. The physical field constraint is a temperature gradient ∇T ≥ 2°C / m.

[0090] Voxel attribute fusion rules: , weight coefficient Dynamically adjusted according to sensor confidence, where represents the fused voxel characteristics, Represents the weight coefficient of electromagnetic wave data, the value is 0.6, represents the dielectric constant, Indicates the weight coefficient of elastic wave data, the value is 0.3, E represents the elastic modulus, It represents the weight coefficient of thermal data, with a value of 0.1, and ∇T represents the temperature gradient.

[0091] Voxelized models (macro, meso, and micro layers) serve as the basis for the digital representation of highway infrastructure, storing complete spatial structure and physical property information at different resolution levels. When these multi-scale voxel data are input into the neural radiation field, the system can simultaneously analyze all-round characteristics from the overall structure to tiny cracks. Combined with physical constraints such as dielectric constant, elastic modulus, and temperature gradient, the neural radiation field renders this data and extrapolates the time dimension. Based on the historical change trends of these physical parameters and combined with the time series prediction model, it deduces the development status of the disease in the next 30 days, and ultimately generates a disease evolution cloud map with spatial accuracy and temporal prediction capabilities.

[0092] Afterwards, the neural radiation field is combined with the time series prediction model to generate a disease evolution cloud map. The improved neural radiation field architecture includes: Physical features embedded in the layer: dielectric constant , longitudinal wave velocity vp as additional input channels.

[0093] Stiffness constraint activation function: suppresses the voxel rendering intensity in the region with elastic modulus E < 0.5 GPa.

[0094] Multi-scale sampling strategy: macro-level sampling interval is 10 cm, and micro-level sampling interval is 1 mm.

[0095] Volume rendering formula:

[0096] .

[0097] Building a time series prediction model based on Neural ODEs: .

[0098] Among them, h(t) is the hidden state, which includes characteristics such as crack expansion and void growth. E(t) and v(t) are elastic parameters for real-time inversion.

[0099] Finally, a disease evolution simulation is performed. The 3D imaging module constructs a disease evolution prediction model through neural differential equations (Neural ODEs). The real-time inverted elastic parameters E(t) and Poisson's ratio v(t) are input, and a supervised learning method is used to train the Neural ODE model based on historical disease evolution data. The model learns the dynamic characteristics of disease evolution over time and outputs a probability cloud map of disease development in the next 30 days with a spatial resolution of 0.1m.

[0100] The disaster warning module triggers a three-level warning mechanism based on the results of disease identification, and provides prompts and warnings to the identified high-risk areas, medium-risk areas or low-risk areas, and cooperates with the monitoring center to trigger an emergency response.

[0101] In one specific implementation, the disaster warning module can enhance detection in key monitoring areas where landslides and debris flows are likely to occur, such as after rainfall or snowfall. This allows for real-time analysis and prediction of these hazards. Upon detecting a potential hazard, the system will issue a three-level warning signal and communicate with the monitoring center system to ensure a timely early warning response.

[0102] Three-level early warning mechanism: ① Level 1 warning: High-risk area, triggered by crack width >5mm or settlement >10cm (measured by lidar and high-resolution camera), it is recommended to immediately close the road and implement emergency repairs.

[0103] ②Level 2 warning: Medium-risk area, triggered by crack width of 2-5mm or settlement of 5-10cm. It is recommended to carry out key monitoring and maintenance in the near future.

[0104] ③Level 3 warning: low-risk area, triggered by crack width <2mm or settlement <5cm, it is recommended to be included in the long-term observation plan.

[0105] Warning information is transmitted to the monitoring center via wireless communication modules and simultaneously pushed to relevant management departments. The system also automatically generates an emergency response plan, including damage area demarcation, equipment requirements, emergency repair plans, and recommendations for further detection. It also activates the disaster linkage detection function, where drones and vehicle-mounted equipment collaborate to conduct additional detection and verification of high-risk areas.

[0106] Afterwards, the system generates a visual three-dimensional imaging report, including: three-dimensional models of shallow and deep structures, disease distribution maps, risk level assessment maps, and future disease development trend prediction maps. At the same time, the system compares current inspection data with historical inspection records to analyze disease change trends. The report is stored in the form of an interactive electronic file to facilitate subsequent query and analysis.

[0107] The present invention proposes an air-ground collaborative multi-dimensional intelligent inspection and detection system for highway infrastructure defects.

[0108] The specific working process is as follows Figure 3 As shown, drones coordinate operations, performing dual-mode scanning simultaneously. In road mode, wide-area inspections are performed using visible light, infrared, and lidar. When road surface anomalies such as cracks or voids are detected, the onboard system (onboard seismic detection module and onboard 3D radar detection module) is dispatched for scanning using the vehicle's radar, and the seismic system performs in-depth verification. If a defect is confirmed, a 3D road report is generated. If no defect is confirmed, continued road monitoring is performed. In slope mode, slope scanning is performed using multispectral and oblique photography. When slope anomalies such as cracks or water are detected, the onboard system (drone landing platform and onboard detection module) is dispatched, including the airborne ground-penetrating radar, which uses lidar for in-depth modeling and slope ground-penetrating radar for detection. If the risk assessment results in a high risk, the slope reinforcement plan is initiated. If the risk assessment results in a low risk, continued slope monitoring is performed. Regardless of the mode, data is aggregated and fed into the comprehensive decision-making platform of the air-ground data fusion center. Processing is then performed based on the correlation between the anomalies. Specifically, if the anomaly is related to a road slope, a joint review will be initiated, using drones and vehicles for collaborative re-survey to update the existing road slope coupling model. If the anomaly is independent, separate processing will be carried out.

[0109] In this embodiment, the vehicle-mounted 3D radar detection module, equipped with ground-penetrating radars of varying frequencies, detects structures below the highway's surface. The vehicle-mounted seismic detection module detects changes in the highway's deeper structures. The data transmission and processing module integrates and processes multiple sources of detection data, including those from airborne and vehicle-mounted equipment. Through an air-ground collaborative multi-dimensional detection architecture, it integrates high-precision drone sensors (infrared, lidar, and ground-penetrating radar) with vehicle-mounted 3D radar and seismic detection systems. This intelligent fusion of multi-source data, combined with an axial attention mechanism and a differentiable morphological residual module, addresses the industry challenges of low inspection efficiency, shallow detection depth, and high false positive rates associated with traditional patrol inspections. Leveraging multi-physics field joint inversion and cross-scale neural radiation field modeling, the system generates 0.1-meter-resolution 3D disease cloud maps and 30-day evolution forecasts. Furthermore, it establishes a dynamic three-level early warning system, significantly reducing maintenance costs and providing an intelligent solution for safe highway operation and maintenance throughout its lifecycle.

[0110] Example 2: A second embodiment of the present invention provides a system method for collaborative air-ground inspection and detection of highway infrastructure defects, including the following steps: Collect highway infrastructure data from different dimensions; A multimodal neural network is used to align and extract highway infrastructure data of different dimensions in time and space, and the extracted features are fused by combining a cross-modal attention mechanism and a multi-scale feature fusion strategy. Based on the characteristics of data processing module fusion, multi-physical field joint inversion is performed to construct a three-dimensional model of highway infrastructure, and disease identification and early warning are carried out based on the three-dimensional model of highway infrastructure.

[0111] The steps involved in the above embodiment 2 correspond to those in embodiment 1. For the specific implementation method, please refer to the relevant description part of embodiment 1.

[0112] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0113] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A highway infrastructure disease air-ground collaborative inspection and detection system, characterized by: include: Data collection module, used to collect highway infrastructure data from different dimensions; The data processing module is used to align and extract highway infrastructure data of different dimensions in time and space using a multimodal neural network, and to fuse the extracted features using a cross-modal attention mechanism and a multi-scale feature fusion strategy; The three-dimensional imaging module is used to perform multi-physics field joint inversion based on the features fused by the data processing module, construct a three-dimensional model of highway infrastructure, and perform disease identification and early warning based on the three-dimensional model of highway infrastructure.

2. The highway infrastructure defect air-ground collaborative inspection and detection system according to claim 1 is characterized in that: The data acquisition module includes a UAV take-off and landing platform and an airborne detection module, a vehicle-mounted three-dimensional radar detection module and a vehicle-mounted seismic detection module. The UAV take-off and landing platform and the airborne detection module are used for the take-off and landing of the UAV and for carrying detection equipment for detection. The vehicle-mounted three-dimensional radar detection module is used to detect the structure below the shallow surface of the highway pavement. The vehicle-mounted seismic detection module is used to verify and further detect the detection results of the vehicle-mounted three-dimensional radar detection module.

3. The highway infrastructure defect air-ground collaborative inspection and detection system according to claim 1 is characterized in that: The drone take-off and landing platform and airborne detection module include a drone take-off and landing platform module and a drone module. The drone take-off and landing platform module is used for the drone's take-off and landing, as well as autonomous planning of inspection routes, and dynamically adjusts the flight route based on real-time collected data. The drone module includes a drone and detection equipment carried by the drone. The drone can achieve different detection purposes by carrying different types of detection equipment.

4. The highway infrastructure defect air-ground collaborative inspection and detection system according to claim 1 is characterized in that: The vehicle-mounted 3D radar detection module uses ground-penetrating radars of different frequencies to analyze the reflection characteristics of underground structures, thereby generating a 3D image below the surface.

5. The highway infrastructure defect air-ground collaborative inspection and detection system according to claim 1 is characterized in that: The vehicle-mounted seismic detection module arranges active excitation source devices and seismic detectors at the front and bottom of the inspection vehicle, and adjusts the source frequency and energy output according to different strata to accurately restore the spatial position of underground abnormal targets.

6. The highway infrastructure defect air-ground collaborative inspection and detection system according to claim 1 is characterized in that: It also includes a data transmission module for pre-processing the data collected by the data acquisition module and transmitting it to the data processing module in real time.

7. The highway infrastructure defect air-ground collaborative inspection and detection system according to claim 6 is characterized in that: The multi-scale feature fusion strategy introduces a pyramid structure and a skip connection mechanism. The pyramid structure is used to dynamically adjust the downsampling rate, and the skip connection mechanism is embedded with electromagnetic filtering features. It can simultaneously retain the coarse-grained global features and fine-grained local features of the data, reduce cross-modal alignment errors, and enhance the model's target recognition capabilities at different scales.

8. The highway infrastructure defect air-ground collaborative inspection and detection system according to claim 1 is characterized in that: Based on the results of multi-physics field joint inversion, a three-dimensional model of highway infrastructure is obtained by cross-scale voxel modeling at the macro, meso and micro levels.

9. The highway infrastructure defect air-ground collaborative inspection and detection system according to claim 1, characterized in that: The three-dimensional imaging module includes a disaster warning module, which triggers a three-level warning mechanism based on the disease identification results, and provides prompts and warnings to the identified high-risk areas, medium-risk areas or low-risk areas respectively.

10. A highway infrastructure disease air-ground collaborative inspection and detection system method, characterized by: The following steps are involved: Collect highway infrastructure data from different dimensions; A multimodal neural network is used to align and extract highway infrastructure data of different dimensions in time and space, and the extracted features are fused by combining a cross-modal attention mechanism and a multi-scale feature fusion strategy. Based on the characteristics of data processing module fusion, multi-physical field joint inversion is performed to construct a three-dimensional model of highway infrastructure, and disease identification and early warning are carried out based on the three-dimensional model of highway infrastructure.

Images