Bridge damage detection method and system

Abstract

This invention discloses a bridge damage detection method and system, belonging to the field of bridge engineering technology. The method obtains high-resolution images, three-dimensional point cloud data, and spectral data of the bridge to be inspected; inputs the obtained high-resolution images, three-dimensional point cloud data, and spectral data into the input module of a constructed bridge damage detection model to determine the image feature vector, the point cloud global feature vector, and the spectral feature vector; the constructed bridge damage detection model's feature fusion module fuses the image feature vector, the point cloud global feature vector, and the spectral feature vector, and interacts and fuses the feature representations of each fused modality at different levels to obtain a fused feature vector. The bridge damage detection type is output by a classifier at the DNN classification layer. This method can improve the accuracy of bridge damage detection.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge engineering, and more particularly to a bridge damage detection method and system. Background Art

[0002] With the continuous development of transportation infrastructure, the safety and reliability of bridges, as important transportation hubs, are receiving increasing attention. Bridges are complex structures and subject to variable stresses, making them susceptible to various damages such as cracks, corrosion, and deformation after prolonged use. Therefore, regular inspection and maintenance of bridges are particularly important.

[0003] Traditional bridge inspection methods rely heavily on manual inspections, which are not only time-consuming and labor-intensive, but also susceptible to human error in determining the accuracy and reliability of inspection results. Furthermore, manual inspections often make it difficult to access certain critical parts of a bridge, making it difficult to detect potential safety hazards in a timely manner.

[0004] With the advancement of science and technology, the rapid development of drone technology has provided a new solution for bridge inspection. Drones offer advantages such as high flexibility, ease of operation, and low cost. They can quickly cover various parts of a bridge and capture high-resolution images and videos, greatly improving inspection efficiency and coverage.

[0005] However, existing technologies rely solely on drones to obtain data from a single sensor. By extracting key features from the data from a single sensor, they are unable to comprehensively obtain information about the bridge's various states. They can only detect surface damage and cannot accurately identify internal damage to the bridge, affecting the accuracy of bridge damage detection and making it impossible to achieve comprehensive inspection of the bridge structure. Summary of the Invention

[0006] In response to the problems existing in the above-mentioned fields, the present invention proposes a bridge damage detection method and system. Through multi-angle and multi-dimensional data collection of the bridge, various status information of the bridge can be obtained more comprehensively. The constructed bridge damage detection model can accurately identify the internal damage of the bridge, improve the accuracy of bridge damage detection, and thus comprehensively evaluate the health status of the bridge.

[0007] To solve the above technical problems, the present invention discloses a bridge damage detection method, comprising the following steps:

[0008] The drone system autonomously navigates to collect high-resolution images of the bridge to be inspected, identifying surface defects such as cracks, rust, and surface damage. The LiDAR device generates three-dimensional point cloud data to provide information on the bridge's geometry and structure, identifying deformation, displacement, and geometric defects. Infrared cameras capture the bridge's temperature distribution data to detect internal temperature anomalies caused by corrosion, cracks, or material defects. Spectral data provided by multispectral or hyperspectral sensors is used to analyze changes in the chemical composition of the bridge's materials, including coating aging and corrosion reactions.

[0009] Constructing a network model; the network model includes an input module and a feature fusion module; introducing multiple cross-attention modules and a multimodal Transformer network into the feature fusion module; each cross-attention module fuses the feature vectors of each modality corresponding to the high-resolution image, three-dimensional point cloud data, and spectral data through a cross-attention mechanism and a Transformer self-attention mechanism, obtaining a feature representation of the fused feature vectors of each modality, inputting the feature representation into a DNN classification layer, and outputting a classification result; training the constructed network model to obtain a bridge damage detection model;

[0010] The high-resolution image, 3D point cloud data, and spectral data of the bridge to be inspected are input into the input module of the bridge damage detection model to determine the image feature vector, the global feature vector of the point cloud, and the spectral feature vector. The feature fusion module fuses the image feature vector, the global feature vector of the point cloud, and the spectral feature vector. The feature representations of each modality obtained after fusion are interacted and fused at different levels to obtain the fused feature vector. The bridge damage detection type is output through the classifier in the DNN classification layer.

[0011] By introducing a recursive neural network, the image feature vector, the global feature vector of the point cloud, and the spectral feature vector are processed to extract the time series features from each modal data and obtain the temporal features of each modality;

[0012] The system introduces a time series prediction module and a dynamic update mechanism. Based on an extended state space model or a nonlinear dynamic system model, it inputs current and historical data, predicts future damage trends, and outputs real-time updated damage identification results. When new data is input, the dynamic update mechanism corrects the current damage identification results.

[0013] A damage accumulation model is introduced, which accumulates damage information in each time period. The damage information in each time period is processed through a weighted accumulation and attenuation mechanism to evaluate the long-term health status of the bridge, conduct trend analysis, and identify potential structural degradation trends. By observing the acceleration of crack propagation speed, the failure time of the bridge structure is predicted and preventive maintenance is recommended.

[0014] Preferably, determining the eigenvector of the image, the global eigenvector of the point cloud, and the spectral eigenvector specifically comprises the following steps:

[0015] The high-resolution image of the bridge to be inspected is preprocessed using a convolutional neural network (CNN) to extract high-level features from the image and convert them into fixed-size feature vectors. The extracted fixed-size feature vectors are then fed into the multimodal Transformer network of the feature fusion module as input for the image modality.

[0016] For 3D point cloud data, use PointNet, PointNet++, or other point cloud processing networks to learn the features of each point and generate a global feature vector for the point cloud through global feature aggregation. The generated global feature vector of the point cloud is input into the multimodal Transformer network of the feature fusion module.

[0017] The acquired spectral data of the bridge to be inspected is processed by a 1D convolutional neural network (1D-CNN) or a recurrent neural network (RNN / LSTM) to extract local features from the spectral data and obtain spectral feature vectors. The obtained spectral feature vectors are then input into the multimodal Transformer network of the feature fusion module.

[0018] Preferably, each cross-attention module fuses the feature vectors of each modality corresponding to the high-resolution image, three-dimensional point cloud data, and spectral data respectively through the cross-attention mechanism and the Transformer self-attention mechanism, specifically comprising the following steps:

[0019] After preliminary feature extraction of high-resolution images, 3D point cloud data, and spectral data, the extracted features are adaptively weighted using SE-Block or CBAM.

[0020] The multimodal Transformer network includes the Transformer self-attention mechanism, encoder layers, and positional encoding;

[0021] The Transformer self-attention mechanism, as a core component of the multimodal Transformer network, captures the correlation within and across modalities of feature vectors by calculating the relationship between queries, keys, and values. In the multimodal Transformer network, the feature vectors of each modality are mapped to query, key, and value vectors, and the similarity between features is calculated. The calculated similarities between features are weighted and summed to generate an enhanced feature representation.

[0022] The encoder layer consists of multiple stacked Transformer layers, each of which includes a Transformer self-attention mechanism and a feedforward neural network. The feature vector of each modality is passed and processed layer by layer in the encoder to capture the global context of the data from different modalities. The encoder output is a fused multimodal feature representation that retains the key information in the input data and captures the complementary characteristics between modalities through the Transformer attention mechanism to obtain the feature vector of each modality.

[0023] By introducing position encoding into the Transformer encoder, in the multimodal Transformer network, position encoding provides position information for the feature vector of each modality, obtaining the feature representation of each modality that is consistent in time and space.

[0024] Preferably, the determining of the image feature vector, the global feature vector and the spectral feature vector of the point cloud further comprises the following steps:

[0025] After integrating the channel attention mechanism SE-Block into the high-resolution image, 3D point cloud data, and spectral data feature extraction modules, the feature vectors of each modality are channel-weighted before being input into the Transformer encoder.

[0026] SE-Block compresses the spatial dimension of each feature map into a global feature representation through the Squeeze operation, learns the correlation between channels through the Excitation operation, and adaptively reweights the features of each channel.

[0027] Preferably, the step of obtaining the fused feature representation of each modality feature vector comprises the following steps:

[0028] After being processed by multiple stacked Transformer layers of the Transformer self-attention mechanism, the acquired temporally and spatially consistent feature representations of each modality are interacted and fused at different levels. Feature fusion is performed using feature concatenation, weighted summation, or Transformer layers to generate a fused feature vector that integrates key information from different modalities to form a multimodal joint representation.

[0029] Preferably, the output classification result specifically includes:

[0030] The generated multimodal joint representation is processed through multiple fully connected layers of the DNN classification layer, and classified, regressed or detected through the softmax classifier to output the type or location of the bridge damage to be detected.

[0031] Preferably, a bridge damage detection system is further included, comprising:

[0032] The data acquisition module is used to collect high-resolution images of the bridge to be inspected through autonomous navigation of the UAV system and identify defects on the bridge surface, including cracks, rust, and surface damage. The module uses LiDAR equipment to generate three-dimensional point cloud data to provide information on the bridge's geometry and structure, and to identify deformation, displacement, and geometric defects. The module uses infrared cameras to capture the bridge's temperature distribution data and detect internal temperature anomalies caused by corrosion, cracks, or material defects. The module uses spectral data provided by multispectral or hyperspectral sensors to analyze changes in the chemical composition of the bridge's materials, including coating aging and corrosion reactions.

[0033] A bridge damage detection model construction module is used to construct a network model; the network model includes an input module and a feature fusion module; multiple cross-attention modules and a multimodal Transformer network are introduced into the feature fusion module. Each cross-attention module fuses the feature vectors of each modality corresponding to the high-resolution image, three-dimensional point cloud data, and spectral data through the cross-attention mechanism and the Transformer self-attention mechanism, obtains the feature representation of the fused feature vectors of each modality, inputs it into the DNN classification layer, and outputs the classification result; the constructed network model is trained to obtain a bridge damage detection model;

[0034] The bridge damage detection module is used to input the acquired high-resolution image, three-dimensional point cloud data and spectral data of the bridge to be detected into the input module of the bridge damage detection model to determine the image feature vector, the global feature vector of the point cloud and the spectral feature vector; the feature fusion module fuses the feature vector of the image, the global feature vector of the point cloud and the spectral feature vector, and obtains the fused feature vector by interacting and fusing the feature representations of each modality obtained after fusion at different levels, and outputs the bridge damage detection type through the classifier in the DNN classification layer; by introducing a recursive neural network, the feature vector of the image, the global feature vector of the point cloud and the spectral feature vector are processed to extract the time series features from each modal data. The system uses a time series prediction module and a dynamic update mechanism to input current and historical data based on an extended state space model or a nonlinear dynamic system model, predict future damage trends, and output real-time updated damage identification results. When new data is input, the current damage identification results are corrected through the dynamic update mechanism. A damage accumulation model is introduced, which accumulates damage information in each time period. The damage information in each time period is processed through a weighted accumulation and attenuation mechanism to evaluate the long-term health status of the bridge, conduct trend analysis, and identify potential structural degradation trends. By observing the acceleration of crack propagation speed, the failure time of the bridge structure is predicted, and preventive maintenance is recommended.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The bridge damage detection method proposed in this paper integrates multiple sensors to collect data from multiple angles and dimensions, enabling more comprehensive information on the bridge's status. The constructed bridge damage detection model effectively fuses the collected heterogeneous sensor data from multiple sources. This fusion process forms a comprehensive assessment of the bridge's health. Feature extraction from high-resolution bridge images, three-dimensional point cloud data, and spectral data is combined to analyze the bridge's multidimensional features, enabling comprehensive inspection and accurate identification of the bridge structure. By introducing multiple cross-attention modules and a multimodal Transformer network into the feature fusion module, feature extraction and fusion of multimodal data address data heterogeneity, extracting key features from image, point cloud, and spectral data to ensure the accuracy and reliability of the fusion results. This provides more accurate and comprehensive data support for autonomous flight and intelligent obstacle avoidance for drones. The bridge damage detection model dynamically adjusts the focus on different damage features when inspecting the acquired bridge data. By generating detailed inspection reports that include defect location, type, or severity, it enables accurate diagnosis and prediction of bridge health. The method proposed in the present invention improves the accuracy of bridge damage detection and can accurately detect the health status of bridges, providing useful reference and suggestions for researchers and engineering practitioners in the field of infrastructure engineering. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is a flow chart of the bridge damage detection method proposed in the present invention;

[0038] Figure 2 A network architecture diagram of the bridge damage detection model constructed for the present invention;

[0039] Figure 3 Schematic diagram of an environmental perception-based UAV platform provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following is a combination of the embodiments of the present invention Figure 1-Figure 3 , the technical solutions in the embodiments of the present invention are clearly and completely described. It should be understood that the terms used in the present invention are only used to describe specific implementation methods and are not intended to limit the present invention.

[0041] Example

[0042] like Figure 1 As shown, the present invention proposes a bridge damage detection method, which includes the following steps:

[0043] S1: Use the autonomous navigation of the UAV system to collect high-resolution images of the bridge to be inspected and identify surface defects, including cracks, rust, and surface damage. Use the LiDAR device to generate 3D point cloud data to provide information on the bridge's geometry and structure, and identify deformation, displacement, and geometric defects. Use infrared cameras to capture the bridge's temperature distribution data and detect internal temperature anomalies caused by corrosion, cracks, or material defects. Use spectral data provided by multispectral or hyperspectral sensors to analyze changes in the chemical composition of the bridge's materials, including coating aging and corrosion reactions.

[0044] S2: Construct a bridge damage detection model;

[0045] Constructing a network model; the network model includes an input module and a feature fusion module; introducing multiple cross-attention modules and a multimodal Transformer network into the feature fusion module; each cross-attention module fuses the feature vectors of each modality corresponding to the high-resolution image, three-dimensional point cloud data, and spectral data through a cross-attention mechanism and a Transformer self-attention mechanism, obtaining a feature representation of the fused feature vectors of each modality, inputting the feature representation into a DNN classification layer, and outputting a classification result; training the constructed network model to obtain a bridge damage detection model;

[0046] S3: The high-resolution image, three-dimensional point cloud data, and spectral data of the bridge to be inspected are input into the input module of the bridge damage detection model to determine the image feature vector, the global feature vector of the point cloud, and the spectral feature vector. The feature fusion module fuses the image feature vector, the global feature vector of the point cloud, and the spectral feature vector. The feature representations of the fused modalities are interacted and fused at different levels to obtain the fused feature vector. The bridge damage detection type is output through the classifier in the DNN classification layer.

[0047] In step S1, a UAV platform with autonomous flight and intelligent obstacle avoidance functions is used. Figure 3 As shown, it is equipped with multiple sensors such as high-definition visual images, thermal imagers, lidar, multispectral sensors, etc. to achieve efficient and safe bridge inspections, especially at high altitudes and in complex environments.

[0048] The high-definition camera carried by the drone platform obtains detailed images of the bridge, the thermal imager obtains the internal damage of the bridge, the lidar provides accurate three-dimensional structural data, and the multispectral sensor detects the characteristics and potential damage of the material.

[0049] The automated flight and data collection capabilities of the drone platform allow for comprehensive inspection coverage of all bridge sections. The diverse configuration of sensor integration modules enables the system to operate efficiently in diverse environmental conditions.

[0050] The present invention provides an intelligent bridge damage detection method based on multi-source data fusion and UAV, such as Figure 1 As shown, the following steps are included:

[0051] The drone platform is equipped with a variety of sensors such as high-definition cameras, thermal imagers, lidar, etc. for data acquisition, color recognition and distance measurement, thereby achieving autonomous navigation and data collection.

[0052] High-definition cameras capture detailed images of the bridge, thermal imagers detect internal damage, millimeter-wave radar identifies distant obstacles, ultrasonic radar identifies close-up obstacles, lidar provides precise 3D structural data, and multispectral sensors detect material properties and potential damage. By fusing this multi-sensor data, the controller can execute decisions, achieving comprehensive perception and detailed modeling of the bridge.

[0053] To enhance environmental perception and obstacle avoidance capabilities, the system captures color and depth images based on the Intel RealSense D435 camera for precise environmental perception. The color images are fed into the YOLO V8 model for obstacle classification and bounding box generation, enabling rapid identification and location of key obstacles.

[0054] Due to its advantages in speed, accuracy, and computational efficiency, the YOLO V8 model is particularly suitable for resource-constrained environments such as embedded drone systems such as drones. YOLO V8 optimizes the model architecture and enhances multi-scale feature extraction capabilities to better detect targets of different sizes and distances. Specifically, the following steps are included:

[0055] First, the YOLOv8 model introduces an enhanced Feature Pyramid Network (FPN), which strengthens the model's ability to detect objects of varying scales. YOLOv8's improvements to FPN go beyond simply fusing features from low-level to high-level layers. Instead, it utilizes more skip connections to enable efficient interaction between features of different scales. This design increases the flexibility of the feature pyramid and better captures the details of small objects. Furthermore, by extracting features at different scales, YOLOv8 is better able to handle objects with large scale variations, resulting in superior performance when detecting distant or tiny objects. The improved FPN is able to better process features at different levels, preserving more contextual information through layer-by-layer transmission and fusion.

[0056] Secondly, YOLOv8 optimizes the Path Aggregation Network (PAN), a module used to enhance the propagation of contextual information. In YOLOv8, PAN not only considers bottom-up path aggregation but also introduces top-down paths, making the flow of information within the model smoother and improving detection performance. Furthermore, by introducing a more complex path structure, PAN is able to more effectively propagate contextual information, especially when dealing with complex backgrounds. This improvement significantly improves the model's robustness and detection accuracy.

[0057] In a depth image, the grayscale value of each pixel directly represents the depth at that pixel (i.e., the distance from the object's surface to the camera). When detecting obstacles, a bounding box is typically generated for the obstacle, encompassing all pixels of the object. The most reliable depth data typically comes from the center of the bounding box (Pc), whose grayscale value generally represents the average distance from the obstacle to the camera. However, due to variations in the object's shape and pose, or environmental conditions such as lighting and occlusion, the bounding box may deform, resulting in distorted depth data at the center. To compensate for potential errors, in addition to the bounding box center (Pc), the object's center of gravity (Pg) is also considered. This center of gravity (Pg) is calculated by taking the weighted average of all pixels within the obstacle region and generally better represents the obstacle's actual location and depth. Furthermore, averaging the depth values ​​within and around the bounding box, or applying a local weighted average, can reduce distortion caused by individual pixels (possibly due to noise or errors). Therefore, it is necessary to combine the center of gravity calculation with the regional depth mean calculation to optimize the depth data, correct the depth of the obstacle, obtain more reliable depth data, ensure that the UAV can accurately perceive the position of obstacles in complex environments, and formulate effective obstacle avoidance strategies.

[0058] By fusing the target detection results with the optimized depth image, the drone system can extract the outline and three-dimensional spatial information of obstacles, greatly improving detection accuracy and providing detailed data for subsequent obstacle avoidance decisions. By integrating this data, the drone system generates an optimal obstacle avoidance strategy, taking into account factors such as obstacle type, profile characteristics, and location distribution. It calculates the obstacle avoidance trajectory that minimizes distance, ensuring that the drone avoids obstacles in the safest way and successfully completes its mission.

[0059] After combining the target detection results with the optimized depth image, the UAV system can effectively extract the outline and three-dimensional spatial information of the obstacle, thereby greatly improving the detection accuracy and providing detailed data support for subsequent obstacle avoidance decisions.

[0060] The YOLOv8 model is used to detect the category and location of obstacles in the RGB image and generate corresponding bounding boxes. These bounding boxes contain the coordinate information of the obstacles in the two-dimensional image. These bounding boxes are then mapped to the corresponding depth image. In the depth image, the grayscale value of each pixel represents the depth information at that location, that is, the distance from the object surface to the camera. By extracting depth information from the pixels within these bounding boxes, the UAV system can fuse the target detection results with the three-dimensional spatial information in the depth image. This fusion process can be performed by taking a weighted average of the pixels within the bounding box or other statistical methods to extract the average depth value of the area, which better reflects the actual spatial position and size of the obstacle.

[0061] The drone system extracts the outlines of obstacles by applying edge detection algorithms (such as Canny edge detection or Sobel operator) to the optimized depth image. These algorithms can identify areas in the image where the depth values ​​change significantly, that is, the boundaries of the object. The extracted outlines can be combined with the bounding box of the target detection to further accurately locate the edges of the obstacle, ensuring that the drone system can accurately identify the shape and boundaries of the obstacle. The drone system combines the outline information and depth data to convert the two-dimensional image information of the obstacle into three-dimensional space coordinates through techniques such as triangulation, voxel modeling, or point cloud generation. Specifically, a three-dimensional model of the obstacle can be constructed by sampling multiple points within the bounding box area and based on the depth values ​​of these points.

[0062] The drone system integrates the category information, depth information, contour information, and position and shape information of obstacles. Category information refers to the obstacle category (such as trees, buildings, pedestrians, etc.) extracted from the target detection results; depth information refers to the obstacle distance data obtained from the depth image, which includes the depth value of each pixel and the regional depth information after weighted averaging or other processing. Contour information is the obstacle boundary outline extracted through edge detection, and is combined with depth information for three-dimensional modeling and obstacle avoidance path planning. Position and shape information refers to the exact position and shape of the obstacle in space extracted through three-dimensional modeling and point cloud generation. The fusion of these data enables the drone system to fully understand the spatial distribution and dynamic characteristics of obstacles, providing detailed data support for subsequent obstacle avoidance decisions.

[0063] UAV systems use deep learning-based path planning algorithms to generate optimal obstacle avoidance strategies. These algorithms typically include, but are not limited to, convolutional neural networks (CNNs), reinforcement learning (RL), and generative adversarial networks (GANs). The specific algorithm chosen depends on the complexity of the mission and environmental conditions.

[0064] Reinforcement learning methods are particularly suitable for obstacle avoidance tasks in dynamic environments. By training in a simulated environment, reinforcement learning algorithms can learn the best obstacle avoidance strategy, that is, to safely avoid obstacles while minimizing the path length. Path planning algorithms such as the A algorithm, Dijkstra algorithm, RRT (rapidly expanding random trees), etc., can also be used for obstacle avoidance path planning in static environments. The drone system will consider the category of obstacles, profile characteristics (such as height, width, etc.) and position distribution in the environment, and calculate an obstacle avoidance trajectory that minimizes the distance. These algorithms combine perception data and environmental models, and through a comprehensive evaluation of possible paths, select the optimal path to ensure that the drone can avoid obstacles in the safest way and perform its mission smoothly.

[0065] This enables the drone system to accurately perceive and avoid obstacles in complex and dynamic environments, ensuring safe and efficient mission execution. This combination of multi-data fusion and deep learning algorithms empowers the drone with intelligent obstacle avoidance capabilities, providing strong support for autonomous flight and mission execution. Using the mission control drone system and custom scripts, the drone executes straight-line flight missions between multiple mission points while autonomously avoiding detected obstacles. These custom scripts utilize software packages such as ROS, MAVROS, and OpenCV to optimize images, run deep learning algorithms, generate obstacle avoidance strategies, and plan flight paths.

[0066] During flight, different types of obstacles pose varying threats to flight safety, necessitating corresponding avoidance strategies. Using sensors onboard the drone (such as a depth camera or LiDAR), the drone system can acquire depth data within its forward field of view. The depth value of each pixel represents its distance from the drone. Based on these depth values, the drone system divides the space within the field of view into four zones: clear sky, warning, action, and emergency. The clear sky zone indicates there are no obstacles ahead, allowing safe flight. The warning zone detects potential risks but is uncertain of their specific location, requiring the drone to reduce speed and maintain continuous monitoring. The action zone requires the drone system to take specific avoidance actions based on the detected obstacle's properties. The emergency zone requires an immediate emergency stop to ensure flight safety. Based on the drone's flight speed and acceleration, as well as the properties of the obstacle (e.g., dynamic or static), the drone system can adjust the distance thresholds for these zones in real time to ensure optimal obstacle avoidance performance in different environments.

[0067] Through this zoning management and avoidance strategy, the drone can adapt to different environmental conditions, reduce collision risks, and ensure the successful completion of the mission. To monitor flight status in real time, QGroundControl ground control station software is used to observe and record flight parameters and adjust mission parameters to ensure safe and accurate flight execution.

[0068] The present invention supports intelligent decision-making and analysis in complex environments by extracting and fusing key information from multiple sensor data.

[0069] Based on a deep convolutional autoencoder (CAE), its core consists of two parts: an encoder and a decoder. The encoder maps multimodal input data into a low-dimensional latent space and extracts key features. The decoder reconstructs the original data from the latent space, preserving the data's key information while eliminating noise and errors.

[0070] To align data from different sensors, the encoder incorporates multiple parallel convolutional layers. These layers capture and align low-level features from different modalities. Through weight-sharing or cross-modal convolution operations, they map the data from different modalities into a unified feature space. Through multiple layers of convolution and pooling, the encoder compresses and maps the data from each modality into a low-dimensional latent space. This process not only aligns the data but also preserves key structural information from each modality. During the encoding process, the filters in the convolutional layers automatically learn to identify and suppress noise. By introducing noisy samples during training, the deep convolutional autoencoder effectively recovers a noise-free signal from noisy data. In the latent space representation, the noise is compressed into small perturbations. Through deconvolution operations in the decoder, the UAV system removes this noise when reconstructing the data, ensuring clarity and accuracy of the output. To eliminate numerical variations in the data from different sensors, the encoder incorporates batch normalization or layer normalization. These operations standardize the data in the latent space, ensuring that multimodal data is analyzed at the same scale. During the decoder reconstruction process, the UAV system can restore the original numerical characteristics of the data through the inverse normalization operation, so that the processed data maintains consistency while retaining its unique quantitative characteristics.

[0071] The decoder consists of a series of deconvolutional layers, which remap the low-dimensional representation in the latent space back to the original data space. In this process, the decoder not only reconstructs the primary structure of the data but also further optimizes its quality. The autoencoder is trained by minimizing the reconstruction error, typically using the mean squared error (MSE) as a loss function. During training, the UAV system continuously adjusts the encoder and decoder weights to ensure that the final output data, after calibration, denoising, and normalization, is as accurate and consistent as possible.

[0072] Through these preprocessing steps, a high-quality multimodal dataset was constructed, laying a solid foundation for subsequent feature extraction and detection. By combining autoencoders and attention mechanisms, key features from multimodal data were accurately extracted and fused, significantly improving the drone system's ability to process heterogeneous data in complex environments. Combining the Transformer attention mechanism with an adaptive weighting strategy dynamically weights the importance of different sensor data, ensuring the accuracy and reliability of the fusion results.

[0073] A deep convolutional autoencoder performs nonlinear transformations on data from various sensors. The encoder portion of the autoencoder, comprised of a multi-layer convolutional network, progressively extracts low-level to high-level features from the data. For data of different modalities (such as images, point clouds, and spectral data), the autoencoder generates latent feature representations that compress redundant information in the original data while preserving key structural features.

[0074] Through this nonlinear transformation, the UAV system can effectively map various sensor data into a unified feature space. In this feature space, the features of different modalities have been adjusted to the same scale and distribution, laying the foundation for the subsequent fusion process.

[0075] In order to adaptively adjust the fusion parameters of various sensor data during the data fusion process, an optimization strategy based on reinforcement learning is introduced into the UAV system. The specific implementation process is as follows:

[0076] The policy function learns to determine the actions to take under different conditions in order to maximize future rewards. The reward function evaluates the performance of each action under specific conditions, assigning rewards or penalties based on metrics such as the accuracy of the fusion results, the degree of redundancy reduction, and the effectiveness of conflict resolution. Through continuous interaction with the environment, the UAV system gradually optimizes the policy function and learns the optimal fusion parameter combinations for different environments. Ultimately, the UAV system is able to dynamically select the optimal parameters in practical applications to respond to environmental changes. In practical applications, the UAV system possesses adaptive adjustment capabilities, dynamically adjusting fusion parameters based on the current environmental state, ensuring high-precision data to support tasks such as autonomous flight and intelligent obstacle avoidance.

[0077] The Wasserstein GAN (WGAN) is used to improve training stability and ensure the reliability of image enhancement results. First, WGAN guides generator optimization by calculating the Wasserstein distance between the generated distribution and the true distribution. Compared to the traditional JS divergence (Jensen-Shannon Divergence), the Wasserstein distance still provides meaningful gradient information even when the distributions do not completely overlap. Therefore, by optimizing the Wasserstein distance, WGAN ensures that the generator can consistently learn to produce high-quality enhanced images that are similar to real images. Second, in WGAN, the traditional discriminator is replaced with a "critic." Instead of a binary classification task, it directly outputs a real number representing the position of the input sample in the Wasserstein distance. By optimizing the critic, WGAN can stably converge to the true distribution. By learning the gradient of the Wasserstein distance, the critic network can more accurately assess the difference between the generated and real images. This assessment not only helps the generator improve the quality of generated images but also provides more meaningful gradient information, making the training process more stable.

[0078] SRGAN focuses on converting low-resolution images into high-resolution ones. Specifically, with the support of the ResNet architecture, it can preserve image details and improve the resolution of damaged features. SRGAN combines a pixel-level loss function with a perceptual loss to further optimize image clarity and naturalness by comparing the differences between the generated and real images in feature space.

[0079] Furthermore, a diffusion model is introduced to generate high-quality enhanced images by gradually adding noise and learning a denoising process. This model is particularly adept at removing noise while preserving image structure, making details such as cracks and corrosion in bridge images more distinct. Compared to traditional GANs and SRGANs, the diffusion model excels in generating natural and high-fidelity images.

[0080] Throughout the image enhancement process, convolutional neural network (CNN) layers and batch normalization techniques were integrated to optimize image enhancement stability and efficiency. Ultimately, these deep learning algorithms significantly improved the resolution and clarity of the enhanced images while also effectively reducing redundant information, providing a more accurate and reliable data foundation for subsequent damage identification and diagnosis.

[0081] In step S2, the present invention adopts advanced computer vision technology and deep learning models to build a bridge damage detection model and comprehensively analyze the multi-dimensional characteristics of the bridge to achieve comprehensive detection and accurate identification of the bridge structure.

[0082] like Figure 2 As shown in Figures (a) and (b) in the figure, the image feature vector, the global feature vector of the point cloud, and the spectral feature vector are determined through the input module and used as the input of the multimodal Transformer network. The specific steps include:

[0083] The high-resolution image of the bridge to be inspected is preprocessed using a convolutional neural network (CNN) to extract high-level features of the image and convert them into a fixed-size feature vector, namely an I-vector sequence. The extracted fixed-size feature vector is input into a multimodal Transformer network as the input of the image modality.

[0084] For three-dimensional point cloud data, use PointNet, PointNet++ or other point cloud processing networks to learn the features of each point and generate the global feature vector of the point cloud, that is, the P vector sequence, through global feature aggregation; the generated global feature vector of the point cloud is input into the multimodal Transformer network.

[0085] The acquired spectral data of the bridge to be inspected is processed by a 1D convolutional neural network (1D-CNN) or a recurrent neural network (RNN / LSTM) to extract local features from the spectral data and obtain a spectral feature vector, i.e., an S-vector sequence. The obtained spectral feature vector is then input into a multimodal Transformer network.

[0086] Each cross-attention module fuses the feature vectors of each modality corresponding to image data, point cloud data, and spectral data through the cross-attention mechanism and the Transformer self-attention mechanism. Specifically, it includes the following steps:

[0087] The multimodal Transformer network includes a multi-head self-attention mechanism, an encoder layer, and position encoding;

[0088] Among them, the Transformer self-attention mechanism, as the core component of the multimodal Transformer network, captures the correlation within the feature vector and across modalities by calculating the relationship between Query, Key and Value; in the multimodal Transformer network, the feature vector of each modality is mapped to Query, Key and Value vectors, and the similarity between features is calculated; the calculated similarities between features are weighted and summed to generate an enhanced feature representation.

[0089] The encoder layer consists of multiple stacked Transformer layers, each of which includes a Transformer self-attention mechanism and a feedforward neural network. The feature vector of each modality is transmitted and processed layer by layer in the encoder to capture the global context information of data in different modalities. The output of the encoder is a fused multimodal feature representation, which retains the key information in the input data and captures the complementary characteristics between modalities through the Transformer attention mechanism to obtain the feature vector of each modality.

[0090] Since the Transformer model itself does not have the ability to process sequence information, position encoding is introduced in the Transformer encoder to help the model recognize sequence information. In the multimodal Transformer network, position encoding provides position information for each modal feature vector to obtain temporally and spatially consistent features of each modality.

[0091] Obtaining the fused feature vector specifically includes the following steps:

[0092] After being processed by multiple Transformer layers, the acquired temporally and spatially consistent features of each modality are interacted and fused at different levels. Feature fusion is performed using feature concatenation, weighted summation, or Transformer layers to generate a fused feature vector that integrates key information from different modalities to form a high-quality multimodal joint representation.

[0093] The resulting multimodal joint representation can be further processed through a series of fully connected layers to adapt to the needs of specific tasks (including classification, regression, detection, etc.). These layers typically include nonlinear activation functions and normalization layers to ensure the model's training stability and representational capabilities. Ultimately, the output layer outputs the corresponding prediction results (including the type and location of bridge damage, etc.) based on the requirements of the specific task.

[0094] During the training process of the constructed bridge damage detection model, loss functions suitable for the task are used, such as cross entropy loss (for classification tasks), mean square error (for regression tasks), etc.

[0095] To improve the effectiveness of multimodal data fusion, additional regularization terms, such as the Multimodal Alignment Loss, can be introduced to ensure consistency between the features of each modality during the fusion process. Common optimization algorithms include Adam and RMSprop, which are effective in handling large-scale parameter optimization in multimodal Transformer networks. During training, model performance can be further optimized by adjusting hyperparameters such as the learning rate and batch size, as well as by introducing a learning rate scheduler (such as Cosine Annealing).

[0096] This paper integrates adaptive attention mechanisms (such as SE-Block and CBAM) into the constructed bridge damage detection model to further enhance the model's feature representation capabilities, especially when handling complex structural damage identification tasks. These adaptive attention mechanisms dynamically adjust the model's attention to different damage features, allowing the network to more accurately focus on key structural defects, thereby improving the accuracy and reliability of damage identification.

[0097] SE-Block is a channel attention mechanism that can enhance the model's response to important features by adaptively reweighting channel features. In the bridge damage detection model, SE-Block can be integrated after the feature extraction module of image data, point cloud data, and spectral data, that is, before each modal feature vector is input into the Transformer encoder. By performing channel weighting on the feature vector of each modality, the response to key damage features is enhanced. SE-Block first compresses the spatial dimensions of each feature map into a global feature representation through the Squeeze operation. Then, through the Excitation operation, it learns the correlation between channels and adaptively reweights the features of each channel. This process can enhance the model's attention to important damage features, making the final generated feature representation more discriminative.

[0098] CBAM combines spatial attention and channel attention mechanisms to optimize feature representation in both spatial and channel dimensions, further improving the performance of bridge damage detection models in damage identification. CBAM can be integrated between Transformer encoder layers or after the feature extraction module for each modality.

[0099] Similar to SE-Block, CBAM can process the feature vectors of each modality to optimize its spatial and channel features. First, the feature map is reduced in dimension through global average pooling and global maximum pooling to generate two different channel attention maps. Then, these two attention maps are fused through MLP (Multi-layer Perceptron) to generate the final channel attention map, and the original feature map is weighted. After the channel attention module, CBAM processes the feature map in the spatial dimension, generates a spatial attention map through maximum pooling and average pooling operations, and further optimizes the channel-weighted feature map.

[0100] After initial feature extraction from image, point cloud, and spectral data, the extracted features are adaptively weighted using SE-Block or CBAM. This makes the features processed by the attention mechanism more representative, helping to capture key damage information in data from different modalities. CBAM modules can be inserted between each layer of the multi-layer Transformer encoder to further strengthen the spatial and channel representation of features. This optimizes the features through CBAM before they are processed by the Transformer self-attention mechanism, thereby increasing the overall network's sensitivity to damage signatures.

[0101] In step S3, during the bridge damage detection process, the combined feature representation processed by SE-Block and CBAM enables more accurate detection and classification of various bridge damages, such as cracks, corrosion, and deformation. With these enhanced feature representations, the model can generate more detailed and accurate damage reports, including the location, type, and severity of defects.

[0102] To further enhance the comprehensiveness and accuracy of detection, time series analysis technology was incorporated into multimodal data processing. By incorporating the time series characteristics of multimodal data, the drone system can dynamically capture changing trends in the bridge structure and update damage identification results in real time. This time series analysis not only enhances the drone system's ability to monitor bridge health but also provides important reference data for long-term bridge maintenance. The specific steps are as follows:

[0103] (1) Data acquisition and processing: The UAV system first collects data regularly from multiple sensors on the bridge (such as RGB cameras, LiDAR, infrared cameras, spectral sensors, etc.). The multimodal data collected each time forms a sample at a timestamp. Over time, the UAV system accumulates samples of multiple timestamps to form a time series data set. Before performing time series analysis, the multimodal data needs to be preprocessed. The preprocessing steps include: denoising, alignment, normalization, and feature extraction. In order to unify the time dimension of different modal data, it may be necessary to interpolate or sample the data to ensure that the data at each timestamp is homogeneous.

[0104] (2) Constructing a temporal feature extraction network: In order to extract temporal features, a recursive neural network (RNN) architecture is designed, which can process the temporal features of multimodal inputs. Unlike traditional RNNs, a multi-channel RNN may be used here, with each channel corresponding to one modal data. In this way, the UAV system can extract time series features from each modal data. After each modal data is processed by the RNN, the UAV system fuses the temporal features of each modality through a temporal feature fusion layer. This layer may use a weighted summation or attention mechanism (similar to the self-attention mechanism in the Transformer) to highlight the temporal features of the key modalities. In order to capture the changing trends at different time scales, a multi-scale convolutional layer (MSC) can be designed. This layer performs convolution operations on the time dimension through multiple convolution kernels of different sizes to extract multi-scale temporal features. In this way, the UAV system can capture both short-term sudden changes and long-term slow changes. In addition, a time progressive module can be introduced to identify key change nodes in the time series through sliding windows and gradual accumulation. This module can identify sudden changes in structural state within a specific time period and mark these changes as potential structural damage.

[0105] (3) Dynamic damage identification and updating: After extracting the time series features, the UAV system needs to predict the future structural state. The designed time series prediction module may be based on an extended state space model or a nonlinear dynamic system model. This module can predict future damage trends based on current and historical data. The output of the prediction module will be used to update the damage identification results in real time. Whenever new data is input into the UAV system, the UAV system will correct the current damage identification results through a dynamic update mechanism (such as Kalman filter or Bayesian update) to ensure that the output results reflect the latest structural state. In order to evaluate the long-term damage trend, the UAV system introduces a damage accumulation model that can accumulate damage information in each time period and evaluate the long-term health status of the structure through a weighted accumulation and decay mechanism (Decay Mechanism). Based on the damage accumulation model, the UAV system can perform trend analysis, identify potential structural degradation trends, and issue early warning signals when necessary. For example, the UAV system can predict the failure time of a bridge structure by observing the acceleration of crack propagation speed and recommend preventive maintenance.

[0106] (4) Implementation and optimization of the timing analysis system: The entire timing analysis system adopts a modular design and is mainly divided into modules such as timing feature extraction, dynamic modeling of timing data, dynamic update and trend analysis. The modules exchange data and communicate with each other through a unified interface to achieve overall functional optimization. By integrating adaptive optimization algorithms (such as genetic algorithms or particle swarm optimization algorithms), the parameters of the timing analysis model are tuned to improve the prediction accuracy and response speed of the model. In order to meet the requirements of real-time processing, the UAV system uses GPU acceleration and distributed computing technology to improve the processing speed. The timing processing process of multimodal data is designed as a parallel operation to maximize the response efficiency of the UAV system.

[0107] (5) Long-term monitoring and early warning: The application of this time series analysis technology can monitor the health status of bridge structures in real time, identify early signs of structural degradation, and provide targeted maintenance recommendations. Through time series analysis, the UAV system can predict the long-term change trends of bridge structures, provide scientific data support to bridge management departments, and optimize bridge maintenance and operation strategies.

[0108] Ultimately, the combined application of deep learning models with adaptive attention mechanisms, the Transformer architecture, and time series analysis effectively improved the comprehensiveness and accuracy of bridge structural damage detection. It not only identifies and annotates various structural defects, but also provides a more comprehensive analysis of bridge health, providing strong data support for bridge maintenance and management.

[0109] This application also proposes a bridge damage detection system, comprising:

[0110] The data acquisition module is used to collect high-resolution images of the bridge to be inspected through autonomous navigation of the UAV system and identify defects on the bridge surface, including cracks, rust, and surface damage. The module uses LiDAR equipment to generate three-dimensional point cloud data to provide information on the bridge's geometry and structure, and to identify deformation, displacement, and geometric defects. The module uses infrared cameras to capture the bridge's temperature distribution data and detect internal temperature anomalies caused by corrosion, cracks, or material defects. The module uses spectral data provided by multispectral or hyperspectral sensors to analyze changes in the chemical composition of the bridge's materials, including coating aging and corrosion reactions.

[0111] A bridge damage detection model construction module is used to construct a network model; the network model includes an input module and a feature fusion module; multiple cross-attention modules and a multimodal Transformer network are introduced into the feature fusion module. Each cross-attention module fuses the feature vectors of each modality corresponding to the high-resolution image, three-dimensional point cloud data, and spectral data through the cross-attention mechanism and the Transformer self-attention mechanism, obtains the feature representation of the fused feature vectors of each modality, inputs it into the DNN classification layer, and outputs the classification result; the constructed network model is trained to obtain a bridge damage detection model;

[0112] The bridge damage detection module is used to input the acquired high-resolution image, three-dimensional point cloud data and spectral data of the bridge to be detected into the input module of the bridge damage detection model to determine the image feature vector, the global feature vector of the point cloud and the spectral feature vector; the feature fusion module fuses the feature vector of the image, the global feature vector of the point cloud and the spectral feature vector, and obtains the fused feature vector by interacting and fusing the feature representations of each modality obtained after fusion at different levels, and outputs the bridge damage detection type through the classifier in the DNN classification layer; by introducing a recursive neural network, the feature vector of the image, the global feature vector of the point cloud and the spectral feature vector are processed to extract the time series features from each modal data. The system uses a time series prediction module and a dynamic update mechanism to input current and historical data based on an extended state space model or a nonlinear dynamic system model, predict future damage trends, and output real-time updated damage identification results. When new data is input, the current damage identification results are corrected through the dynamic update mechanism. A damage accumulation model is introduced, which accumulates damage information in each time period. The damage information in each time period is processed through a weighted accumulation and attenuation mechanism to evaluate the long-term health status of the bridge, conduct trend analysis, and identify potential structural degradation trends. By observing the acceleration of crack propagation speed, the failure time of the bridge structure is predicted, and preventive maintenance is recommended.

[0113] Through advanced data processing and analysis technologies, the accuracy and efficiency of bridge health monitoring are improved, providing comprehensive technical support for bridge maintenance and management.

[0114] In the data cleaning and processing module, data cleaning methods based on adaptive filtering algorithms, such as Kalman filtering and adaptive wavelet transform, are used to improve the accuracy and reliability of diverse sensor data in complex environments. Through these algorithms, the drone system effectively eliminates noise and interference, ensuring higher accuracy and consistency in collected multi-source data, laying a solid foundation for subsequent health assessments.

[0115] By comprehensively analyzing data from different sensors, a comprehensive assessment of the bridge's health status is achieved. Specifically, after obtaining fused data from a multi-source heterogeneous sensor data fusion algorithm, deep learning models such as long short-term memory (LSTM) and gated recurrent units (GRU) are used.

[0116] LSTM networks excel at processing sequential data and can capture temporal trends in bridge health. The drone system feeds the multimodal fusion feature vector into the LSTM network, which recursively processes the time series data to learn how bridge health changes over time. Through LSTM analysis of the time series, the drone system can identify material degradation trends, such as coating aging rates and crack growth rates.

[0117] GRU, a lightweight alternative to LSTM, has fewer parameters and faster computation. Unmanned aerial vehicle systems can use GRU to process time series data based on mission requirements, making it particularly suitable for real-time applications with limited resources. Furthermore, through GRU's time series analysis of point cloud and infrared data, UAV systems can identify and predict dynamic changes in bridge structural deformation and changes in stress distribution due to load variations.

[0118] Based on the analysis results of the LSTM and GRU algorithms, the drone system comprehensively assesses the overall health of the bridge, generating a health score that reflects the current bridge health level. Through cumulative analysis using time series models, the drone system can predict future deterioration trends and provide preventive maintenance recommendations.

[0119] Based on this analysis, the generated assessment report includes detailed information such as the overall health of the bridge, the extent of material degradation, the extent of structural deformation, and the impact of load changes. It also includes visualizations of various types of damage, such as crack growth, stress distribution, and temperature changes, helping managers gain an intuitive understanding of the bridge's health. Based on the analysis results, the drone system provides targeted maintenance recommendations, including areas requiring attention, possible repair measures, and a recommended time for the next inspection.

[0120] This application also designs a real-time monitoring system that combines time series analysis and anomaly warning and prediction algorithms, using time series analysis models (such as AutoRegressive Integrated Moving Average, ARIMA) and anomaly detection algorithms (such as Isolation Forest) to accurately identify potential structural damage and abnormal conditions, as follows:

[0121] ARIMA is a classic time series analysis model widely used for forecasting and analyzing time series data. The ARIMA model models time series data using three components: autoregression (AR), differencing (I), and moving average (MA). ARIMA is used to analyze time series data of bridge structural health indicators (such as stress, displacement, and vibration), predict future conditions, and identify long-term trends and cyclical changes. To address the potential nonlinear characteristics in bridge health monitoring, ARIMA is combined with nonlinear models (such as LSTM and GRU) to form a hybrid model. This hybrid model first uses ARIMA to capture the linear component and then uses a nonlinear model to process the residual component, thereby improving prediction accuracy. For the aforementioned multimodal data, the multivariate ARIMA model (VARIMA) is introduced. It can simultaneously process multiple time series data and capture the relationships between the variables. By jointly modeling multimodal bridge data (such as stress, temperature, and humidity), bridge health can be more accurately predicted.

[0122] Isolation Forest is a tree-based, unsupervised learning algorithm specifically designed for anomaly detection. It randomly selects data features and split points to construct a set of decision trees. It then determines whether a data point is an outlier based on its isolation depth (i.e., the length of its path in the tree). A shorter path means that the point is more easily isolated and therefore likely an anomaly. Isolation Forest is used to detect anomalies in multimodal data, such as sudden stress increases, temperature anomalies, and changes in vibration patterns. These anomalies may indicate structural damage or potential failure. To incorporate time series characteristics, the model's anomaly detection capabilities are enhanced by incorporating temporal dependencies into the model as additional features or by preprocessing the time series before isolating data points. Isolation Forest is also combined with the LSTM-Autoencoder. The LSTM-Autoencoder first learns the time series patterns of normal data and feeds the reconstruction error as input to the Isolation Forest, thereby enhancing the detection of anomalies in complex time series.

[0123] By analyzing the temporal characteristics of multimodal data, the drone system predicts the future health status of the bridge and issues early warning signals, helping managers take preventive maintenance measures before major damage occurs to the bridge, ensuring the long-term safety of the bridge.

[0124] To enable real-time data processing and efficient transmission for the bridge monitoring system, a bridge monitoring system based on the Internet of Things (IoT) architecture was developed. Leveraging distributed computing, the drone system efficiently processes and stores massive amounts of data, ensuring real-time responsiveness. Furthermore, the integration of edge computing and cloud computing optimizes the timeliness and reliability of data processing. The drone system was tested in actual bridge environments, verifying its reliability and accuracy, ensuring real-time monitoring and early warning of bridge health.

[0125] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

[0126] In addition, unless otherwise specified, all technical and scientific terms used in the present invention have the same meaning as commonly understood by those skilled in the art to which the present invention belongs. All documents mentioned in this specification are incorporated by reference to disclose and describe the methods related to the documents. In the event of any conflict with any incorporated document, the content of this specification shall prevail.

Claims

1. A bridge damage detection method, characterized in that: The following steps are involved: The drone system autonomously navigates to collect high-resolution images of the bridge to be inspected, identifying surface defects such as cracks, rust, and surface damage. The LiDAR device generates three-dimensional point cloud data to provide information on the bridge's geometry and structure, identifying deformation, displacement, and geometric defects. Infrared cameras capture the bridge's temperature distribution data to detect internal temperature anomalies caused by corrosion, cracks, or material defects. Spectral data provided by multispectral or hyperspectral sensors is used to analyze changes in the chemical composition of the bridge's materials, including coating aging and corrosion reactions. Constructing a network model; the network model includes an input module and a feature fusion module; Multiple cross-attention modules and multimodal Transformer networks are introduced into the feature fusion module. Each cross-attention module fuses the feature vectors of each modality corresponding to high-resolution images, 3D point cloud data, and spectral data through the cross-attention mechanism and Transformer self-attention mechanism, obtaining the fused feature representation of each modality. This representation is then input into the DNN classification layer and the classification result is output. The constructed network model is trained to obtain a bridge damage detection model. The high-resolution image, 3D point cloud data, and spectral data of the bridge to be inspected are input into the input module of the bridge damage detection model to determine the image feature vector, the global feature vector of the point cloud, and the spectral feature vector. The feature fusion module fuses the image feature vector, the global feature vector of the point cloud, and the spectral feature vector. The feature representations of each modality obtained after fusion are interacted and fused at different levels to obtain the fused feature vector. The bridge damage detection type is output through the classifier in the DNN classification layer. By introducing a recursive neural network, the image feature vector, the global feature vector of the point cloud, and the spectral feature vector are processed to extract the time series features from each modal data and obtain the temporal features of each modality; The system introduces a time series prediction module and a dynamic update mechanism. Based on an extended state space model or a nonlinear dynamic system model, it inputs current and historical data, predicts future damage trends, and outputs real-time updated damage identification results. When new data is input, the dynamic update mechanism corrects the current damage identification results. A damage accumulation model is introduced, which accumulates damage information in each time period and processes the damage information in each time period through a weighted accumulation and attenuation mechanism to evaluate the long-term health status of the bridge, conduct trend analysis, and identify potential structural degradation trends; By observing the acceleration of crack growth rate, the failure time of the bridge structure is predicted and preventive maintenance is recommended.

2. The bridge damage detection method according to claim 1, characterized in that: The determining of the image feature vector, the global feature vector of the point cloud, and the spectral feature vector specifically includes the following steps: The high-resolution image of the bridge to be inspected is preprocessed using a convolutional neural network (CNN) to extract high-level features from the image and convert them into fixed-size feature vectors. The extracted fixed-size feature vectors are then fed into the multimodal Transformer network of the feature fusion module as input for the image modality. For 3D point cloud data, use PointNet, PointNet++, or other point cloud processing networks to learn the features of each point and generate a global feature vector for the point cloud through global feature aggregation. The generated global feature vector of the point cloud is input into the multimodal Transformer network of the feature fusion module. The acquired spectral data of the bridge to be inspected is processed by a 1D convolutional neural network (1D-CNN) or a recurrent neural network (RNN / LSTM) to extract local features from the spectral data and obtain spectral feature vectors. The obtained spectral feature vectors are then input into the multimodal Transformer network of the feature fusion module.

3. The bridge damage detection method according to claim 2, characterized in that: Each cross-attention module fuses the feature vectors of each modality corresponding to the high-resolution image, 3D point cloud data, and spectral data through the cross-attention mechanism and the Transformer self-attention mechanism, specifically including the following steps: After preliminary feature extraction of high-resolution images, 3D point cloud data, and spectral data, the extracted features are adaptively weighted using SE-Block or CBAM. The multimodal Transformer network includes the Transformer self-attention mechanism, encoder layers, and positional encoding; The Transformer self-attention mechanism, as a core component of the multimodal Transformer network, captures the correlation within and across modalities of feature vectors by calculating the relationship between queries, keys, and values. In the multimodal Transformer network, the feature vectors of each modality are mapped to query, key, and value vectors, and the similarity between features is calculated. The calculated similarities between features are weighted and summed to generate an enhanced feature representation. The encoder layer consists of multiple stacked Transformer layers, each of which includes a Transformer self-attention mechanism and a feedforward neural network. The feature vector of each modality is passed and processed layer by layer in the encoder to capture the global context of the data from different modalities. The encoder output is a fused multimodal feature representation that retains the key information in the input data and captures the complementary characteristics between modalities through the Transformer attention mechanism to obtain the feature vector of each modality. By introducing position encoding into the Transformer encoder, in the multimodal Transformer network, position encoding provides position information for the feature vector of each modality, obtaining the feature representation of each modality that is consistent in time and space.

4. The bridge damage detection method according to claim 3, characterized in that: The feature vector of the image, the global feature vector of the point cloud and the spectral feature vector are determined. The following steps are involved: After integrating the channel attention mechanism SE-Block into the high-resolution image, 3D point cloud data, and spectral data feature extraction modules, the feature vectors of each modality are channel-weighted before being input into the Transformer encoder. SE-Block compresses the spatial dimension of each feature map into a global feature representation through the Squeeze operation, learns the correlation between channels through the Excitation operation, and adaptively reweights the features of each channel.

5. The bridge damage detection method according to claim 4, characterized in that: The step of obtaining the feature representation after fusion of the feature vectors of each modality specifically includes the following steps: After being processed by multiple stacked Transformer layers of the Transformer self-attention mechanism, the acquired temporally and spatially consistent feature representations of each modality are interacted and fused at different levels. Feature fusion is performed using feature concatenation, weighted summation, or Transformer layers to generate a fused feature vector that integrates key information from different modalities to form a multimodal joint representation.

6. The bridge damage detection method according to claim 5, characterized in that: The output classification result specifically includes: The generated multimodal joint representation is processed through multiple fully connected layers of the DNN classification layer, and classified, regressed or detected through the softmax classifier to output the type or location of the bridge damage to be detected.

7. A bridge damage detection system, characterized in that: include: A data acquisition module is used to collect high-resolution images of the bridge to be inspected through autonomous navigation of the UAV system, and identify defects on the bridge surface, including cracks, rust, and surface damage; LiDAR equipment generates 3D point cloud data to provide information on the bridge's geometry and structure, identifying deformation, displacement, and geometric defects. Infrared cameras capture temperature distribution data on the bridge to detect internal temperature anomalies caused by corrosion, cracks, or material defects. Spectral data provided by multispectral or hyperspectral sensors analyzes chemical composition changes in the bridge's materials, including coating aging and corrosion reactions. A bridge damage detection model building module is used to build a network model; the network model includes an input module and a feature fusion module; Multiple cross-attention modules and multimodal Transformer networks are introduced into the feature fusion module. Each cross-attention module fuses the feature vectors of each modality corresponding to high-resolution images, 3D point cloud data, and spectral data through the cross-attention mechanism and Transformer self-attention mechanism, obtaining the fused feature representation of each modality. This representation is then input into the DNN classification layer and the classification result is output. The constructed network model is trained to obtain a bridge damage detection model. The bridge damage detection module is used to input the high-resolution image, three-dimensional point cloud data and spectral data of the bridge to be detected into the input module of the bridge damage detection model to determine the feature vector of the image, the global feature vector of the point cloud and the spectral feature vector; the feature fusion module fuses the feature vector of the image, the global feature vector of the point cloud and the spectral feature vector, and obtains the fused feature vector by interacting and fusing the feature representations of each fused mode at different levels, and outputs the bridge damage detection type through the classifier in the DNN classification layer; by introducing a recursive neural network, the feature vector of the image, the global feature vector of the point cloud and the spectral feature vector are fused. Processing, extracting time series features from each modal data to obtain the time series features of each mode; introducing a time series prediction module and a dynamic update mechanism, based on an extended state space model or a nonlinear dynamic system model, inputting current and historical data, predicting future damage trends, and outputting real-time updated damage identification results; when new data is input, correcting the current damage identification results through the dynamic update mechanism; introducing a damage accumulation model, which accumulates damage information in each time period, processes the damage information in each time period through a weighted accumulation and attenuation mechanism, evaluates the long-term health status of the bridge, conducts trend analysis, and identifies potential structural degradation trends; By observing the acceleration of crack growth rate, the failure time of the bridge structure is predicted and preventive maintenance is recommended.

Images