Bridge damage detection method and system

Abstract

The invention discloses a bridge damage detection method and system, and belongs to the technical field of bridge engineering.The method comprises the steps that image data, point cloud data and spectral data of a to-be-detected bridge are obtained; inputting the obtained image data, point cloud data and spectrum data of the to-be-detected bridge into an input module of the constructed bridge damage detection model, and determining feature vectors of the image, the point cloud and the spectrum; and a feature fusion module of the constructed bridge damage detection model fuses feature vectors of images, point clouds and spectrums, interacts and fuses obtained fused feature representations of all modals on different levels to obtain fused feature vectors, and outputs a bridge damage detection type through a classifier in a DNN classification layer. The method can improve the accuracy of bridge damage detection.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge engineering, and more specifically to a bridge damage detection method and system. Background Art

[0002] With the continuous development of transportation infrastructure, bridges, as important transportation hubs, have received more and more attention for their safety and reliability. Bridges have complex structures and variable stress conditions. After long-term use, they are prone to various damages such as cracks, corrosion, and deformation. Therefore, regular inspection and maintenance of bridges are particularly important.

[0003] Traditional bridge inspection methods mostly rely on manual inspection, which is not only time-consuming and labor-intensive, but also the accuracy and reliability of the inspection results are easily affected by human factors. In addition, manual inspections often make it difficult to reach certain key parts of the bridge, making it difficult to detect potential safety hazards in a timely manner.

[0004] With the advancement of science and technology, the rapid development of drone technology has provided a new solution for bridge inspection. Drones have the advantages of strong flexibility, easy operation, and low cost. They can quickly cover all parts of the bridge and take high-resolution images and videos, greatly improving the inspection efficiency and inspection range.

[0005] However, the existing technology only relies on drones to obtain data from a single sensor. By extracting the key features of the data from a single sensor, it is unable to fully obtain various status information of the bridge. It can only detect surface damage and cannot accurately identify internal damage of the bridge, which affects the accuracy of bridge damage detection and makes it impossible to achieve comprehensive inspection of the bridge structure. Summary of the invention

[0006] In response to the problems existing in the above-mentioned fields, the present invention proposes a bridge damage detection method and system. Through multi-angle and multi-dimensional data collection of the bridge, various status information of the bridge can be obtained more comprehensively. The constructed bridge damage detection model can accurately identify the internal damage of the bridge, improve the accuracy of bridge damage detection, and then comprehensively evaluate the health status of the bridge.

[0007] In order to solve the above technical problems, the present invention discloses a bridge damage detection method, comprising the following steps:

[0008] Obtain image data, point cloud data and spectral data of the bridge to be inspected;

[0009] Construct a network model; the network model includes an input module and a feature fusion module; introduce multiple cross-attention modules and a multimodal Transformer network into the feature fusion module, each cross-attention module fuses the feature vectors of each modality corresponding to the image data, point cloud data and spectral data through the cross-attention mechanism and the Transformer self-attention mechanism, obtains the feature representation after the fusion of the feature vectors of each modality, and inputs it into the DNN classification layer, and outputs the classification result; trains the constructed network model to obtain a bridge damage detection model;

[0010] The acquired image data, point cloud data and spectral data of the bridge to be inspected are input into the input module of the bridge damage detection model to determine the feature vectors of the image, point cloud and spectrum. The feature fusion module fuses the feature vectors of the image, point cloud and spectrum, and obtains the fused feature vector by interacting and fusing the feature representations of each modality at different levels. The bridge damage detection type is output through the classifier in the DNN classification layer.

[0011] Preferably, the acquiring of image data, point cloud data and spectral data of the bridge to be inspected specifically includes:

[0012] The drone can autonomously navigate to collect high-resolution images of the bridge to be inspected, identifying defects on the bridge surface, including cracks, rust, and surface damage;

[0013] The three-dimensional point cloud data generated by LiDAR equipment provides the geometric shape and structural information of the bridge and identifies the deformation, displacement and geometric defects of the bridge. The temperature distribution data of the bridge is captured by infrared cameras to detect internal temperature anomalies caused by corrosion, cracks or material defects.

[0014] The spectral data provided by multispectral or hyperspectral sensors are used to analyze the changes in the chemical composition of the material of the bridge to be inspected, including coating aging and corrosion reactions.

[0015] Preferably, the determining of the feature vectors of the image, point cloud and spectrum specifically comprises the following steps:

[0016] The acquired image data of the bridge to be inspected is preprocessed by a convolutional neural network (CNN) to extract high-level features of the image and convert them into feature vectors of fixed size; the extracted feature vectors of fixed size are input into the multimodal Transformer network of the feature fusion module as input of the image modality;

[0017] For point cloud data, use PointNet, PointNet++ or other point cloud processing networks to learn the features of each point and generate a global feature vector of the point cloud through global feature aggregation; input the generated point cloud feature vector into the multimodal Transformer network of the feature fusion module;

[0018] The acquired spectral data of the bridge to be inspected is processed by 1D convolutional neural network 1D-CNN or recurrent neural network RNN / LSTM to extract local features in the spectral data and obtain spectral feature vectors, which are then input into the multimodal Transformer network of the feature fusion module.

[0019] Preferably, each cross attention module fuses the feature vectors of each modality corresponding to the image data, the point cloud data and the spectral data respectively through the cross attention mechanism and the Transformer self-attention mechanism, specifically comprising the following steps:

[0020] After the image data, point cloud data and spectral data have undergone preliminary feature extraction, the extracted features are first adaptively weighted through SE-Block or CBAM;

[0021] The multimodal Transformer network includes the Transformer self-attention mechanism, encoder layer, and position encoding;

[0022] Among them, the Transformer self-attention mechanism, as the core component of the multimodal Transformer network, captures the correlation within the feature vector and across modalities by calculating the relationship between Query, Key and Value; in the multimodal Transformer network, the feature vector of each modality is mapped to Query, Key and Value vectors, and the similarity between features is calculated; the similarity between the calculated features is weighted and summed to generate an enhanced feature representation;

[0023] The encoder layer consists of multiple stacked Transformer layers, each of which includes a Transformer self-attention mechanism and a feedforward neural network. The feature vector of each modality is transmitted and processed layer by layer in the encoder to capture the global context information of data of different modalities. The output of the encoder is the fused multi-modal feature representation, which retains the key information in the input data and captures the complementary characteristics between modalities through the Transformer attention mechanism to obtain the feature vector of each modality.

[0024] By introducing position encoding in the Transformer encoder, in the multimodal Transformer network, position encoding provides position information for the feature vector of each modality, obtaining feature representations of each modality that are consistent in time and space.

[0025] Preferably, the step of determining the feature vector of the image, the global feature vector of the point cloud and the spectral feature vector further comprises the following steps:

[0026] After the channel attention mechanism SE-Block is integrated into the feature extraction modules of image data, point cloud data and spectral data, the feature vector of each modality is channel-weighted before it is input into the Transformer encoder.

[0027] SE-Block compresses the spatial dimension of each feature map into a global feature representation through the Squeeze operation, learns the correlation between channels through the Excitation operation, and adaptively reweights the features of each channel.

[0028] Preferably, the step of obtaining the feature representation after fusion of the feature vectors of each modality specifically includes the following steps:

[0029] After being processed by multiple stacked Transformer layers of the Transformer self-attention mechanism, the acquired temporally and spatially consistent feature representations of each modality are interacted and fused at different levels; feature concatenation, weighted summation, or Transformer layers are used for feature fusion to generate a fused feature vector that integrates key information from different modalities to form a multimodal joint representation.

[0030] Preferably, the output classification result specifically includes:

[0031] The generated multimodal joint representation is processed through multiple fully connected layers of the DNN classification layer, and classified, regressed or detected through the softmax classifier to output the type or location of the bridge damage to be detected.

[0032] Preferably, the method further includes extracting time series features from each modal data, specifically including the following steps:

[0033] By introducing a recursive neural network, the feature vector of the image, the global feature vector of the point cloud, and the spectral feature vector are processed, and the time series features are extracted from each modal data to obtain the time series features of each modality;

[0034] Introduce a time series prediction module and a dynamic update mechanism. Based on an extended state space model or a nonlinear dynamic system model, input current and historical data, predict future damage trends, and output real-time updated damage identification results. When new data is input, the current damage identification results are corrected through a dynamic update mechanism.

[0035] A damage accumulation model is introduced, which accumulates damage information in each time period. The damage information in each time period is processed through weighted accumulation and attenuation mechanism, the long-term health status of the bridge is evaluated for trend analysis, and potential structural degradation trends are identified; by observing the acceleration of crack propagation speed, the failure time of the bridge structure is predicted, and preventive maintenance is recommended.

[0036] Preferably, a bridge damage detection system is also included, comprising:

[0037] A data acquisition module, used to acquire image data, point cloud data and spectral data of the bridge to be inspected;

[0038] A bridge damage detection model construction module is used to construct a network model; the network model includes an input module and a feature fusion module; multiple cross-attention modules and a multimodal Transformer network are introduced into the feature fusion module, each cross-attention module fuses the feature vectors of each modality corresponding to the image data, point cloud data and spectral data through the cross-attention mechanism and the Transformer self-attention mechanism, obtains the feature representation after the fusion of the feature vectors of each modality, and inputs it into the DNN classification layer, and outputs the classification result; the constructed network model is trained to obtain a bridge damage detection model;

[0039] The bridge damage detection module is used to input the acquired image data, point cloud data and spectral data of the bridge to be detected into the input module of the bridge damage detection model to determine the feature vectors of the image, point cloud and spectrum; the feature fusion module fuses the feature vectors of the image, point cloud and spectrum, and obtains the fused feature vector by interacting and fusing the feature representations of each modality at different levels, and outputs the bridge damage detection type through the classifier in the DNN classification layer.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The bridge damage detection method proposed in the present invention can obtain various status information of the bridge more comprehensively by integrating multiple sensors to collect data from multiple angles and dimensions of the bridge. The constructed bridge damage detection model effectively fuses the collected multi-source heterogeneous sensor data, and a comprehensive evaluation of the health status of the bridge can be formed by fusing these multi-source data. By extracting features from the image data, point cloud data and spectral data of the bridge, the multi-dimensional features of the bridge are comprehensively analyzed to achieve comprehensive detection and accurate identification of the bridge structure. By introducing multiple cross-attention modules and multimodal Transformer networks in the feature fusion module, feature extraction and fusion of multimodal data are achieved, data heterogeneity problems are solved, key features are extracted from image, point cloud and spectral data, and the accuracy and reliability of the fusion results are ensured, thereby providing more accurate and comprehensive data support for autonomous flight and intelligent obstacle avoidance of unmanned aerial vehicles. By detecting the acquired bridge data to be detected through the bridge damage detection model, the attention to different damage features can be dynamically adjusted, and accurate diagnosis and prediction of the bridge status can be achieved by generating a detailed detection report containing the defect location, type or severity. The method proposed in the present invention improves the accuracy of bridge damage detection, can accurately detect the health status of the bridge, and provides useful references and suggestions for researchers and engineering practitioners in the field of infrastructure engineering. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of the bridge damage detection method proposed by the present invention;

[0043] Figure 2 A network architecture diagram of a bridge damage detection model constructed for the present invention;

[0044] Figure 3 A schematic diagram of an environment-aware UAV platform provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the attached embodiment of the present invention Figure 1-Figure 3 , the technical solutions in the embodiments of the present invention are clearly and completely described. It should be understood that the terms described in the present invention are only used to describe specific implementation methods and are not used to limit the present invention.

[0046] Example

[0047] like Figure 1 As shown, the present invention proposes a bridge damage detection method, which includes the following steps:

[0048] S1: Acquire image data, point cloud data and spectral data of the bridge to be inspected;

[0049] S2: Construct bridge damage detection model;

[0050] Construct a network model; the network model includes an input module and a feature fusion module; introduce multiple cross-attention modules and a multimodal Transformer network into the feature fusion module, each cross-attention module fuses the feature vectors of each modality corresponding to the image data, point cloud data and spectral data through the cross-attention mechanism and the Transformer self-attention mechanism, obtains the feature representation after the fusion of the feature vectors of each modality, and inputs it into the DNN classification layer, and outputs the classification result; trains the constructed network model to obtain a bridge damage detection model;

[0051] S3: The acquired image data, point cloud data and spectral data of the bridge to be inspected are input into the input module of the bridge damage detection model to determine the feature vectors of the image, point cloud and spectrum; the feature fusion module fuses the feature vectors of the image, point cloud and spectrum, and obtains the fused feature representations of each modality through interaction and fusion at different levels to obtain the fused feature vector, and outputs the bridge damage detection type through the classifier in the DNN classification layer.

[0052] In step S1, image data, point cloud data and spectral data of the bridge to be inspected are obtained, specifically including:

[0053] Through drone platforms with autonomous flight and intelligent obstacle avoidance capabilities, such as Figure 3 As shown, it is equipped with multiple sensors such as high-definition visual images, thermal imagers, lidar, multi-spectral sensors, etc. to achieve efficient and safe bridge inspections, especially at high altitudes and in complex environments.

[0054] The high-definition camera carried by the drone platform obtains detailed images of the bridge, the thermal imager obtains the internal damage of the bridge, the lidar provides accurate three-dimensional structural data, and the multispectral sensor detects the characteristics and potential damage of the material.

[0055] Specifically, the drone uses autonomous navigation to collect high-resolution images of the bridge to be inspected and identify defects on the bridge surface, including cracks, rust, and surface damage.

[0056] Using radar equipment, the generated three-dimensional point cloud data provides the geometric shape and structural information of the bridge, and identifies the deformation, displacement and geometric defects of the bridge; the temperature distribution data of the bridge is captured by infrared cameras to detect internal temperature anomalies caused by corrosion, cracks or material defects.

[0057] The spectral data provided by multispectral or hyperspectral sensors are used to analyze the changes in the chemical composition of the material of the bridge to be inspected, including coating aging and corrosion reactions.

[0058] Through the automated flight and data collection of the UAV platform, all parts of the bridge can be covered to ensure comprehensive inspection. The diverse configuration of the sensor integration module enables the system to work efficiently under different environmental conditions.

[0059] The present invention provides an intelligent bridge damage detection method based on multi-source data fusion and unmanned aerial vehicle, such as Figure 1 As shown, the following steps are included:

[0060] The drone platform is equipped with a variety of sensors such as high-definition cameras, thermal imagers, lidar, etc. for data acquisition, color recognition and distance measurement, thereby achieving autonomous navigation and data collection.

[0061] High-definition cameras are used to obtain detailed images of the bridge, thermal imagers detect internal damage to the bridge, millimeter-wave radars identify long-distance obstacles, ultrasonic radars identify close-range obstacles, lidars provide accurate three-dimensional structural data, and multispectral sensors are used to detect material properties and potential damage. By fusing multi-sensor data, the controller is able to execute decisions and achieve panoramic perception and detailed modeling of the bridge.

[0062] In order to enhance environmental perception and obstacle avoidance capabilities, the Intel RealSense D435 camera is used to capture color and depth images for accurate environmental perception. The color images are input into the YOLOV8 model for obstacle classification and bounding boxes are generated to quickly identify and locate key obstacles.

[0063] Due to the advantages of YOLOV8 model in speed, accuracy and computational efficiency, it is particularly suitable for resource-constrained environments such as embedded drone systems such as drones. YOLO V8 optimizes the model architecture and enhances the multi-scale feature extraction capability to better detect targets of different sizes and distances. Specifically, it includes the following steps:

[0064] First, the enhanced feature pyramid network (FPN) is introduced in the YOLOv8 model to enhance the model's ability to detect objects of different scales. YOLOv8's improvement on FPN is not just a simple feature fusion from low to high layers, but rather an efficient interaction between features of different scales through more skip connections. This design improves the flexibility of the feature pyramid and can better capture the details of small objects. And by extracting features at different scales, YOLOv8 can better handle objects with large scale changes, making the model perform better when detecting distant or tiny objects. The improved FPN can better handle features at different levels, and retain more contextual information by transferring and fusing them layer by layer.

[0065] Secondly, YOLOv8 optimizes the path aggregation network (PAN), which is a module used to enhance the propagation of contextual information. In YOLOv8, PAN not only considers bottom-up path aggregation, but also introduces top-down paths, making the information flow smoother within the model, thereby improving detection performance. And by introducing a more complex path structure, PAN is able to propagate contextual information more effectively, especially when dealing with complex backgrounds. This improvement significantly improves the robustness and detection accuracy of the model.

[0066] In the depth image, the grayscale value of each pixel directly represents the depth of the pixel position (i.e., the distance from the object surface to the camera). When performing obstacle target detection, a bounding box is usually generated for the obstacle, which contains all the pixels of the object. The most reliable depth data usually comes from the center of the bounding box (Pc), and the grayscale value of this point can usually represent the average distance from the obstacle to the camera. However, due to changes in the shape and posture of the object or environmental conditions (such as lighting, occlusion, etc.), the bounding box may be deformed, resulting in distortion of the depth data at the center point. To compensate for possible errors, in addition to the center point of the bounding box (Pc), the center of gravity position (Pg) of the object is also concerned. The center of gravity position (Pg) is obtained by calculating the weighted average of all pixels in the obstacle area, which usually better represents the actual position and depth of the obstacle. At the same time, averaging the depth values ​​in and around the bounding box, or applying local weighted averaging, can reduce the distortion caused by a single pixel (possible noise or error). Therefore, it is necessary to combine the center of gravity calculation with the regional depth mean calculation to optimize the depth data, correct the depth of the obstacle, obtain more reliable depth data, ensure that the UAV can accurately perceive the position of obstacles in complex environments, and formulate effective obstacle avoidance strategies.

[0067] After the target detection results are fused with the optimized depth image, the UAV system can extract the outline and three-dimensional spatial information of the obstacle, greatly improving the detection accuracy and providing detailed data for subsequent obstacle avoidance decisions. By fusing these data, the UAV system generates the optimal obstacle avoidance strategy, taking into account factors such as the category, profile characteristics, and location distribution of the obstacle, and calculates the obstacle avoidance trajectory with the minimum distance, ensuring that the UAV avoids obstacles in the safest path and successfully performs the mission.

[0068] After combining the target detection results with the optimized depth image, the UAV system can effectively extract the outline and three-dimensional spatial information of the obstacle, thereby greatly improving the detection accuracy and providing detailed data support for subsequent obstacle avoidance decisions.

[0069] The YOLOv8 model is used to detect the category and position of obstacles on the RGB image and generate corresponding bounding boxes. These bounding boxes contain the coordinate information of the obstacles in the two-dimensional image; these bounding boxes are mapped to the corresponding depth image, in which the grayscale value of each pixel represents the depth information of the position, that is, the distance from the surface of the object to the camera. By extracting the depth information of the pixels in the bounding box area, the UAV system can fuse the target detection results with the three-dimensional spatial information in the depth image. This fusion process can extract the average depth value of the area by weighted averaging or other statistical methods of the pixels in the bounding box to better reflect the actual spatial position and size of the obstacle.

[0070] The UAV system extracts the outline of the obstacle by applying edge detection algorithms (such as Canny edge detection or Sobel operator) to the optimized depth image. These algorithms can identify areas in the image where the depth value changes significantly, that is, the boundaries of the object. The extracted outline can be combined with the bounding box of the target detection to further accurately locate the edge of the obstacle, ensuring that the UAV system can accurately identify the shape and boundaries of the obstacle. The UAV system combines the outline information and depth data to convert the two-dimensional image information of the obstacle into three-dimensional space coordinates through techniques such as triangulation, voxel modeling or point cloud generation. Specifically, a three-dimensional model of the obstacle can be constructed by sampling multiple points in the bounding box area and based on the depth values ​​of these points.

[0071] The UAV system integrates the category information, depth information, contour information, and position and shape information of obstacles. Category information refers to the obstacle category (such as trees, buildings, pedestrians, etc.) extracted from the target detection results; depth information refers to the obstacle distance data obtained from the depth image, including the depth value of each pixel and the regional depth information after weighted averaging or other processing. Contour information is the obstacle boundary contour extracted by edge detection, and combined with depth information for 3D modeling and obstacle avoidance path planning. Position and shape information refers to the exact position and shape of obstacles in space extracted by 3D modeling and point cloud generation. The fusion of these data enables the UAV system to fully understand the spatial distribution and dynamic characteristics of obstacles, providing detailed data support for subsequent obstacle avoidance decisions.

[0072] The UAV system uses a deep learning-based path planning algorithm to generate the optimal obstacle avoidance strategy. These algorithms usually include but are not limited to convolutional neural networks (CNN), reinforcement learning (RL), and generative adversarial networks (GAN). The specific algorithm selection depends on the complexity of the task and the environmental conditions.

[0073] Reinforcement learning methods are particularly suitable for obstacle avoidance tasks in dynamic environments. By training in a simulated environment, reinforcement learning algorithms can learn the best obstacle avoidance strategy, that is, safely avoiding obstacles while minimizing the path length. Path planning algorithms such as the A algorithm, Dijkstra algorithm, RRT (rapidly expanding random trees), etc. can also be used for obstacle avoidance path planning in static environments. The drone system will consider the category of obstacles, profile features (such as height, width, etc.) and location distribution in the environment, and calculate an obstacle avoidance trajectory with the minimum distance. These algorithms combine perception data and environmental models, and select the optimal path through a comprehensive evaluation of possible paths to ensure that the drone can avoid obstacles in the safest way and perform its mission smoothly.

[0074] Therefore, the UAV system can accurately sense and avoid obstacles in complex and dynamic environments, ensuring the safe and efficient execution of tasks. This combination of multi-data fusion and deep learning algorithms enables the UAV to have intelligent obstacle avoidance capabilities, providing strong support for autonomous flight and mission execution. The UAV performs straight-line flight missions between multiple mission points through the mission control UAV system and customized scripts, while autonomously avoiding detected obstacles. These customized scripts use software packages such as ROS, MAVROS, and OpenCV to optimize images, run deep learning algorithms, generate obstacle avoidance strategies, and plan flight paths.

[0075] During flight, different types of obstacles pose different threats to flight safety, so corresponding avoidance strategies need to be formulated. Through sensors on drones (such as depth cameras or LiDAR), drone systems can obtain depth data in the field of view ahead. The depth value of each pixel represents its distance to the drone. Based on these depth values, the drone system divides the space in the field of view into four areas: clear sky, warning, action, and emergency. The clear sky area indicates that there are no obstacles ahead and it is safe to fly; the warning area senses potential risks but is uncertain about the specific location, so the drone will reduce speed and continue to monitor; the action area requires the drone system to take specific obstacle avoidance actions based on the properties of the detected obstacles; and the emergency area immediately takes emergency stop measures to ensure flight safety. Based on the drone's flight speed, acceleration, and the properties of the obstacles (such as dynamic or static), the drone system can adjust the distance thresholds of these areas in real time to ensure optimal obstacle avoidance performance in different environments.

[0076] Through this partition management and avoidance strategy, the drone can adapt to different environmental conditions, reduce the risk of collision, and ensure the smooth completion of the mission. In order to monitor the flight status in real time, the QGroundControl ground control station software is used to observe and record flight parameters, adjust mission parameters, and ensure the safety and accurate execution of flight missions.

[0077] The present invention supports intelligent decision-making and analysis in complex environments by extracting and fusing key information from multiple sensor data.

[0078] Based on the deep convolutional autoencoder (CAE), its core consists of two parts: encoder and decoder. The encoder maps multimodal input data to a low-dimensional latent space and extracts key features; the decoder reconstructs the original data from the latent space, retaining the main information of the data and eliminating noise and errors.

[0079] In order to achieve data calibration from different sensors, multiple parallel convolutional layers are introduced in the encoder part. These convolutional layers can capture and align the low-level features of data from different modalities, and map the data from different modalities into a unified feature space through shared weights or cross-modal convolution operations. After multiple layers of convolution and pooling operations, the encoder compresses and maps the data from each modality to a low-dimensional latent space. This process not only achieves data calibration, but also retains the key structural information of the data from each modality. During the encoding process, the filters in the convolutional layer automatically learn how to identify and suppress noise. By introducing noisy samples during training, the deep convolutional autoencoder can effectively recover noise-free signals from noisy data. In the latent space representation, the noise is compressed into small perturbations. Through the deconvolution operation in the decoder, the UAV system can eliminate these noises when reconstructing the data, ensuring the clarity and accuracy of the output. In order to eliminate the numerical differences in data from different sensors, batch normalization (Batch Normalization) or layer normalization (LayerNormalization) operations are introduced in the encoder. These operations implement data standardization in the latent space to ensure that multimodal data is analyzed at the same scale. During the decoder reconstruction process, the UAV system is able to restore the original numerical characteristics of the data through the inverse normalization operation, so that the processed data maintains consistency while retaining its unique quantitative characteristics.

[0080] The decoder part consists of a series of deconvolution layers, which are responsible for remapping the low-dimensional representation in the latent space back to the original data space. In this process, the decoder not only reconstructs the main structure of the data, but also further optimizes the quality of the data. The autoencoder is trained by minimizing the reconstruction error, usually using the mean square error (MSE) as the loss function. During the training process, the drone system continuously adjusts the weights of the encoder and decoder to ensure that the final output data has the highest accuracy and consistency after calibration, denoising and standardization.

[0081] Through these preprocessing steps, we built a high-quality multimodal dataset, which laid a solid foundation for subsequent feature extraction and detection. By combining autoencoders and attention mechanisms, we can accurately extract and fuse key features in multimodal data, thereby greatly improving the drone system's ability to handle heterogeneous data in complex environments. Combining the Transformer attention mechanism and adaptive weighting strategy, we dynamically weight the importance of different sensor data to ensure the accuracy and reliability of the fusion results.

[0082] Use deep convolutional autoencoders to perform nonlinear transformations on data from different sensors. The encoder part of the autoencoder consists of a multi-layer convolutional network that can gradually extract low-level to high-level features of the data. For data of different modalities (such as image data, point cloud data, spectral data), the autoencoder generates their own potential feature representations, which compress the redundant information in the original data while retaining key structural features.

[0083] Through this nonlinear transformation, the UAV system can effectively map various sensor data into a unified feature space. In this feature space, the features of different modalities have been adjusted to the same scale and distribution, which lays the foundation for the subsequent fusion process.

[0084] In order to adaptively adjust the fusion parameters of various sensor data during the data fusion process, an optimization strategy based on reinforcement learning is introduced into the UAV system. The specific implementation process is as follows:

[0085] The policy function learns to decide which actions to choose in different states to maximize future rewards. The reward function evaluates the performance of each action in a specific state and rewards or punishes based on indicators such as the accuracy of the fusion results, the degree of redundancy reduction, and the effect of conflict resolution. Through continuous interaction with the environment, the UAV system gradually optimizes the policy function and learns the best combination of fusion parameters in different environments. Ultimately, the UAV system can dynamically select the optimal parameters in actual applications to cope with environmental changes. The UAV system has the ability to adaptively adjust the fusion parameters according to the current environmental status in actual applications to ensure high-precision data to support the UAV's autonomous flight and intelligent obstacle avoidance tasks.

[0086] Wasserstein GAN (WGAN) is used to improve the stability of training and ensure the reliability of image enhancement effects. First, WGAN guides the optimization of the generator by calculating the Wasserstein distance between the generated distribution and the true distribution. Compared with the traditional JS divergence (Jensen-Shannon Divergence), the Wasserstein distance can still provide meaningful gradient information when the distribution is not completely overlapping. Therefore, WGAN ensures that the generator can continue to learn to generate high-quality enhanced images similar to real images by optimizing the Wasserstein distance. Secondly, in WGAN, the traditional discriminator is replaced by "Critic", which is no longer a binary classification task, but directly outputs a real number, which represents the position of the input sample in the Wasserstein distance. By optimizing the Critic, WGAN can stably converge to the true distribution. By learning the gradient of the Wasserstein distance, the Critic network can more accurately evaluate the difference between the generated image and the real image. This evaluation not only helps the generator improve the quality of the generated image, but also can feedback more meaningful gradient information, making the training process more stable.

[0087] SRGAN focuses on converting low-resolution images into high-resolution images. In particular, with the support of the residual network (ResNet) structure, it can retain the detailed information of the image and improve the resolution of the damage features. SRGAN combines the pixel-level loss function with the perceptual loss (Perceptual Loss), and further optimizes the clarity and naturalness of the image by comparing the difference between the generated image and the real image in the feature space.

[0088] In addition, a diffusion model is introduced to generate high-quality enhanced images by gradually adding noise and learning the denoising process. This model is particularly good at removing noise while maintaining image structural information, making details such as cracks and corrosion in bridge images clearer. Compared with traditional GAN ​​and SRGAN, the diffusion model performs well in generating natural and high-fidelity images.

[0089] In the entire image enhancement process, the convolutional layer of the convolutional neural network, batch normalization (BatchNormalization) and other technologies are used to optimize the stability and efficiency of image enhancement. Finally, after processing by these deep learning algorithms, the enhanced images are not only significantly improved in resolution and clarity, but also effectively reduce redundant information, providing a more accurate and reliable data basis for subsequent damage identification and diagnosis.

[0090] In step S2, the present invention adopts advanced computer vision technology and deep learning models to build a bridge damage detection model and comprehensively analyze the multi-dimensional characteristics of the bridge to achieve comprehensive detection and accurate identification of the bridge structure.

[0091] like Figure 2 As shown in Figures (a) and (b) in Figure 1, the feature vector of the image, the global feature vector of the point cloud, and the spectral feature vector are determined through the input module and used as the input of the multimodal Transformer network, which specifically includes the following steps:

[0092] The acquired image data of the bridge to be inspected is preprocessed through a convolutional neural network (CNN) to extract high-level features of the image and convert them into feature vectors of a fixed size, namely, an I vector sequence; the extracted feature vectors of a fixed size are input into a multimodal Transformer network as input of the image modality.

[0093] For point cloud data, use PointNet, PointNet++ or other point cloud processing networks to learn the features of each point and aggregate global features to generate a global feature vector of the point cloud, that is, a P vector sequence; the generated point cloud feature vector is input into the multimodal Transformer network.

[0094] The acquired spectral data of the bridge to be inspected is processed by 1D convolutional neural network 1D-CNN or recurrent neural network RNN / LSTM to extract local features in the spectral data and obtain spectral feature vectors, i.e., S vector sequences. The obtained spectral feature vectors are input into the multimodal Transformer network.

[0095] Each cross-attention module fuses the feature vectors of each modality corresponding to the image data, point cloud data, and spectral data through the cross-attention mechanism and the Transformer self-attention mechanism, which specifically includes the following steps:

[0096] The multimodal Transformer network includes a multi-head self-attention mechanism, an encoder layer, and position encoding;

[0097] Among them, the Transformer self-attention mechanism, as the core component of the multimodal Transformer network, captures the correlation within the feature vector and across modalities by calculating the relationship between Query, Key and Value; in the multimodal Transformer network, the feature vector of each modality is mapped to Query, Key and Value vectors, and the similarity between features is calculated; the similarities between the calculated features are weighted and summed to generate an enhanced feature representation.

[0098] The encoder layer consists of multiple stacked Transformer layers, each of which includes a Transformer self-attention mechanism and a feedforward neural network. The feature vector of each modality is transmitted and processed layer by layer in the encoder to capture the global context information of data in different modalities. The output of the encoder is a fused multimodal feature representation, which retains the key information in the input data and captures the complementary characteristics between modalities through the Transformer attention mechanism to obtain the feature vector of each modality.

[0099] Since the Transformer model itself does not have the ability to process sequence information, position encoding is introduced in the Transformer encoder to help the model recognize sequence information. In the multimodal Transformer network, position encoding provides position information for each modal feature vector to obtain temporally and spatially consistent features of each modality.

[0100] Obtaining the fused feature vector includes the following steps:

[0101] After being processed by multiple Transformer layers, the acquired temporally and spatially consistent features of each modality are interacted and fused at different levels; feature concatenation, weighted summation, or Transformer layers are used for feature fusion to generate a fused feature vector that integrates key information from different modalities to form a high-quality multimodal joint representation.

[0102] The generated multimodal joint representation can be further processed through a series of fully connected layers to adapt to the needs of specific tasks (including classification, regression, detection, etc.). These layers usually contain nonlinear activation functions and normalization layers to ensure the training stability and representation ability of the model. Finally, the output layer outputs the corresponding prediction results (including the category and location of bridge damage, etc.) according to the requirements of the specific task.

[0103] During the training process of the constructed bridge damage detection model, loss functions suitable for the task are used, such as cross entropy loss (for classification tasks), mean square error (for regression tasks), etc.

[0104] In order to improve the effect of multimodal data fusion, additional regularization terms can be introduced, such as Multimodal Alignment Loss, to ensure that the features of each modality remain consistent during the fusion process. Commonly used optimization algorithms include Adam, RMSprop, etc., which can effectively handle large-scale parameter optimization problems in multimodal Transformer networks. During training, model performance can be further optimized by adjusting hyperparameters such as learning rate and batch size, and introducing a learning rate scheduler (such as CosineAnnealing).

[0105] The present invention integrates adaptive attention mechanisms (such as SE-Block and CBAM) into the constructed bridge damage detection model to further enhance the model's feature representation capabilities, especially when dealing with complex structural damage identification tasks. These adaptive attention mechanisms can dynamically adjust the model's attention to different damage features, allowing the network to focus more accurately on key structural defects, thereby improving the accuracy and reliability of damage identification.

[0106] SE-Block is a channel attention mechanism that can enhance the model's response to important features by adaptively reweighting channel features. In the bridge damage detection model, SE-Block can be integrated after the feature extraction module of image data, point cloud data, and spectral data, that is, before each modal feature vector is input into the Transformer encoder. By performing channel weighting on the feature vector of each modality, the response to key damage features is enhanced. SE-Block first compresses the spatial dimensions of each feature map into a global feature representation through the Squeeze operation. Then, through the Excitation operation, the correlation between channels is learned and the features of each channel are adaptively reweighted. This process can enhance the model's attention to important damage features, making the final generated feature representation more discriminative.

[0107] CBAM combines spatial attention and channel attention mechanisms, which can optimize feature representation from both spatial and channel dimensions, and can further improve the performance of bridge damage detection models in damage identification. CBAM can be integrated between Transformer encoder layers or after the feature extraction module of each modality.

[0108] Similar to SE-Block, CBAM can process the feature vector of each modality to optimize its spatial and channel features. First, the feature map is reduced in dimension through global average pooling and global maximum pooling to generate two different channel attention maps. Then, these two attention maps are fused through MLP (multi-layer perceptron) to generate the final channel attention map, and the original feature map is weighted. After the channel attention module, CBAM processes the feature map in the spatial dimension, generates the spatial attention map through maximum pooling and average pooling operations, and further optimizes the channel-weighted feature map.

[0109] After the image data, point cloud data and spectral data have undergone preliminary feature extraction, the extracted features are first adaptively weighted through SE-Block or CBAM, so that the features processed by the attention mechanism will be more representative and help capture key damage information in different modal data. Between each layer of the multi-layer Transformer encoder, a CBAM module can be inserted to further strengthen the spatial and channel representation of the features, so that the features have been optimized through CBAM before being processed by the Transformer self-attention mechanism, thereby improving the overall network's sensitivity to damage features.

[0110] In step S3, during the bridge damage detection process, the joint feature representation after SE-Block and CBAM processing can more accurately detect and classify various bridge damages, such as cracks, corrosion, and deformation. With these enhanced feature representations, the model can generate more detailed and accurate damage reports, including the location, type, and severity of defects.

[0111] In order to further improve the comprehensiveness and accuracy of detection, time series analysis technology is also introduced in multimodal data processing. By combining the time series characteristics of multimodal data, the UAV system can dynamically capture the changing trend of the bridge structure and update the damage identification results in real time. This time series analysis not only enhances the UAV system's ability to monitor the health status of the bridge, but also provides important reference data for the long-term maintenance of the bridge. The specific steps are as follows:

[0112] (1) Data acquisition and processing: The UAV system first collects data regularly from multiple sensors of the bridge (such as RGB cameras, LiDAR, infrared cameras, spectral sensors, etc.). The multimodal data collected each time forms a sample at a timestamp. Over time, the UAV system accumulates samples of multiple timestamps to form a time series data set. Before performing time series analysis, the multimodal data needs to be preprocessed. The preprocessing steps include: denoising, alignment, normalization, and feature extraction. In order to unify the time dimension of different modal data, it may be necessary to interpolate or sample the data to ensure that the data at each timestamp is homogeneous.

[0113] (2) Constructing a temporal feature extraction network: In order to extract temporal features, a recurrent neural network (RNN) architecture is designed, which can process the temporal features of multimodal inputs. Different from traditional RNN, a multi-channel RNN may be used here, with each channel corresponding to one modal data. In this way, the UAV system can extract time series features from each modal data. After each modal data is processed by RNN, the UAV system fuses the temporal features of each modality through a temporal feature fusion layer. This layer may use weighted summation or attention mechanism (similar to the self-attention mechanism in Transformer) to highlight the temporal features of key modalities. In order to capture the changing trends at different time scales, a multi-scale convolution layer (MSC) can be designed. This layer performs convolution operations on the time dimension through multiple convolution kernels of different sizes to extract multi-scale temporal features. In this way, the UAV system can capture both short-term sudden changes and long-term slow changes. In addition, a time progressive module can be introduced to identify key change nodes in the time series by sliding windows and gradual accumulation. This module is able to identify sudden changes in structural states within a specific time period and mark these changes as potential structural damage.

[0114] (3) Dynamic damage identification and updating: After extracting the time series features, the UAV system needs to predict the future structural state. The designed time series prediction module may be based on an extended state space model or a nonlinear dynamic system model, which can predict future damage trends based on current and historical data. The output of the prediction module will be used to update the damage identification results in real time. Whenever new data is input into the UAV system, the UAV system will correct the current damage identification results through a dynamic update mechanism (such as Kalman filter or Bayesian update) to ensure that the output results reflect the latest structural state. In order to evaluate the long-term damage trend, the UAV system introduces a damage accumulation model that can accumulate damage information in each time period and evaluate the long-term health status of the structure through a weighted accumulation and decay mechanism. Based on the damage accumulation model, the UAV system can perform trend analysis, identify potential structural degradation trends, and issue early warning signals when necessary. For example, the UAV system can predict the failure time of a bridge structure by observing the acceleration of crack propagation speed and recommend preventive maintenance.

[0115] (4) Implementation and optimization of the timing analysis system: The entire timing analysis system adopts a modular design and is mainly divided into modules such as timing feature extraction, dynamic modeling of timing data, dynamic update and trend analysis. The modules exchange data and communicate with each other through a unified interface to achieve overall functional optimization. By integrating adaptive optimization algorithms (such as genetic algorithms or particle swarm optimization algorithms), the parameters of the timing analysis model are tuned to improve the prediction accuracy and response speed of the model. In order to meet the requirements of real-time processing, the UAV system improves the processing speed through GPU acceleration and distributed computing technology. The timing processing process of multimodal data is designed as a parallel operation to maximize the response efficiency of the UAV system.

[0116] (5) Long-term monitoring and early warning: The application of this time series analysis technology can monitor the health status of bridge structures in real time, identify early signs of structural degradation, and provide targeted maintenance recommendations. Through time series analysis, the UAV system can predict the long-term change trend of bridge structures, provide scientific data support for bridge management departments, and optimize bridge maintenance and operation strategies.

[0117] Finally, through the comprehensive application of deep learning models with adaptive attention mechanisms, Transformer architecture and time series analysis, the comprehensiveness and accuracy of bridge structure damage detection have been effectively improved. It can not only identify and mark various structural defects, but also provide a more comprehensive analysis of bridge health status, providing strong data support for bridge maintenance and management.

[0118] This application also proposes a bridge damage detection system, comprising:

[0119] A data acquisition module, used to acquire image data, point cloud data and spectral data of the bridge to be inspected;

[0120] A bridge damage detection model construction module is used to construct a network model; the network model includes an input module and a feature fusion module; multiple cross-attention modules and a multimodal Transformer network are introduced into the feature fusion module, each cross-attention module fuses the feature vectors of each modality corresponding to the image data, point cloud data and spectral data through the cross-attention mechanism and the Transformer self-attention mechanism, obtains the feature representation after the fusion of the feature vectors of each modality, and inputs it into the DNN classification layer, and outputs the classification result; the constructed network model is trained to obtain a bridge damage detection model;

[0121] The bridge damage detection module is used to input the acquired image data, point cloud data and spectral data of the bridge to be detected into the input module of the bridge damage detection model to determine the feature vectors of the image, point cloud and spectrum; the feature fusion module fuses the feature vectors of the image, point cloud and spectrum, and obtains the fused feature vector by interacting and fusing the feature representations of each modality at different levels, and outputs the bridge damage detection type through the classifier in the DNN classification layer.

[0122] Through advanced data processing and analysis technologies, the accuracy and efficiency of bridge health monitoring are improved, providing comprehensive technical support for bridge maintenance and management.

[0123] In the data cleaning and processing module, data cleaning methods based on adaptive filtering algorithms, such as Kalman filtering and adaptive wavelet transform, are used to improve the accuracy and reliability of different sensor data in complex environments. Through these algorithms, the drone system can effectively eliminate noise and interference, ensure that the collected multi-source data has higher accuracy and consistency, and lay a solid foundation for subsequent health assessments.

[0124] By comprehensively analyzing the data from different sensors, a comprehensive assessment of the health status of the bridge can be achieved. Specifically, after obtaining the fused data from the multi-source heterogeneous sensor data fusion algorithm, deep learning models such as the long short-term memory network (LSTM) and the gated recurrent unit (GRU) are used.

[0125] Among them, the LSTM network is good at processing sequence data and can capture the time-varying trend of the health status of the bridge. The UAV system inputs the feature vector after multimodal fusion into the LSTM network, processes the time series data in a recursive way, and learns the law of the change of the health status of the bridge over time. Through the LSTM analysis of the time series, the UAV system can identify the trend of material degradation, such as the aging rate of the coating and the crack propagation rate.

[0126] As a lightweight alternative to LSTM, GRU has fewer parameters and faster computing speed. The UAV system can select GRU to process time series data according to mission requirements, which is particularly suitable for real-time applications with limited resources. At the same time, through GRU's time series analysis of point cloud and infrared data, the UAV system can identify and predict the dynamic changes of bridge structure deformation and changes in stress distribution due to load changes.

[0127] Based on the analysis results of LSTM and GRU, the UAV system comprehensively evaluates the overall health status of the bridge. A health score is generated to reflect the current health level of the bridge. Through the cumulative analysis of the time series model, the UAV system can predict the future deterioration trend of the bridge and provide preventive maintenance suggestions.

[0128] Based on this, the generated assessment report contains detailed information such as the description of the overall health status of the bridge, the degree of material degradation, the range of structural deformation, the impact of load changes, etc. It also includes visualization charts of various types of damage, such as crack extension charts, stress distribution charts, temperature change charts, etc., to help managers intuitively understand the health status of the bridge. Based on the analysis results, the drone system provides targeted maintenance suggestions, including areas that need to be focused on, possible repair measures, and the recommended time for the next inspection.

[0129] The present application also designs a real-time monitoring system, which combines time series analysis and abnormal warning prediction algorithms, and uses time series analysis models (such as AutoRegressive Integrated MovingAverage, ARIMA) and anomaly detection algorithms (such as Isolation Forest) to accurately identify potential structural damage and abnormal conditions, as follows:

[0130] ARIMA is a classic time series analysis model, which is widely used to predict and analyze time series data. The ARIMA model models time series data through three parts: autoregression (AR), difference (I), and moving average (MA). ARIMA is used to analyze the time series data of bridge structure health indicators (such as stress, displacement, vibration, etc.), predict future status, and identify long-term trends and cyclical changes. In order to deal with the nonlinear characteristics that may exist in bridge health monitoring, ARIMA is combined with nonlinear models (such as LSTM, GRU) to form a hybrid model. The hybrid model first uses ARIMA to capture the linear part, and then uses the nonlinear model to process the residual part, thereby improving the prediction accuracy. For the above-mentioned multimodal data, the multivariate ARIMA model (VARIMA) is introduced, which can process multiple time series data at the same time and capture the relationship between each variable. By jointly modeling the multimodal data of the bridge (such as stress, temperature, humidity, etc.), the health status of the bridge can be predicted more accurately.

[0131] Isolation Forest is an unsupervised learning algorithm based on a tree structure, which is specifically used for anomaly detection. It constructs a set of decision trees by randomly selecting features and split points of the data, and then determines whether the data point is an anomaly based on its isolation depth (i.e., the length of the path in the tree). A shorter path means that the point is easier to isolate and therefore may be an anomaly. Isolation Forest is used to detect anomalies in multimodal data, such as sudden stress increases, temperature anomalies, changes in vibration patterns, etc. These anomalies may indicate structural damage or potential failures. In order to combine time series characteristics, the anomaly detection ability of the model is improved by inputting time dependence into the model as an additional feature, or preprocessing the time series before isolating the data points, so that it can handle time series data. At the same time, Isolation Forest is used in combination with LSTM-Autoencoder. LSTM-Autoencoder first learns the time series pattern of normal data and provides the reconstruction error as input to Isolation Forest, thereby enhancing the detection ability of complex time series anomalies.

[0132] The drone system predicts the future health status of the bridge by analyzing the time series characteristics of multimodal data and issues early warning signals, helping managers take preventive maintenance measures before the bridge suffers major damage and ensuring the long-term safety of the bridge.

[0133] At the same time, in order to achieve real-time data processing and efficient transmission of the bridge monitoring system, a bridge monitoring system based on the Internet of Things architecture was developed. Through distributed computing, the drone system can efficiently process and store massive amounts of data to ensure real-time response capabilities. At the same time, the combination of edge computing and cloud computing optimizes the timeliness and reliability of data processing. The drone system has been tested in an actual bridge environment to verify its reliability and accuracy, ensuring real-time monitoring and early warning of the health status of the bridge.

[0134] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

[0135] In addition, unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as commonly understood by those skilled in the art to which the present invention belongs. All documents mentioned in this specification are incorporated by reference to disclose and describe the methods related to the documents. In the event of any conflict with any incorporated document, the content of this specification shall prevail.

Claims

1. A bridge damage detection method, characterized in that: The following steps are involved: Obtain image data, point cloud data and spectral data of the bridge to be inspected; Constructing a network model; the network model includes an input module and a feature fusion module; In the feature fusion module, multiple cross-attention modules and multimodal Transformer networks are introduced. Each cross-attention module fuses the feature vectors of each modality corresponding to the image data, point cloud data and spectral data through the cross-attention mechanism and the Transformer self-attention mechanism, obtains the feature representation after the fusion of the feature vectors of each modality, and inputs it into the DNN classification layer to output the classification result; the constructed network model is trained to obtain the bridge damage detection model; The acquired image data, point cloud data and spectrum data of the bridge to be inspected are input into the input module of the bridge damage detection model to determine the feature vectors of the image, point cloud and spectrum; The fusion module fuses the feature vectors of the image, point cloud and spectrum, and obtains the fused feature vector by interacting and fusing the feature representations of each modality at different levels. The bridge damage detection type is output through the classifier in the DNN classification layer.

2. The bridge damage detection method according to claim 1, characterized in that: The obtaining of image data, point cloud data and spectral data of the bridge to be inspected specifically includes: The drone can autonomously navigate to collect high-resolution images of the bridge to be inspected, identifying defects on the bridge surface, including cracks, rust, and surface damage; The three-dimensional point cloud data generated by LiDAR equipment provides the geometric shape and structural information of the bridge and identifies the deformation, displacement and geometric defects of the bridge. The temperature distribution data of the bridge is captured by infrared cameras to detect internal temperature anomalies caused by corrosion, cracks or material defects. The spectral data provided by multispectral or hyperspectral sensors are used to analyze the changes in the chemical composition of the material of the bridge to be inspected, including coating aging and corrosion reactions.

3. The bridge damage detection method according to claim 2, characterized in that: The determining of the feature vectors of the image, point cloud and spectrum specifically comprises the following steps: The acquired image data of the bridge to be inspected is preprocessed by a convolutional neural network (CNN) to extract high-level features of the image and convert them into feature vectors of fixed size; the extracted feature vectors of fixed size are input into the multimodal Transformer network of the feature fusion module as input of the image modality; For point cloud data, use PointNet, PointNet++ or other point cloud processing networks to learn the features of each point and generate a global feature vector of the point cloud through global feature aggregation; input the generated point cloud feature vector into the multimodal Transformer network of the feature fusion module; The acquired spectral data of the bridge to be inspected is processed by 1D convolutional neural network 1D-CNN or recurrent neural network RNN / LSTM to extract local features in the spectral data and obtain spectral feature vectors, which are then input into the multimodal Transformer network of the feature fusion module.

4. The bridge damage detection method according to claim 3, characterized in that: Each cross attention module fuses the feature vectors of each modality corresponding to the image data, point cloud data and spectral data respectively through the cross attention mechanism and the Transformer self-attention mechanism, specifically including the following steps: After the image data, point cloud data and spectral data have undergone preliminary feature extraction, the extracted features are first adaptively weighted through SE-Block or CBAM; The multimodal Transformer network includes the Transformer self-attention mechanism, encoder layer, and position encoding; Among them, the Transformer self-attention mechanism, as the core component of the multimodal Transformer network, captures the correlation within the feature vector and across modalities by calculating the relationship between Query, Key and Value; in the multimodal Transformer network, the feature vector of each modality is mapped to Query, Key and Value vectors, and the similarity between features is calculated; the similarity between the calculated features is weighted and summed to generate an enhanced feature representation; The encoder layer consists of multiple stacked Transformer layers, each of which includes a Transformer self-attention mechanism and a feedforward neural network. The feature vector of each modality is transmitted and processed layer by layer in the encoder to capture the global context information of data of different modalities. The output of the encoder is the fused multi-modal feature representation, which retains the key information in the input data and captures the complementary characteristics between modalities through the Transformer attention mechanism to obtain the feature vector of each modality. By introducing position encoding in the Transformer encoder, in the multimodal Transformer network, position encoding provides position information for the feature vector of each modality, obtaining feature representations of each modality that are consistent in time and space.

5. The bridge damage detection method according to claim 4, characterized in that: The determining of the feature vector of the image, the global feature vector of the point cloud and the spectral feature vector also includes: The following steps are involved: After the channel attention mechanism SE-Block is integrated into the feature extraction modules of image data, point cloud data and spectral data, the feature vector of each modality is channel-weighted before it is input into the Transformer encoder. SE-Block compresses the spatial dimension of each feature map into a global feature representation through the Squeeze operation, learns the correlation between channels through the Excitation operation, and adaptively reweights the features of each channel.

6. The bridge damage detection method according to claim 5, characterized in that: The step of obtaining the feature representation after fusion of the feature vectors of each modality specifically includes the following steps: After being processed by multiple stacked Transformer layers of the Transformer self-attention mechanism, the acquired temporally and spatially consistent feature representations of each modality are interacted and fused at different levels; feature concatenation, weighted summation, or Transformer layers are used for feature fusion to generate a fused feature vector that integrates key information from different modalities to form a multimodal joint representation.

7. The bridge damage detection method according to claim 6, characterized in that: The output classification result specifically includes: The generated multimodal joint representation is processed through multiple fully connected layers of the DNN classification layer, and classified, regressed or detected through the softmax classifier to output the type or location of the bridge damage to be detected.

8. The bridge damage detection method according to claim 7, characterized in that: It also includes extracting time series features from each modality data, which specifically includes the following steps: By introducing a recursive neural network, the feature vector of the image, the global feature vector of the point cloud, and the spectral feature vector are processed, and the time series features are extracted from each modal data to obtain the time series features of each modality; Introduce a time series prediction module and a dynamic update mechanism. Based on an extended state space model or a nonlinear dynamic system model, input current and historical data, predict future damage trends, and output real-time updated damage identification results. When new data is input, the current damage identification results are corrected through a dynamic update mechanism. A damage accumulation model is introduced, which accumulates damage information in each time period. The damage information in each time period is processed through weighted accumulation and attenuation mechanism, the long-term health status of the bridge is evaluated for trend analysis, and potential structural degradation trends are identified; by observing the acceleration of crack propagation speed, the failure time of the bridge structure is predicted, and preventive maintenance is recommended.

9. A bridge damage detection system, characterized in that: include: A data acquisition module, used to acquire image data, point cloud data and spectral data of the bridge to be inspected; A bridge damage detection model building module is used to build a network model; the network model includes an input module and a feature fusion module; In the feature fusion module, multiple cross-attention modules and multimodal Transformer networks are introduced. Each cross-attention module fuses the feature vectors of each modality corresponding to the image data, point cloud data and spectral data through the cross-attention mechanism and the Transformer self-attention mechanism, obtains the feature representation after the fusion of the feature vectors of each modality, and inputs it into the DNN classification layer to output the classification result; the constructed network model is trained to obtain the bridge damage detection model; The bridge damage detection module is used to input the acquired image data, point cloud data and spectrum data of the bridge to be detected into the input module of the bridge damage detection model to determine the feature vectors of the image, point cloud and spectrum; feature The fusion module fuses the feature vectors of the image, point cloud and spectrum, and obtains the fused feature vector by interacting and fusing the feature representations of each modality at different levels. The bridge damage detection type is output through the classifier in the DNN classification layer.

Images