Multi-modal vision-based crack risk assessment method and system after bridge crossing of vehicle

Abstract

The invention provides a multi-modal vision-based crack risk assessment method and system after a vehicle passes through a bridge, relates to the field of bridge detection, and solves the technical problem that the crack risk assessment is inaccurate due to the fact that the prior art only depends on static crack width risk judgment and does not fuse response characteristics such as a dynamic expansion rate after the vehicle passes through the bridge. The method comprises the following steps: constructing a cross-modal feature map based on an obtained multi-modal data sequence; inputting the cross-modal feature map into an improved U-Net + + lightweight segmentation network, generating a crack binary segmentation mask, and calculating the width of the crack based on the crack binary segmentation mask; based on multi-view stereoscopic vision and depth sensor data, a three-dimensional point cloud model of the crack is reconstructed, the depth of the crack is calculated, and the dynamic expansion rate and width variation of the crack after the vehicle passes through the bridge are calculated; and calculating a risk index of the crack based on the crack parameter. The method is used in the bridge crack risk assessment process.

Description

Technical Field

[0001] This application relates to the field of bridge inspection, and in particular to a method and system for assessing the risk of cracks after a vehicle crosses a bridge based on multimodal vision. Background Technology

[0002] With the continuous increase in highway traffic volume and the aging of many bridges in my country, vehicle dynamic loads have become a key external factor inducing and accelerating the propagation of cracks in bridge structures. Cracks, as an early and direct manifestation of structural damage, directly reflect the remaining load-bearing capacity and failure risk of the structure through minute but crucial dynamic changes following the typical transient event of a vehicle crossing the bridge. Current technologies, based on static geometric parameters of cracks, classify cracks as high-risk if their width exceeds a preset threshold, failing to incorporate dynamic factors after a vehicle crosses the bridge, such as the dynamic propagation rate, leading to inaccurate crack risk assessments. Summary of the Invention

[0003] This application provides a method and system for assessing crack risk after a vehicle crosses a bridge based on multimodal vision. It solves the technical problem that existing technologies rely solely on static crack width judgments and do not integrate response features such as the dynamic expansion rate after a vehicle crosses the bridge, resulting in inaccurate crack risk assessment.

[0004] To achieve the above objectives, this application adopts the following technical solution: Firstly, a multimodal vision-based method for assessing the risk of cracks after a vehicle crosses a bridge is provided, including: The multimodal data sequence of the key monitoring area of ​​the bridge is obtained. The multimodal data sequence is continuous data before the arrival of the vehicle, when the vehicle passes, and after the vehicle passes. The key monitoring area is the area where the structure is under concentrated stress or is prone to damage. The multimodal data sequence is subjected to spatiotemporal alignment and enhancement processing to construct a cross-modal feature map; The cross-modal feature map is input into the improved U-Net++ lightweight segmentation network to generate a binary segmentation mask for cracks. Based on the binary segmentation mask, the width of the crack is calculated. Based on multi-view stereo vision and depth sensor data, a three-dimensional point cloud model of the crack is reconstructed; the depth of the crack is calculated based on the three-dimensional point cloud model; and the dynamic expansion rate and width change of the crack after the vehicle crosses the bridge are calculated based on the high-frequency image inter-frame optical flow method. Based on the crack parameters, the risk index of the crack is calculated; the risk level of the crack is assessed according to the risk index; the crack parameters include the crack width, depth, dynamic propagation rate and width variation.

[0005] Based on the above technical solutions, in the multimodal vision-based bridge crack risk assessment method provided in this application, traditional bridge crack detection methods mostly rely on manual or single-sensor static images, which are difficult to comprehensively and accurately depict the true state of the cracks, especially unable to obtain their dynamic behavior (such as instantaneous opening and expansion rate) and three-dimensional geometric features (such as depth) under vehicle loads, leading to subjective, delayed, or even misjudgment of risk assessment. To solve this problem, this solution constructs a multimodal intelligent perception and assessment system for the "vehicle crossing the bridge" scenario: visible light, infrared, depth, and vibration data are simultaneously collected in key stress areas before, during, and after the vehicle's arrival, and high-precision spatiotemporal alignment and modal enhancement are used to ensure the consistency of multi-source information; a high-precision crack mask is generated using an improved U-Net++ lightweight network to calculate the crack width, and multi-view stereo vision and depth sensing are integrated to reconstruct a three-dimensional point cloud to accurately invert the crack depth; combined with a high-frequency optical flow algorithm, the width change and dynamic expansion rate at the physical scale are extracted from the inter-frame displacement. These dynamic and static parameters are integrated into a comprehensive risk index to achieve scientific graded assessment. This proposed solution breaks through the limitations of traditional static and single-modal monitoring, and achieves synchronous, accurate and robust quantification of the multi-dimensional characteristics of cracks in terms of "geometric + dynamic + mechanical" dimensions, which significantly improves the objectivity, real-time performance and engineering applicability of bridge health status assessment.

[0006] In conjunction with the first aspect above, in one possible implementation, the calculation of the crack risk index includes: Based on crack parameters, through formula The risk index R of the crack was calculated. in, , , , and is the weighting coefficient; D is the depth of the crack. This represents the critical threshold for depth. This represents the change in the width of the crack. This is the critical threshold for the width variation. The dynamic propagation rate of the crack. This is the critical threshold for the dynamic expansion rate. For the material's yield strength, This represents the critical threshold for the material's yield strength. For temperature gradient, The temperature gradient is the critical threshold for the temperature gradient; the temperature gradient is the temperature difference per unit length along the thickness direction in the region where the crack is located in the bridge structure.

[0007] In conjunction with the first aspect above, in one possible implementation, the process of constructing the cross-modal feature map includes: Multimodal data includes at least visible light images, infrared thermal images, depth images, and structural vibration signals; Spatial registration and temporal synchronization are performed on the multimodal data; The system enhances the illumination invariance of visible light images, suppresses thermal noise in infrared thermal images, repairs holes in depth images, and performs frequency band selective denoising on vibration signals. The processed modal data are input into the corresponding feature extraction modules to obtain their respective multi-scale features. Calculate the feature attention weights for each modality, and perform weighted fusion of the multi-scale features; The fused features are then used to restore spatial resolution and integrate contextual information through an encoder-decoder structure to obtain a cross-modal feature map.

[0008] In conjunction with the first aspect above, in one possible implementation, calculating the weights of each modality's data during the fusion process includes: ; H( )=- ; Where i is the sequence number of the modal data. H( represents the weight of the i-th modality data) ) is the feature map of the i-th modality data. The information entropy, p(f) is the probability distribution of pixel f in the feature map.

[0009] In conjunction with the first aspect above, in one possible implementation, the improvement of the U-Net++ lightweight segmentation network includes: In the downsampling path of the encoder, at least one level introduces a multi-scale dilated convolution module, which uses dilated convolution kernels with different dilation rates in parallel. In the upsampling path of the decoder, an attention fusion module is set up. The attention fusion module performs channel attention and spatial attention calculations on the skip connection features from the corresponding level of the encoder and the current decoding features, respectively, and generates weighted fusion features based on the joint response of global average pooling and global max pooling. A learnable threshold segmentation module is embedded before the network output layer. The learnable threshold segmentation module dynamically outputs a binarized threshold based on the statistical distribution of the input features.

[0010] In conjunction with the first aspect above, in one possible implementation, calculating the width of the crack includes: The crack binarization segmentation mask obtained based on the improved U-Net++ lightweight segmentation network is used to extract the center line of the crack and measure the pixel width along the normal direction perpendicular to the crack direction at each point. The average of several pixel widths is the width of the crack. The pixel width is converted into physical scale based on camera calibration parameters.

[0011] In conjunction with the first aspect above, in one possible implementation, the reconstructing of the 3D point cloud model of the crack based on multi-view stereo vision and depth sensor data includes: Feature matching and pose estimation are performed using multi-view visible light images to generate an initial point cloud based on stereo vision; the depth map acquired by the depth sensor is converted into an auxiliary point cloud and registered and fused with the initial point cloud to obtain a three-dimensional point cloud model.

[0012] In conjunction with the first aspect above, in one possible implementation, calculating the crack depth based on a three-dimensional point cloud model includes: Based on the three-dimensional point cloud model of the crack, a robust plane fitting algorithm is used to fit the reference plane of the structural surface from the undamaged area around the crack. The maximum or average value of the vertical distance from the crack point cloud to the reference plane of the structural surface is calculated to obtain the depth of the crack.

[0013] In conjunction with the first aspect above, in one possible implementation, calculating the dynamic propagation rate and width change of the crack after the vehicle crosses the bridge includes: The inter-frame pixel displacement of the high-frequency image sequence acquired during the vehicle crossing the bridge is calculated using an optical flow algorithm, and the local normal direction is determined based on the crack orientation; the local normal direction is a unit vector perpendicular to the tangent direction at each point on the crack centerline; The displacement vector is projected onto the local normal direction to obtain the normal displacement, which is then converted into a physical scale. The normal displacement is spatiotemporally aggregated and differentiated in the time domain to obtain the crack width variation and dynamic propagation rate.

[0014] Secondly, this application provides a multimodal vision-based system for assessing the risk of bridge cracks after a vehicle crosses a bridge, comprising an analysis module and a risk assessment module. The analysis module acquires multimodal data sequences of key monitoring areas of the bridge, which are continuous data before, during, and after a vehicle's arrival. The key monitoring areas are areas of concentrated structural stress or prone to damage. The analysis module performs spatiotemporal alignment and enhancement processing on the multimodal data sequences to construct a cross-modal feature map. The cross-modal feature map is input into an improved U-Net++ lightweight segmentation network to generate a binary segmentation mask for cracks. Based on the binary segmentation mask, the crack width is calculated. A three-dimensional point cloud model of the crack is reconstructed based on multi-view stereo vision and depth sensor data. The crack depth is calculated based on the three-dimensional point cloud model. The dynamic propagation rate and width change of the crack after a vehicle crosses the bridge are calculated using the high-frequency image inter-frame optical flow method. The risk assessment module calculates the risk index of the crack based on crack parameters and assesses the risk level of the crack according to the risk index. The crack parameters include the crack width, depth, dynamic propagation rate, and width change.

[0015] This application provides a method and system for risk assessment of bridge cracks after vehicles pass over them, based on multimodal vision. This system can synchronously sense, accurately quantify, and intelligently assess the multidimensional state of bridge cracks under vehicle loads. By deploying visible light, infrared, depth, and vibration sensors in key stress areas, the system achieves millisecond-level synchronous data acquisition throughout the entire vehicle passage process (before the vehicle arrives – while the vehicle passes – after the vehicle passes), and performs spatiotemporal alignment and modal enhancement to effectively suppress interference from sudden changes in illumination, thermal noise, and depth voids. An improved U-Net++ lightweight network is used for robust segmentation of crack regions, and a 3D point cloud is reconstructed using multi-view stereo vision and depth data to accurately calculate crack width and depth. Simultaneously, the dynamic expansion rate and width change at the physical scale are extracted using a high-frequency image optical flow method. Finally, geometric, dynamic, and mechanical parameters are integrated into a comprehensive risk index to achieve three levels of intelligent assessment: low risk, early warning, and emergency. This solution breaks through the limitations of traditional static and single-modal detection, and for the first time realizes the integrated characterization of cracks in terms of "shape, movement and force", which significantly improves the accuracy, real-time performance and adaptability to the field environment, and provides reliable and deployable technical support for intelligent operation and maintenance of bridges.

[0016] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0017] Figure 1 A system architecture diagram of a multimodal vision-based vehicle-bridge crack risk assessment system provided in this application embodiment; Figure 2 A flowchart illustrating a method for assessing the risk of cracks after a vehicle crosses a bridge based on multimodal vision, provided in an embodiment of this application; Figure 3 This is a flowchart illustrating a method for constructing a cross-modal feature map, as provided in an embodiment of this application. Detailed Implementation

[0018] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The multimodal vision-based method for assessing the risk of cracks after a vehicle crosses a bridge, as provided in this application, can be applied to a multimodal vision-based system for assessing the risk of cracks after a vehicle crosses a bridge. Figure 1 As shown, the communication system includes: an analysis module and a risk assessment module; The analysis module is used to acquire multimodal data sequences of key monitoring areas of the bridge. The multimodal data sequences are continuous data before the arrival of vehicles, during the passage of vehicles, and after the passage of vehicles. The key monitoring areas are areas where the structure is under concentrated stress or is prone to damage. Spatiotemporal alignment and enhancement processing are performed on multimodal data sequences to construct cross-modal feature maps; the cross-modal feature maps are input into the improved U-Net++ lightweight segmentation network to generate a crack binary segmentation mask; and the crack width is calculated based on the crack binary segmentation mask. Based on multi-view stereo vision and depth sensor data, a three-dimensional point cloud model of the crack is reconstructed; the depth of the crack is calculated based on the three-dimensional point cloud model; and the dynamic expansion rate and width change of the crack after the vehicle crosses the bridge are calculated based on the high-frequency image inter-frame optical flow method. The risk assessment module is used to calculate the risk index of a crack based on crack parameters; and to assess the risk level of the crack based on the risk index; wherein the crack parameters include the crack width, depth, dynamic propagation rate, and width variation.

[0020] To address the problem that existing technologies rely solely on static crack width for risk assessment, failing to incorporate dynamic expansion rates and other response features after a vehicle crosses the bridge, leading to inaccurate crack risk assessment, this application provides a multimodal vision-based crack risk assessment method after a vehicle crosses a bridge. This method includes: acquiring multimodal data sequences of key monitoring areas of the bridge; performing spatiotemporal alignment and enhancement processing on the multimodal data sequences to construct a cross-modal feature map; inputting the cross-modal feature map into an improved U-Net++ lightweight segmentation network to generate a crack binary segmentation mask; calculating the crack width based on the crack binary segmentation mask; reconstructing a 3D point cloud model of the crack based on multi-view stereo vision and depth sensor data; calculating the crack depth based on the 3D point cloud model; and calculating the crack depth based on high-frequency image inter-frame illumination. This application employs a flow-based approach to calculate the dynamic propagation rate and width change of cracks after a vehicle crosses a bridge. Based on crack parameters, a risk index is calculated. The risk level of the cracks is then assessed according to this index. Based on this, a multimodal collaborative perception and intelligent assessment system for "vehicle crossing a bridge" scenarios is constructed. This system simultaneously collects visible light, infrared, depth, and vibration data in key stress areas before, during, and after vehicle passage. High-precision spatiotemporal alignment and targeted enhancement processing ensure the consistency of multi-source information. An improved U-Net++ lightweight network is used to achieve robust crack segmentation for width calculation. Multi-view stereo vision and depth sensing are integrated to reconstruct 3D point clouds and accurately invert crack depth. A high-frequency optical flow algorithm is used to extract the width change and dynamic propagation rate at the physical scale. Finally, dynamic and static parameters such as width, depth, and propagation rate are integrated into a comprehensive risk index to achieve scientific graded early warning. This scheme overcomes the limitations of traditional static, single-modal methods, achieving for the first time the synchronous quantification of the multi-dimensional "shape-dynamic-force" state of cracks, significantly improving the accuracy, real-time performance, and environmental adaptability of bridge health assessment.

[0021] like Figure 2 As shown in the embodiments of this application, the method for assessing the risk of cracks after a vehicle crosses a bridge based on multimodal vision includes: S201. Obtain the multimodal data sequence of the bridge monitoring area.

[0022] The multimodal data sequence consists of continuous data before the vehicle arrives, during the vehicle's passage, and after the vehicle has passed; the multimodal data includes at least visible light images, infrared thermal images, depth images, and structural vibration signals. The key monitoring areas of a bridge refer to the parts of the structure where stress is concentrated, damage is likely to occur, or cracks already exist, including but not limited to the bottom of the beam, near the supports, the mid-span area, and the location of previous defects.

[0023] In some implementations, multiple sensors are configured in the key monitoring areas of the bridge, including but not limited to visible light cameras, infrared thermal imagers, depth sensors, and vibration sensors. These sensors are synchronously acquired and controlled by an edge computing unit, so that when a vehicle passes by, the sensors can be triggered to work synchronously, ensuring that the collected multimodal data sequence has a time error accuracy of no more than 1 millisecond.

[0024] For complex structures (such as irregular buildings and curved tunnel surfaces), drones equipped with multi-sensor platforms are used for inspection, and path planning algorithms are used to collect data from multiple angles; or crawling robots are deployed to move and collect data in narrow and high-risk areas (such as inside pipelines and bridge box girders), thereby improving the flexibility and coverage of data collection.

[0025] Based on the above technical solutions, and addressing the core monitoring challenge of the transient dynamic response of bridge cracks under vehicle loads, an innovative multi-sensor collaborative acquisition system of "visible light + infrared + depth + vibration" was constructed. Through vehicle identification triggering synchronization mechanisms (such as geomagnetic coils or video event detection), high-precision time alignment (error ≤ 1ms) of the four types of heterogeneous sensors is achieved during key time periods (before vehicle arrival – after vehicle passage – after vehicle passage), ensuring that all modal data reflect the same physical state and avoiding distortion of dynamic parameters (such as propagation rate) due to time offset. An adaptive filtering algorithm is employed to specifically enhance or suppress vibration blur and illumination abrupt changes in visible light images, reduce thermal drift noise in infrared thermal images, repair holes in depth maps, and filter out non-structural frequency band interference in vibration signals. This ensures that the geometric, thermal, and dynamic characteristics of the cracks are fully preserved even in complex field environments (such as strong light, rain, fog, and nighttime). This system not only solves the risks of information fragmentation, feature distortion, and misjudgment caused by traditional single-modal or asynchronous acquisition, but also achieves a leap from "static snapshot" to "dynamic process holographic recording". It provides a high-quality, spatiotemporally consistent raw data foundation for subsequent high-precision segmentation, three-dimensional reconstruction, and risk assessment, and significantly improves the real-time performance, robustness, and engineering practicality of bridge health monitoring.

[0026] S202. Perform spatiotemporal alignment and enhancement processing on the multimodal data sequences to construct cross-modal feature maps.

[0027] S203. Input the cross-modal feature map into the improved U-Net++ lightweight segmentation network to identify crack regions and obtain the crack width.

[0028] S204. Based on multi-view stereo vision and depth sensor data, reconstruct a three-dimensional point cloud model of the crack; calculate the depth of the crack based on the three-dimensional point cloud model; and calculate the dynamic expansion rate and width change of the crack after the vehicle crosses the bridge based on the high-frequency image inter-frame optical flow method.

[0029] S205. Calculate the risk index of the crack based on the crack parameters; assess the risk level of the crack based on the risk index.

[0030] The crack parameters include crack width, depth, dynamic propagation rate, and width variation.

[0031] Based on the above technical solutions, traditional bridge crack monitoring relies heavily on manual inspections or single-modal static images, making it difficult to accurately acquire the three-dimensional geometric morphology of cracks and their dynamic response behavior under vehicle loads. This leads to delayed risk assessment and a high misjudgment rate. Especially in complex environments (such as changes in lighting and surface contamination), crack features are easily masked, and it is impossible to distinguish between structural damage and non-hazardous surface cracks. To address this, this solution proposes an integrated "perception-fusion-quantification-assessment" technical system: by simultaneously collecting visible light, infrared, depth, and vibration multimodal data throughout the entire vehicle passage process, a spatiotemporally aligned cross-modal feature map is constructed; a modified U-Net++ lightweight network is used to achieve highly robust crack segmentation, and a three-dimensional point cloud is reconstructed using stereo vision and depth sensing to accurately calculate depth; and a high-frequency optical flow method is used to capture transient displacements, obtaining the width change and dynamic expansion rate at the physical scale. This solution deeply integrates static geometric parameters and dynamic mechanical responses, achieving for the first time the synchronous quantification of the multidimensional "shape-dynamic-force" state of cracks. The constructed risk index model can accurately reflect the structural safety margin, significantly improve early warning capabilities and decision-making scientificity, and solve the fundamental shortcomings of existing methods in terms of accuracy, real-time performance and environmental adaptability, providing reliable technical support for intelligent bridge operation and maintenance.

[0032] In one possible implementation of the embodiments of this application, such as Figure 3 As shown, the above S202 can be specifically implemented through the following S301, S302, S303 and S304, which are explained in detail below: S301. Perform spatial registration and temporal synchronization on multimodal data; In some implementations, spatial registration is typically performed by calibrating extrinsic parameters using a calibration board, or by using feature point matching such as SIFT / SURF combined with robust RANSAC estimation to complete automatic registration.

[0033] It should be noted that images acquired from different sensors, such as visible light cameras, infrared thermal imagers, and depth cameras, are mapped to the same physical coordinate system through geometric transformations, such as affine transformations and homography matrices, to ensure that the same structural point is positioned consistently across different modal images. This avoids pixel misalignment caused by differences in viewing angle and installation location, providing a spatially consistent foundation for subsequent feature fusion.

[0034] Time synchronization ensures that all sensors trigger data acquisition synchronously at the critical moment when the vehicle crosses the bridge, and assign a unified, high-precision timestamp (such as PTP or NTP protocol, with an error ≤10 ms). This guarantees that visible light, infrared, depth, and vibration data reflect the same instantaneous state, avoiding distortion in dynamic analysis due to time shifts.

[0035] In addition, illumination invariance enhancement is performed on visible light images, thermal noise suppression is performed on infrared thermal images, hole repair is performed on depth images, and frequency band selective denoising is performed on vibration signals.

[0036] In some implementations, illumination invariance enhancement is achieved through methods such as Retinex decomposition, histogram equalization, or adaptive white balance, which weaken the impact of illumination variations on crack texture. This allows cracks to be stably identified under different lighting conditions, including strong light, shadow, or nighttime illumination.

[0037] Thermal noise suppression employs nonlocal mean denoising, wavelet threshold filtering, or temporal smoothing to suppress the inherent thermal drift and random noise of infrared sensors. It also preserves genuine thermal anomaly areas, such as internal frictional heating caused by cracks, to avoid misjudgments.

[0038] Hole repair: This involves filling in invalid pixels in a depth image due to occlusion, reflection, or low reflectivity using neighborhood interpolation, such as bilateral filtering or surface fitting-based repair. This ensures the integrity of the 3D reconstruction, especially in crack edge areas.

[0039] Frequency-selective denoising retains the dominant frequency band relevant to the bridge structure's dynamic response while filtering out high-frequency electronic noise or low-frequency temperature drift through bandpass filtering, wavelet packet decomposition, or empirical mode decomposition. Effective vibration components directly related to vehicle loads are extracted for subsequent dynamic extension correction.

[0040] S302. Input the processed modal data into the corresponding feature extraction module to obtain their respective multi-scale features.

[0041] It should be noted that each type of modality data is fed into a dedicated or shared convolutional neural network, such as a lightweight ResNet, MobileNet, or a custom CNN, to extract multi-scale feature maps from detailed textures to semantic abstractions at different network layers (shallow to deep).

[0042] Shallow features preserve fine structures such as crack edges and widths, while deep features capture the overall distribution and contextual relationships, providing rich representations for subsequent fusion.

[0043] For example, visible light is used to extract crack texture; infrared light is used to extract thermal anomaly areas; depth is used to extract geometric depressions; and vibration time-frequency maps are used to extract the concentrated energy frequency band.

[0044] S303. Calculate the feature attention weights of each modality data and perform weighted fusion of multi-scale features; Among them, attention weight : ; H( )=- ; Where i is the sequence number of the modal data. H( represents the weight of the i-th modality data) ) is the feature map of the i-th modality data. The information entropy, p(f) is the probability distribution of pixel f in the feature map.

[0045] S304. The fused features are restored in terms of spatial resolution and integrated with contextual information through an encoder-decoder structure to obtain a cross-modal feature map.

[0046] It should be noted that the encoder-decoder structure usually refers to a U-Net-like architecture, which includes downsampling (encoding) to extract high-level semantics, and then upsampling (decoding) to gradually restore spatial details; spatial resolution restoration: by transposing convolution or interpolation, low-resolution fused features are restored to the original input size; context information integration: during the decoding process, high-resolution shallow features (skip connections) from the encoding stage are fused to preserve key details such as crack edges; The cross-modal feature map integrates multi-dimensional information such as visible light texture, infrared thermal anomaly, depth geometry, and vibration response, serving as a unified input for subsequent crack segmentation and parameter extraction.

[0047] Based on the above technical solutions, in complex field bridge monitoring scenarios, single-modal data is highly susceptible to environmental interference (such as strong light, shadow, rain, fog, surface stains, thermal drift, etc.), leading to incomplete or even failed crack feature extraction. While multimodal sensors can provide complementary information, due to differences in viewing angle, installation offset, and asynchronous acquisition, the original data suffers from spatial misalignment and temporal mismatch. Direct fusion will introduce noise and artifacts, thus reducing detection reliability. To address this, this solution constructs a cross-modal spatiotemporal alignment framework: pixel-level spatial registration is achieved through calibration or feature matching, combined with high-precision time synchronization (≤10ms) to ensure that the four types of modal data are aligned under a unified spatiotemporal coordinate system; and targeted modal enhancements (such as illumination invariance processing, thermal noise suppression, deep hole repair, and vibration frequency band filtering) are performed to significantly improve the quality of each source data. Furthermore, an adaptive weighted feature fusion algorithm based on information entropy is proposed—the information entropy of the feature map is used to measure its orderliness and discriminative power, automatically assigning higher weights to high-value modalities (such as clear visible light textures or significant thermal anomalies) and suppressing low-quality or redundant information. This solution ensures data consistency from the source, enables intelligent selection during the fusion stage, effectively overcomes environmental interference, and significantly improves the robustness, boundary accuracy, and generalization ability of crack detection in complex scenarios, laying a solid foundation for subsequent high-precision segmentation and risk assessment.

[0048] In one possible implementation of this application embodiment, the above-mentioned S203 can be specifically described as follows: It should be noted that the original architecture of the U-Net++ lightweight segmentation network is as follows: Standard U-Net++ introduces a nested U-shaped structure with dense skip connections on top of U-Net. Its core modules include: Encoder: Composed of multiple downsampling blocks (usually 4-5 layers); each layer contains several convolutions + ReLU + downsampling (such as max pooling); used to extract high-level semantic features step by step and expand the receptive field.

[0049] Decoder: Composed of multiple upsampling blocks, symmetrical to the encoder; each layer recovers resolution through upsampling (such as transposed convolution or interpolation); receives multi-scale skip connection features from the same level and shallower levels of the encoder; used to fuse information from deep and shallow layers to accurately locate boundaries.

[0050] Skip connections: U-Net++ unique: not only connects the same level, but also connects all intermediate levels (forming dense connections); used to alleviate gradient vanishing and enhance detail preservation.

[0051] Output layer: typically a 1×1 convolution + sigmoid activation, outputting a probability map; subsequent binarization requires manually setting a threshold (e.g., 0.5).

[0052] The original U-Net++ lightweight segmentation network has a limited receptive field, making it difficult to capture wide gaps; skip connections directly splice the data without distinguishing the importance of features; fixed threshold segmentation results in poor adaptability; and the large number of parameters (approximately 8.7M in the original U-Net++) makes it difficult to deploy on edge devices.

[0053] Therefore, this application makes the following improvements to the U-Net++ lightweight segmentation network: A multi-scale dilated convolutional module is located at at least one level of the encoder; Structure: Dilated convolution kernels with dilation rates of 3, 5, and 7 are used in parallel, and the outputs are concatenated and then compressed by a 1×1 convolution. Functions: Significantly expands the receptive field (equivalent to 15×15 convolution) without increasing computational load or downsampling; captures multi-scale crack features from fine cracks (narrow) to structural cracks (wide); avoids loss of small crack information due to multiple downsampling.

[0054] Attention fusion modules are located in each upsampling layer of the decoder; Structure: Input: Current decoded features + multiple encoder skip connection features; first calculate channel attention and spatial attention separately; Attention weights are generated by a joint response of GAP (Global Average Pooling) and GMP (Global Max Pooling); the jump features are weighted and then fused with the decoded features.

[0055] Functions: Suppress interference from low-quality modalities or blurred regions, such as visible light features in rain and fog; enhance the response of key regions such as crack edges and weak textures; achieve intelligent selection and fusion of cross-level and cross-modal features.

[0056] The learnable threshold segmentation module is located before the output layer; Structure: Lightweight MLP or global pooling + fully connected layer, receiving the final feature map and outputting a scalar threshold τ∈(0.1,0.9); Function: Dynamically adjusts the segmentation threshold based on the contrast, noise level, and lighting conditions of the input image; replaces the traditional fixed threshold, such as 0.5, to avoid oversegmentation or undersegmentation; improves the robustness and accuracy of segmentation in complex scenes, such as at night, in strong light, and in rain and fog.

[0057] The crack binarization segmentation mask obtained based on the improved U-Net++ lightweight segmentation network is used to extract the center line of the crack and measure the pixel width along the normal direction perpendicular to the crack direction at each point. The average of several pixel widths is the width of the crack. The pixel width is converted into physical scale based on camera calibration parameters.

[0058] It should be noted that the centerline can be extracted using morphological skeleton algorithms, such as the Zhang-Suen algorithm.

[0059] Based on the above technical solutions, while traditional U-Net++ can achieve pixel-level segmentation in bridge crack detection, it still faces three major bottlenecks: limited receptive field leading to missed detection of wide cracks, simple splicing with skip connections introducing redundant noise, and fixed thresholds failing to adapt to complex lighting and weather changes, while the large model size makes it difficult to deploy on edge devices. To address these issues, this solution makes targeted improvements: a multi-scale dilated convolution module is introduced into the encoder to significantly expand the receptive field without downsampling, effectively capturing multi-scale features from microcracks to structural wide cracks; an attention fusion module is designed in the decoder, using a channel-spatial dual attention mechanism combined with GAP and GMP joint responses to intelligently weight high-value features and suppress interference from rain, fog, and shadows; a learnable threshold segmentation module is embedded to dynamically generate the optimal binarized threshold, significantly improving segmentation robustness. After lightweighting, the overall network's parameter count is reduced to approximately 2.3M, supporting real-time inference. Based on this high-precision mask, the centerline is extracted and the width is measured along the normal direction, then combined with camera calibration to convert it into physical dimensions, achieving a leap from "visible" to "accurately measurable". This technical solution balances accuracy, efficiency, and adaptability, solving the core challenge of reliably extracting crack geometric parameters in complex field environments.

[0060] In one possible implementation of this application embodiment, the above-mentioned S204 can be specifically implemented by the following S401, S402, S403 and S404, which are described in detail below: S401. Feature matching and pose estimation are performed using multi-view visible light images to generate an initial point cloud based on stereo vision; the depth map acquired by the depth sensor is converted into an auxiliary point cloud and registered and fused with the initial point cloud to obtain a three-dimensional point cloud model.

[0061] S402. Based on the three-dimensional point cloud model of the crack, a robust plane fitting algorithm is used to fit the reference plane of the structural surface from the undamaged area around the crack, and the maximum or average value of the vertical distance from the crack point cloud to the reference plane of the structural surface is calculated to obtain the depth of the crack.

[0062] It should be noted that the point cloud is generated by a high-precision laser scanner or depth camera, such as Intel RealSense L515, during bridge inspection. After registration and denoising, a local 3D point cloud containing the crack area is generated.

[0063] Undamaged area definition: First, obtain a binary mask of the crack through the S300 step; select the point cloud as the "undamaged area" in the annular area 20-50mm outside the mask (assuming that the area still maintains the original structural flatness); exclude obvious depressions, corrosion or other defects (which can be filtered by curvature threshold).

[0064] Robust plane fitting algorithm: refers to RANSAC (Random Sample Consensus) plane fitting; "robust" because undamaged areas may contain a small amount of noise or minor defects, and RANSAC can automatically remove outliers and fit the optimal plane using only inliers; The output is the equation of the reference plane: Where a, b, and c are the coefficients of the reference plane, which can be obtained by a robust plane fitting algorithm, and (x, y, z) are the three-dimensional coordinates of the crack. Depth calculation: for each crack point Calculate its perpendicular distance to the reference plane. Where j is the crack number; Maximum vertical distance: reflects the deepest point and is used to assess the risk of structural penetration; Average vertical distance: reflects the overall degree of degradation and is used for long-term trend analysis.

[0065] For example, if the point cloud shows a depression of 8.2 mm, then the depth is...

[0066] S403. The high-frequency image sequence acquired during the vehicle crossing the bridge is used to calculate the inter-frame pixel displacement using an optical flow algorithm, and the local normal direction is determined based on the crack direction; the local normal direction is a unit vector perpendicular to the tangent direction of each point on the crack centerline.

[0067] Among them, the acquisition frequency of high-frequency image sequences is ≥100fps to ensure the capture of transient response when a vehicle passes by (typical events last 0.5–2 seconds); covering the entire process of "before the vehicle arrives – when the vehicle passes – after the vehicle passes".

[0068] S404. Project the displacement vector onto the local normal direction to obtain the normal displacement and convert it into a physical scale; perform spatiotemporal aggregation and time-domain differentiation on the normal displacement to obtain the crack width change and dynamic propagation rate.

[0069] The method for converting to physical scale is as follows: Using camera calibration parameters (focal length g, pixel size) ) and shooting distance to calculate spatial resolution (mm / pixel): ; Where, normal displacement (physical) = normal displacement r.

[0070] Based on the above technical solutions, in bridge health monitoring, relying solely on static crack geometric parameters (such as width and length) is insufficient to accurately assess the true risk, as structural damage often manifests as deep development and dynamic expansion. Traditional methods lack the ability to quantify three-dimensional morphology and transient response with high precision. Therefore, this solution proposes integrating multi-view stereo vision and depth sensor data to construct a highly complete three-dimensional point cloud model. RANSAC robust fitting of the undamaged area's reference surface is used to accurately calculate crack depth, effectively overcoming corrosion and noise interference, and achieving millimeter-level indentation measurement. Simultaneously, a high-frequency image sequence of ≥100fps combined with an optical flow algorithm is used to capture the instantaneous deformation of cracks under vehicle loads. Displacement projection is performed based on the local normal direction extracted from the centerline, ensuring the physical accuracy of width change and dynamic expansion rate. This technical solution unifies static three-dimensional deformation and dynamic mechanical response under the same crack coordinate system, not only solving problems such as incomplete single-modal perception, measurement direction deviation, and significant environmental interference, but also achieving a leap from "apparent observation" to "mechanical state inversion." The obtained depth, width changes, and expansion rate can be directly input into the risk index model, significantly improving the scientific rigor, real-time performance, and engineering reliability of bridge crack safety assessment.

[0071] In one possible implementation of this application embodiment, the above-mentioned S205 can be specifically described as follows: Based on the dynamic parameters of the crack, through the formula The risk index R of the crack was calculated. in, , , , and is the weighting coefficient; D is the depth of the crack. This represents the critical threshold for depth. This represents the change in the width of the crack. This is the critical threshold for the width variation. The dynamic propagation rate of the crack. This is the critical threshold for the dynamic expansion rate. For the material's yield strength, This represents the critical threshold for the material's yield strength. For temperature gradient, This represents the critical threshold of the temperature gradient.

[0072] It should be noted that the temperature gradient is the temperature difference per unit length along the thickness direction in the region where the crack is located in the bridge structure.

[0073] The yield strength of a material is the ratio of the actual stress level at or around the crack tip to the material's yield strength. Mises stress is output at the crack location using ANSYS finite element simulation; typically, the material's yield strength is used (e.g., 235 MPa for Q235 steel).

[0074] Based on the value of the risk index R, it is divided into multiple preset risk level ranges. When R falls into different ranges, it corresponds to low risk, warning, or emergency status, respectively, and automatically triggers corresponding structural maintenance or traffic control strategies. More specifically: if the risk index is less than the preset risk threshold range, it is judged as low risk, and routine monitoring is performed; if the risk index is within the preset risk threshold range, it is judged as a warning status, and enhanced observation and recording of dynamic parameters are required; if the risk index is greater than the preset risk threshold range, it is judged as emergency risk, and the emergency response mechanism is immediately activated.

[0075] Based on the above technical solution, this solution constructs an intelligent risk assessment model of "routine monitoring + vehicle-triggered assessment" by organically integrating dynamic and static parameters. Specifically, the crack width change and dynamic propagation rate are calculated using optical flow algorithms based on high-frequency image sequences (e.g., above 100fps) collected when a vehicle passes, accurately capturing the transient response behavior of cracks under load and reflecting their short-term mechanical properties and potential sudden failure risks. Crack depth and material yield strength can be updated periodically (e.g., daily or weekly) to characterize the long-term deterioration trend and cumulative damage level of the structure, reflecting static safety margins. Simultaneously, temperature gradients are acquired in real time and used as environmental disturbance compensation, effectively eliminating the misjudgment of assessment results by non-structural factors such as thermal expansion and contraction. This multi-timescale, multi-physics-field collaborative assessment mechanism realizes a paradigm shift from traditional "static inspection" to "dynamic perception + intelligent decision-making," initiating high-precision dynamic analysis only at critical moments of vehicle passage, significantly improving monitoring efficiency and early warning sensitivity, and truly achieving a highly efficient, low-consumption, and highly reliable risk assessment closed loop of "assessment upon vehicle passage."

[0076] Some of the data in the above formula are calculated by removing dimensions and taking their numerical values. The formula is the closest to the real situation obtained by software simulation of a large amount of collected data. The preset parameters and preset thresholds in the formula are set by those skilled in the art according to the actual situation or obtained through simulation of a large amount of data.

Claims

1. A method for assessing the risk of cracks after a vehicle crosses a bridge based on multimodal vision, characterized in that, include: Acquire multimodal data sequences of key monitoring areas of the bridge; the multimodal data sequences are continuous data before the arrival of a vehicle, during the passage of a vehicle, and after the passage of a vehicle; the key monitoring areas are areas of concentrated structural stress or prone to damage. The multimodal data sequence is subjected to spatiotemporal alignment and enhancement processing to construct a cross-modal feature map; The cross-modal feature map is input into the improved U-Net++ lightweight segmentation network to generate a binary segmentation mask for cracks. The width of the crack is calculated based on the crack binarization segmentation mask; Based on multi-view stereo vision and depth sensor data, a three-dimensional point cloud model of the crack is reconstructed; based on the three-dimensional point cloud model, the depth of the crack is calculated; and based on the high-frequency image inter-frame optical flow method, the dynamic expansion rate and width change of the crack after the vehicle crosses the bridge are calculated. Based on the crack parameters, the risk index of the crack is calculated; the risk level of the crack is assessed according to the risk index; the crack parameters include the crack width, depth, dynamic propagation rate and width variation.

2. The method for assessing the risk of cracks after a vehicle crosses a bridge based on multimodal vision as described in claim 1, characterized in that, The calculation of the crack risk index includes: Based on crack parameters, through formula The risk index R of the crack was calculated. in, , , , and is the weighting coefficient; D is the depth of the crack. This represents the critical threshold for depth. This represents the change in the width of the crack. This is the critical threshold for the width variation. The dynamic propagation rate of the crack. This is the critical threshold for the dynamic expansion rate. For the material's yield strength, This represents the critical threshold for the material's yield strength. For temperature gradient, The temperature gradient is the critical threshold for the temperature gradient; the temperature gradient is the temperature difference per unit length along the thickness direction in the region where the crack is located in the bridge structure.

3. The method for assessing the risk of cracks after a vehicle crosses a bridge based on multimodal vision according to claim 1, characterized in that, The process of constructing the cross-modal feature map includes: Multimodal data includes at least visible light images, infrared thermal images, depth images, and structural vibration signals; Spatial registration and temporal synchronization are performed on the multimodal data; The system enhances the illumination invariance of visible light images, suppresses thermal noise in infrared thermal images, repairs holes in depth images, and performs frequency band selective denoising on vibration signals. The processed modal data are input into the corresponding feature extraction modules to obtain their respective multi-scale features. Calculate the feature attention weights for each modality, and perform weighted fusion of the multi-scale features; The fused features are then used to restore spatial resolution and integrate contextual information through an encoder-decoder structure to obtain a cross-modal feature map.

4. The method for assessing the risk of cracks after a vehicle crosses a bridge based on multimodal vision according to claim 3, characterized in that, The calculation of the weights of each modality data in the fusion process includes: ； H( )=- ； Where i is the sequence number of the modal data. H( represents the weight of the i-th modality data) ) is the feature map of the i-th modality data. The information entropy, p(f) is the probability distribution of pixel f in the feature map.

5. The method for assessing the risk of cracks after a vehicle crosses a bridge based on multimodal vision according to claim 1, characterized in that, The improvements to the U-Net++ lightweight segmentation network include: In the downsampling path of the encoder, at least one level introduces a multi-scale dilated convolution module, which uses dilated convolution kernels with different dilation rates in parallel. In the upsampling path of the decoder, an attention fusion module is set up. The attention fusion module performs channel attention and spatial attention calculations on the skip connection features from the corresponding level of the encoder and the current decoding features, respectively, and generates weighted fusion features based on the joint response of global average pooling and global max pooling. A learnable threshold segmentation module is embedded before the network output layer. The learnable threshold segmentation module dynamically outputs a binarized threshold based on the statistical distribution of the input features.

6. The method for assessing the risk of cracks after a vehicle crosses a bridge based on multimodal vision according to claim 1, characterized in that, The calculation of the crack width includes: The crack binarization segmentation mask obtained based on the improved U-Net++ lightweight segmentation network is used to extract the center line of the crack and measure the pixel width along the normal direction perpendicular to the crack direction at each point. The average of several pixel widths is the width of the crack. The pixel width is converted into physical scale based on camera calibration parameters.

7. The method for assessing the risk of cracks after a vehicle crosses a bridge based on multimodal vision according to claim 1, characterized in that, The reconstruction of the 3D point cloud model of the crack based on multi-view stereo vision and depth sensor data includes: Feature matching and pose estimation are performed using multi-view visible light images to generate an initial point cloud based on stereo vision; the depth map acquired by the depth sensor is converted into an auxiliary point cloud and registered and fused with the initial point cloud to obtain a three-dimensional point cloud model.

8. The method for assessing the risk of cracks after a vehicle crosses a bridge based on multimodal vision according to claim 1, characterized in that, The calculation of crack depth based on a 3D point cloud model includes: Based on the three-dimensional point cloud model of the crack, a robust plane fitting algorithm is used to fit the reference plane of the structural surface from the undamaged area around the crack. The maximum or average value of the vertical distance from the crack point cloud to the reference plane of the structural surface is calculated to obtain the depth of the crack.

9. The method for assessing the risk of cracks after a vehicle crosses a bridge based on multimodal vision according to claim 1, characterized in that, The calculation of the dynamic propagation rate and width change of the crack after the vehicle crosses the bridge includes: The inter-frame pixel displacement of the high-frequency image sequence acquired during the vehicle crossing the bridge is calculated using an optical flow algorithm, and the local normal direction is determined based on the crack orientation; the local normal direction is a unit vector perpendicular to the tangent direction at each point on the crack centerline; The displacement vector is projected onto the local normal direction to obtain the normal displacement, which is then converted into a physical scale. The normal displacement is spatiotemporally aggregated and differentiated in the time domain to obtain the crack width variation and dynamic propagation rate.

10. A multimodal vision-based system for assessing the risk of cracks after a vehicle crosses a bridge, operating based on the multimodal vision-based method for assessing the risk of cracks after a vehicle crosses a bridge as described in any one of claims 1-9, characterized in that... Includes an analysis module and a risk assessment module; The analysis module is used to acquire multimodal data sequences of key monitoring areas of the bridge. The multimodal data sequences are continuous data before the arrival of a vehicle, during the passage of a vehicle, and after the passage of a vehicle. The key monitoring areas are areas where the structure is under concentrated stress or is prone to damage. The multimodal data sequence is subjected to spatiotemporal alignment and enhancement processing to construct a cross-modal feature map; The cross-modal feature map is input into the improved U-Net++ lightweight segmentation network to generate a binary segmentation mask for cracks. Based on the binary segmentation mask, the width of the crack is calculated. Based on multi-view stereo vision and depth sensor data, a three-dimensional point cloud model of the crack is reconstructed; the depth of the crack is calculated based on the three-dimensional point cloud model; and the dynamic expansion rate and width change of the crack after the vehicle crosses the bridge are calculated based on the high-frequency image inter-frame optical flow method. The risk assessment module is used to calculate the risk index of the crack based on the crack parameters; and to assess the risk level of the crack according to the risk index; wherein the crack parameters include the crack width, depth, dynamic propagation rate and width change.

Images