A multimodal dam crack detection system and method using a dam patrol robot

Abstract

This invention belongs to the field of geographic surveying and water conservancy engineering safety technology, specifically a multimodal dam crack detection system and method for a dam patrol robot. The system includes: a data acquisition unit that simultaneously acquires visual images, sonar echoes, and robot motion state data; a physical field consistency constraint unit that constructs a geometric constraint loss function between the acoustic phase field and the visual gradient field to generate a cross-modal physical consistency map; a dynamic confidence propagation unit that calculates instantaneous confidence based on triaxial acceleration and vibration spectrum, and updates the confidence through a message passing mechanism of a spatiotemporal graph neural network; and a fusion judgment unit that weightedly fuses the consistency map and the confidence, inputs the result into a crack segmentation network, and outputs the detection result. This invention significantly improves the accuracy and robustness of crack detection through acoustic-optical physical consistency constraints and adaptive confidence propagation based on motion state.

Description

Technical Field

[0001] This invention relates to the fields of geographic surveying and water conservancy engineering safety technology, and in particular to a multimodal dam crack detection system and method using a dam patrol robot. Background Technology

[0002] As crucial water conservancy infrastructure, the structural safety of dams directly impacts the lives and property of people downstream and the stability of the ecological environment. Cracks are one of the most common defects in dams, and early and accurate detection is key to preventing major accidents such as dam failures. With the development of automation technology, using inspection robots equipped with sensors for dam inspection has become a mainstream trend.

[0003] Existing technologies for detecting cracks in dams have the following shortcomings: First, the reliance on a single data source leads to poor environmental adaptability and a high false detection rate. Existing methods, such as the invention patent application with publication number CN121120572A, primarily rely on single visual image analysis, distinguishing between real cracks and motion artifacts by identifying the geometric relationship between the motion blur direction and the linear feature direction in the image. However, this method has serious limitations in complex environments: when the dam surface has natural textures (such as vegetation roots, joints in retaining bricks, and water erosion marks) at a specific angle to the motion blur direction, the system is highly prone to misclassifying them as cracks; furthermore, visual information alone cannot penetrate surface coverings (such as moss, mud, and underwater suspended matter), resulting in severely insufficient detection capabilities for internal or hidden cracks.

[0004] Second, there is a lack of effective modeling of the coupling relationship between robot motion state and sensor data quality. Vibrations and bumps during robot movement can affect both visual image clarity and sonar echo signal-to-noise ratio, but existing technologies only use motion state as a criterion for image blurring and fail to use it as a key parameter for dynamically adjusting multimodal fusion weights. Summary of the Invention

[0005] This invention proposes a multimodal dam crack detection system and method using a dam patrol robot, aiming to solve the technical problems of poor environmental adaptability and high false detection rate caused by a single data source in the existing dam crack detection technology.

[0006] In a first aspect, the present invention provides a multimodal embankment crack detection system for a embankment patrol robot, comprising: A data acquisition unit, configured on the robot, is used to synchronously acquire multimodal raw data of the dam surface; the multimodal raw data includes visual image data, sonar echo data, and robot motion state data; The physical field consistency constraint unit is used to construct a geometric constraint loss function between the acoustic phase field and the visual gradient field based on the phase delay of the reflected wave caused by the crack interface in the sonar echo data and the gray-level gradient direction of the crack edge in the visual image data, and to generate a cross-modal physical consistency map that characterizes the spatial consistency between acoustic features and visual features. The dynamic confidence propagation unit is used to calculate the instantaneous confidence of the visual and acoustic modes based on the triaxial acceleration amplitude and vibration spectrum energy in the motion state data of the dike patrol robot, and update the confidence of the sensor data at different spatiotemporal locations based on the continuity of the crack in time and space through the message passing mechanism of the spatiotemporal graph neural network. The fusion determination unit is used to input the cross-modal physical consistency map and the updated confidence level into the crack segmentation network after weighted fusion, and output the crack detection result.

[0007] The technical advantages of the multimodal dam crack detection system for patrol robots disclosed in this invention are as follows: This invention achieves deep coupling of acoustic and optical features by introducing physical field consistency constraints and geometrically aligning acoustic phase distortion with visual edge gradients; it solves the technical challenges of weight allocation and spatiotemporal consistency in multimodal fusion by constructing a spatiotemporal graph neural network to achieve dynamic propagation of confidence; and it realizes dynamic perception of sensor data quality under complex working conditions through motion state adaptive confidence calculation. Compared with existing technologies, this invention significantly improves the accuracy and robustness of dam crack detection.

[0008] Furthermore, the physical field consistency constraint unit includes: The acoustic feature extraction module is used to perform short-time Fourier transform on the sonar echo data, extract the reflected wave envelope and phase delay caused by the crack interface, and generate an acoustic phase field. ; The visual feature extraction module is used to perform edge detection on the visual image data, extract the gray-level gradient direction and magnitude of crack edge points, and generate a visual gradient field. ; The geometric constraint loss calculation module is used to map the acoustic phase field to the image coordinate system and calculate the geometric constraint loss L between the spatial gradient of the acoustic phase field and the visual gradient field. ; Where Ω represents the image spatial domain, The grayscale gradient of the current detection frame visual image data. The scale normalization factor is determined using the following calibration method: During the calibration phase, multiple sets of synchronous acoustic phase fields from known crack-free regions were collected. and visual grayscale images Calculate the standard deviation (std) of the gradient fields respectively and take the ratio as follows: ; ; The consistency map generation module is used to mark pixels with geometric constraint loss below a preset threshold as acoustic-visual consistent regions, and pixels with geometric constraint loss above a preset threshold as acoustic-visual non-consistent regions, and to generate the cross-modal physical consistency map by using the reciprocal of the geometric constraint loss as the cross-modal consistency weight of each pixel.

[0009] Furthermore, the physical field consistency constraint unit also includes: The underwater parameter acquisition module is used to acquire water temperature, salinity, and depth parameters, and to calculate water sound speed and refractive index when the robot is in underwater operation mode. The photoacoustic path correction module is used to correct the propagation path of sound waves in water and the refraction path of light in water based on Snell's law and the sound speed and refractive index of the water body, respectively, and generate coordinate mapping correction values ​​for the acoustic phase field and the visual gradient field. The geometric constraint loss calculation module is also used to calculate the geometric constraint loss after performing coordinate registration of the acoustic phase field and the visual gradient field based on the coordinate mapping correction.

[0010] Furthermore, the dynamic confidence propagation unit includes: The instantaneous confidence calculation module is used to calculate the instantaneous confidence of the visual modality based on the exponential decay combination of the triaxial acceleration amplitude and the visual image signal-to-noise ratio, and to calculate the instantaneous confidence of the acoustic modality based on the logarithmic saturation combination of the triaxial acceleration amplitude and the sonar echo signal-to-noise ratio, and to normalize the calculation results to the 0 to 1 range using the Sigmoid function. The spatiotemporal graph construction module is used to construct a spatiotemporal graph neural network. The nodes of the spatiotemporal graph neural network represent sensor data units with different spatial coordinates and timestamps. The initial confidence of the nodes is the instantaneous confidence. The edge weights between nodes are calculated according to the spatial distance and time interval between nodes using a Gaussian function. The closer the spatial distance and the shorter the time interval, the greater the edge weight. The message passing module is used to propagate the instantaneous confidence of each node along the edge through multi-layer message passing of the spatiotemporal graph neural network. In each layer of message passing, the update amount of the node confidence is the weighted sum of the difference between the confidence of the node and the confidence of the adjacent nodes. After a preset number of iterations, the updated confidence of each node is output.

[0011] Furthermore, the dynamic confidence propagation unit also includes: The crack evolution association module is used to perform spatiotemporal association matching between the region identified as a crack in the current detection cycle and the crack region in the previous detection cycle. When the spatial overlap of the two regions is greater than the preset overlap threshold, it is determined to be the continuous evolution of the same crack. The confidence time-series propagation module is used to propagate the confidence of the crack region in the previous detection period along the time axis to the current detection period based on the matching results of the crack evolution association module, and to correct the confidence of the crack region in the current detection period. The corrected confidence is the larger value of the confidence of the current detection period and the confidence of the previous detection period after time decay.

[0012] Furthermore, the fusion determination unit includes: The weighted fusion module is used to use the cross-modal physical consistency map as a spatial weight matrix and perform pixel-by-pixel weighted fusion with the visual modal feature map and the acoustic modal feature map to generate a fused feature map, wherein the acoustic modal feature map is multiplied by the weight of the corresponding position of the cross-modal physical consistency map during fusion; The crack segmentation module is used to input the fused feature map into a crack segmentation network based on an encoder-decoder structure and output a crack probability map for each pixel. The post-processing module is used to perform morphological closing operations and connected component analysis on the crack probability map, remove isolated noise regions with an area smaller than a preset threshold, and output the final crack detection result.

[0013] Furthermore, the data acquisition unit also includes: The vibration monitoring module is used to calculate the total vibration energy based on the vibration spectrum in the robot's motion state data. The acquisition mode switching module is used to acquire visual image data at a first sampling frequency when the total vibration energy is lower than a first threshold; and to reduce the acquisition frequency of visual image data to a third sampling frequency when the total vibration energy is higher than a second threshold, and to activate image sonar to acquire sonar echo data at a fourth sampling frequency, wherein the fourth sampling frequency is positively correlated with the total vibration energy.

[0014] Furthermore, the system also includes: The elastic wave excitation unit is used to transmit pulsed elastic waves to the surface of the dam and collect the reflected echoes of the elastic waves in the dam medium. The depth feature extraction module is used to calculate the position and opening width of the crack in the depth direction based on the arrival time difference and amplitude attenuation of the reflected echo, and generate a crack depth feature vector. The fusion determination unit is also used to associate the crack depth feature vector with the crack detection result and output the three-dimensional geometric parameters of the crack.

[0015] Furthermore, the system also includes: The crack matching module is used to match the center point coordinates, direction angle and cross-modal physical consistency map of the currently detected crack with the records in the historical crack database. When the center point distance is less than the first preset distance and the direction angle difference is less than the first preset angle, it is determined to be the same crack. The Kalman filter prediction module is used to input the historical length and width sequences of successfully matched cracks into the Kalman filter, predict the crack length and width of the current detection cycle, and calculate the residual between the predicted value and the detected value. The confidence correction module is used to correct the confidence level output by the dynamic confidence propagation unit based on the residual. When the residual is less than a preset residual threshold, the confidence level is increased; otherwise, the confidence level is decreased.

[0016] Secondly, the present invention provides a multimodal dam crack detection method based on the aforementioned system using a dam patrol robot, comprising the following steps: Step S1: Simultaneously collect visual image data, sonar echo data, and robot motion state data of the dam surface; Step S2: Based on the phase delay of the reflected wave caused by the crack interface in the sonar echo data and the gray-level gradient direction of the crack edge in the visual image data, calculate the geometric constraint loss of the acoustic phase field and the visual gradient field to generate a cross-modal physical consistency map. Step S3: Calculate the instantaneous confidence of the visual and acoustic modes based on the triaxial acceleration amplitude and vibration spectrum energy in the robot motion state data, and update the confidence of the sensor data at different spatiotemporal locations through the message passing mechanism of the spatiotemporal graph neural network. Step S4: The cross-modal physical consistency map and the updated confidence level are weighted and fused, and then input into the crack segmentation network to output the crack detection result; Step S5: Match the crack detection results with the historical crack database, predict the crack development trend through Kalman filtering, and correct the confidence level based on the prediction residual. Attached Figure Description

[0017] Figure 1 This is an overall structural block diagram of a multimodal dam crack detection system using a dam patrol robot, as proposed in an embodiment of the present invention. Figure 2 This is a flowchart illustrating the process of the physical field consistency constraint unit of the present invention. Figure 3 This is a schematic diagram of the spatiotemporal graph neural network in the dynamic confidence propagation unit of the present invention; Figure 4 This is a flowchart of the fusion determination unit of the present invention; Figure 5This is a flowchart illustrating the steps of the multimodal dam crack detection method using a dam patrol robot according to the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The method for detecting dam cracks mentioned in the background technology has serious limitations in complex environments: when there are natural textures on the dam surface that are at a specific angle to the direction of motion ambiguity (such as vegetation roots, joints of slope protection bricks, and water erosion marks), the system is very likely to misjudge them as cracks.

[0020] This invention provides a multimodal embankment crack detection system using a embankment patrol robot, with reference to... Figure 1 As shown, it specifically includes: a data acquisition unit 100, a physical field consistency constraint unit 200, a dynamic confidence propagation unit 300, and a fusion determination unit.

[0021] The data acquisition unit 100 is configured on the dike patrol robot to synchronously collect multimodal raw data from the dike surface. The multimodal raw data includes at least visual image data, sonar echo data, and robot motion state data. Specifically, the visual image data is acquired using a high-resolution industrial camera, the sonar echo data is acquired using image sonar, and the robot motion state data is acquired using a six-axis inertial measurement unit.

[0022] In practical applications, the data acquisition unit 100 also includes a vibration monitoring module and an acquisition mode switching module. The vibration monitoring module is used to calculate the total vibration energy based on the vibration spectrum in the robot's motion state data. The calculation formula is as follows: ; Where S(f) is the vibration power spectral density, and f1 and f2 are preset frequency integration upper and lower limits. The acquisition mode switching module is used to acquire visual image data at a first sampling frequency when the total vibration energy is below a first threshold; when the total vibration energy is above a second threshold, it reduces the acquisition frequency of the visual image data to a third sampling frequency and activates the image sonar to acquire sonar echo data at a fourth sampling frequency, wherein the fourth sampling frequency is positively correlated with the total vibration energy. This adaptive acquisition mode can effectively suppress the influence of motion blur on image quality, while ensuring sufficient acquisition of acoustic data when vibration is severe.

[0023] The physical field consistency constraint unit 200 is used to construct a geometric constraint loss function between the acoustic phase field and the visual gradient field based on the phase delay of the reflected wave caused by the crack interface in the sonar echo data and the gray-level gradient direction of the crack edge in the visual image data, and to generate a cross-modal physical consistency map that characterizes the spatial consistency between acoustic features and visual features.

[0024] like Figure 2 As shown, the physical field consistency constraint unit 200 includes an acoustic feature extraction module 201, a visual feature extraction module 202, a geometric constraint loss calculation module 203, and a consistency map generation module 204.

[0025] The acoustic feature extraction module 201 is used to perform short-time Fourier transform on the sonar echo data, extract the reflected wave envelope and phase delay caused by the crack interface, and generate an acoustic phase field. As a type of medium discontinuity, the existence of a crack can cause changes in the propagation path of sound waves and abrupt changes in phase. This phase delay exhibits a spatial distribution pattern related to the crack orientation.

[0026] The visual feature extraction module 202 is used to perform edge detection on the visual image data, extract the gray-level gradient direction and amplitude of the crack edge points, and generate a visual gradient field. The Canny edge detection operator is used to extract edge information from the image, obtaining the gradient vector, i.e., the gray-level gradient, of each pixel. .

[0027] The geometric constraint loss calculation module 203 is used to map the acoustic phase field to the image coordinate system and calculate the geometric constraint loss between the spatial gradient of the acoustic phase field and the visual gradient field. The geometric constraint loss is the sum of squares of the differences between the spatial gradient of the acoustic phase field and the visual gradient field, and its calculation formula is as follows: ; Where Ω represents the image spatial domain, The grayscale gradient of the current detection frame visual image data. The scale normalization factor is determined using the following calibration method: During the calibration phase, multiple sets of synchronous acoustic phase fields from known crack-free regions were collected. and visual grayscale images Calculate the standard deviation (std) of the gradient fields respectively and take the ratio as follows: ; .

[0028] The core idea of ​​this loss function is that a crack, as a single physical entity, manifests as an abnormal phase gradient in the acoustic phase field and an abnormal grayscale gradient in the visual image; these two should be spatially consistent. A smaller geometric constraint loss indicates that the acoustic and visual features are spatially aligned, suggesting a higher probability of a real crack at that location. Conversely, a larger geometric constraint loss indicates a conflict between the acoustic and visual features, potentially indicating artifacts or noise.

[0029] The consistency map generation module 204 is used to mark pixels with geometric constraint loss below a preset threshold as acoustic-visual consistent regions, and to generate the cross-modal physical consistency map by using the reciprocal of the geometric constraint loss as the cross-modal consistency weight of each pixel. The value of this spectrum ranges from 0 to 1, with a larger value indicating a stronger consistency between acoustic and visual features at that location.

[0030] When the robot is in underwater operation mode, the physical field consistency constraint unit 200 also includes an underwater parameter acquisition module and an optical-acoustic path correction module. The underwater parameter acquisition module acquires water temperature, salinity, and depth parameters, and calculates the water sound velocity and refractive index. The optical-acoustic path correction module, based on Snell's law and the aforementioned water sound velocity and refractive index, corrects the propagation path of sound waves and the refraction path of light rays in water, respectively, generating coordinate mapping corrections for the acoustic phase field and the visual gradient field. The geometric constraint loss calculation module calculates the geometric constraint loss after performing coordinate registration of the acoustic phase field and the visual gradient field based on the coordinate mapping corrections. This design enables the invention to adapt to underwater detection scenarios and solves the spatial mismatch problem caused by underwater light refraction and the bending of the sound wave propagation path.

[0031] The dynamic confidence propagation unit 300 is used to calculate the instantaneous confidence of the visual and acoustic modes based on the triaxial acceleration amplitude and vibration spectrum energy in the robot motion state data, and update the confidence of the sensing data at different spatiotemporal locations based on the continuity of the crack in time and space through the message passing mechanism of the spatiotemporal graph neural network.

[0032] like Figure 3 As shown, the dynamic confidence propagation unit 300 includes an instantaneous confidence calculation module 301, a spatiotemporal graph construction module 302, and a message passing module 303.

[0033] The instantaneous confidence calculation module 301 is used to calculate the instantaneous confidence of the visual modality based on the exponentially decaying combination of the triaxial acceleration amplitude and the visual image signal-to-noise ratio. The instantaneous confidence level of the acoustic modes is calculated based on the logarithmic saturation combination of the triaxial acceleration amplitude and the sonar echo signal-to-noise ratio. The calculation results are then normalized to the 0 to 1 range using the Sigmoid function.

[0034] Exponential decay portfolio : ; in, The magnitudes of the three-axis accelerations. The preset attenuation coefficient (values ​​range from 0.1 to 0.5) The signal-to-noise ratio (SNR) of the visual image is given in dB. The final confidence score is normalized using the Sigmoid function to obtain the instantaneous confidence score of the visual modality. .

[0035] Logarithmic saturated combination: ; in, is the total vibration energy, and is the preset attenuation coefficient (value range 0.01~0.1). Sonar echo signal-to-noise ratio (in dB). Instantaneous confidence of acoustic modes normalized by Sigmoid. : ; The larger the triaxial acceleration amplitude A, the lower the visual confidence index; while the larger the vibration energy V, the acoustic confidence index only decreases logarithmically, reflecting the relative advantage of acoustic modes under vibration conditions.

[0036] Among them, the instantaneous confidence of the visual modality The calculation reflects the principle that the greater the acceleration (the more violent the robot vibration), the lower the confidence level, while also considering the signal-to-noise ratio of the image itself. Acoustic modal instantaneous confidence level. The calculation reflects the relative advantage of acoustic modes when the acceleration is greater, because acoustic sensors are not sensitive to vibration. After normalization, the two confidence levels form a quantitative assessment of the reliability of each modal data at the current moment.

[0037] The spatiotemporal graph construction module 302 is used to construct a spatiotemporal graph neural network. The nodes of this neural network represent sensor data units with different spatial coordinates and timestamps. The initial confidence level of each node is the instantaneous confidence level. The edge weights between nodes are calculated using a Gaussian function based on the spatial distance and time interval between nodes; the closer the spatial distance and the shorter the time interval, the larger the edge weight. Specifically, for nodes i and j, their edge weights... The design satisfies the following condition: the closer the spatial distance and the closer the timestamps of two nodes, the closer the edge weight is to 1; as the spatial distance or time interval increases, the edge weight decays exponentially. This design reflects the physical prior of "spatiotemporal continuity" in crack detection: the detection results of the same crack at adjacent spatial locations and adjacent time points should be similar.

[0038] The message passing module 303 is used to propagate the instantaneous confidence of each node along the edges through multi-layer message passing in the spatiotemporal graph neural network. In each layer of message passing, the update amount of the node's confidence is the weighted sum of the differences between the node's confidence and that of its neighboring nodes, where the weights are the weights of the corresponding edges. After a preset number of iterations, the updated confidence of each node is output. The core of the message passing mechanism is that information from high-confidence nodes is propagated to low-confidence nodes through high-weighted edges, thereby achieving collaborative optimization of global confidence. After multiple iterations, the confidence distribution tends to be smoother. If an isolated low-confidence region is surrounded by a high-confidence region, its confidence will be improved; conversely, if an isolated high-confidence region is surrounded by a low-confidence region, its confidence will be suppressed.

[0039] Among them, the message passing module achieves collaborative optimization of confidence through multi-layer message passing. Let the first... Layer nodes The confidence level is Its neighbor node set is N(i), and the edge weight is... Calculated using a Gaussian function based on spatial distance and time interval: ; in For spatial Euclidean distance, For the time difference, This is the preset bandwidth parameter.

[0040] The message passing layer performs the following updates: ; in With a preset propagation step size (ranging from 0.1 to 0.3), the Clip function truncates the confidence score to the [0,1] interval. After L=3 iterations, the updated confidence score of each node is output.

[0041] The dynamic confidence propagation unit 300 also includes a crack evolution association module and a confidence temporal propagation module. The crack evolution association module performs spatiotemporal correlation matching between regions identified as cracks in the current detection cycle and crack regions in the previous detection cycle. When the spatial overlap between the two regions exceeds a preset overlap threshold, they are determined to be continuous evolutions of the same crack. The confidence temporal propagation module, based on the matching results, propagates the confidence of crack regions from the previous detection cycle along the time axis to the current detection cycle, correcting the confidence of crack regions in the current detection cycle. The corrected confidence is the larger value of the current detection cycle's confidence and the previous detection cycle's confidence after time decay. This temporal propagation mechanism gives the detection system a "memory" capability, avoiding missed or false detections caused by fluctuations in the quality of a single detection.

[0042] The fusion determination unit 400 is used to input the cross-modal physical consistency map and the updated confidence level into the crack segmentation network after weighted fusion, and output the crack detection result.

[0043] like Figure 4 As shown, the fusion determination unit 400 includes a weighted fusion module 401, a crack segmentation module 402, and a post-processing module 403.

[0044] The weighted fusion module 401 is used to perform pixel-by-pixel weighted fusion of the cross-modal physical consistency map as a spatial weight matrix with the visual modal feature map and the acoustic modal feature map to generate a fused feature map. The acoustic modal feature map is multiplied by the weights of the corresponding positions in the cross-modal physical consistency map during fusion. Specifically, the calculation formula for the fused feature map is: in, For visual feature maps, Acoustic feature map, This is the cross-modal physical consistency map. and The confidence scores for the visual and acoustic modalities are output separately by the dynamic confidence propagation unit. The core of this fusion rule is that the contributions of visual and acoustic features are determined by their confidence scores, while acoustic features are additionally multiplied by a physical consistency weight—when acoustic and visual features are spatially consistent, their contribution is enhanced; when they are inconsistent, their contribution is suppressed. This design effectively solves the "who to trust" problem in multimodal fusion.

[0045] The crack segmentation module 402 is used to input the fused feature map into a crack segmentation network based on an encoder-decoder structure, and output a crack probability map for each pixel. The crack segmentation network adopts a semantic segmentation network with an encoder-decoder structure. The encoder gradually extracts high-level semantic features through multiple convolutional layers and downsampling layers, and the decoder gradually restores the resolution through deconvolutional layers and upsampling layers, finally outputting a crack probability map with the same size as the input image. The value of each pixel is between 0 and 1, representing the probability that the pixel belongs to a crack.

[0046] In one specific embodiment, the crack segmentation network based on the encoder-decoder structure adopts the U-Net architecture, with the following specific configuration: The encoder consists of four downsampling stages, each containing two 3×3 convolutional layers (stride 1, padding 1). Each convolutional layer is followed by batch normalization and ReLU activation, followed by 2×2 max pooling (stride 2). The number of channels is as follows: 64 → 128 → 256 → 512.

[0047] Decoder: 4 upsampling stages, each containing a 2×2 transposed convolution (stride 2) to double the feature map size, then skip-connected to the feature map of the corresponding layer in the encoder, followed by two 3×3 convolutional layers (stride 1), each convolutional layer followed by batch normalization and ReLU. The number of channels is as follows: 512 → 256 → 128 → 64.

[0048] Output layer: A 1×1 convolutional layer that maps 64 channels to 1 channel, and then outputs a crack probability map through a Sigmoid activation function. The value of each pixel is the probability (0~1) that the point belongs to a crack.

[0049] The post-processing module 403 performs morphological closing operations and connected component analysis on the crack probability map, removes isolated noise regions with an area smaller than a preset threshold, and outputs the final crack detection result. Morphological closing operations can fill small voids inside cracks, connected component analysis can identify continuous crack regions, and area threshold filtering can remove isolated false detection points caused by noise.

[0050] An elastic wave excitation unit, optionally mounted on the dike patrol robot, is used to emit pulsed elastic waves onto the dike surface and collect the reflected echoes of the elastic waves in the dike medium. A depth feature extraction module is used to calculate the location and opening width of the crack in the depth direction based on the arrival time difference and amplitude attenuation of the reflected echoes, generating a crack depth feature vector. A fusion determination unit is further used to correlate the crack depth feature vector with the crack detection results, outputting the three-dimensional geometric parameters of the crack, including its length, width, depth, and orientation.

[0051] The crack matching module and the Kalman filter prediction module are used to achieve long-term crack tracking and trend prediction. The crack matching module matches the center point coordinates, strike angle, and cross-modal physical consistency map of the currently detected crack with records in the historical crack database. When the center point distance is less than a first preset distance and the strike angle difference is less than a first preset angle, they are determined to be the same crack. The Kalman filter prediction module inputs the historical length and width sequences of successfully matched cracks into the Kalman filter to predict the crack length and width for the current detection period and calculates the residual between the predicted and detected values. The confidence correction module corrects the confidence level output by the dynamic confidence propagation unit based on the residual. When the residual is less than a preset residual threshold, the confidence level is increased; otherwise, it is decreased. This design enables the detection system to have self-verification capabilities: when the detection results conform to the historical evolution trend, the confidence level increases; when the detection results deviate significantly from the trend, the confidence level decreases, effectively preventing drastic jumps in the detection results.

[0052] Based on the same inventive concept, this embodiment provides a multimodal dam crack detection method using a dam patrol robot, such as... Figure 5As shown, it includes the following steps: Step S1: Synchronously acquire visual image data, sonar echo data, and robot motion status data from the dam surface. Visual image data is acquired using a high-resolution industrial camera, sonar echo data is acquired using image sonar, and robot motion status data is acquired using a six-axis inertial measurement unit. Synchronous acquisition is achieved through hardware trigger signals to ensure that the data from all sensors have a unified time reference.

[0053] Step S2: Based on the phase delay of the reflected wave caused by the crack interface in the sonar echo data and the gray-level gradient direction of the crack edge in the visual image data, calculate the geometric constraint loss between the acoustic phase field and the visual gradient field to generate a cross-modal physical consistency map. Specifically, first, perform a short-time Fourier transform on the sonar echo data to extract the phase delay and generate an acoustic phase field; then, perform edge detection on the visual image data to generate a visual gradient field; next, map the acoustic phase field to the image coordinate system and calculate the sum of squared differences between the acoustic phase spatial gradient and the visual gradient field as the geometric constraint loss; finally, use the reciprocal of the geometric constraint loss as the pixel weight to generate a cross-modal physical consistency map.

[0054] Step S3: Calculate the instantaneous confidence scores of the visual and acoustic modes based on the triaxial acceleration amplitude and vibration spectrum energy in the robot's motion state data, and update the confidence scores of sensor data at different spatiotemporal locations through the message passing mechanism of the spatiotemporal graph neural network. Specifically, calculate the instantaneous confidence score of the visual mode based on the exponential decay combination of the triaxial acceleration amplitude and the visual image signal-to-noise ratio, and calculate the instantaneous confidence score of the acoustic mode based on the logarithmic saturation combination of the triaxial acceleration amplitude and the sonar echo signal-to-noise ratio; construct a spatiotemporal graph neural network, where nodes represent sensor data units at different spatiotemporal locations, the initial confidence score of each node is the instantaneous confidence score, and the edge weights between nodes are calculated using a Gaussian function based on spatial distance and time interval; propagate and update the confidence scores of each node along the edges through multi-layer message passing.

[0055] Step S4: The cross-modal physical consistency map and the updated confidence level are weighted and fused, and then input into the crack segmentation network to output the crack detection result. Specifically, the cross-modal physical consistency map is used as a spatial weight matrix and weighted and fused pixel-by-pixel with the visual feature map and the acoustic feature map to generate a fused feature map; the fused feature map is input into the crack segmentation network based on the encoder-decoder structure to output a crack probability map; morphological closing operations and connected component analysis are performed on the crack probability map to remove isolated noise points, and the crack detection result is output.

[0056] Step S5: Match the crack detection results with the historical crack database, predict the crack development trend using Kalman filtering, and adjust the confidence level based on the prediction residual. Specifically, perform spatiotemporal matching between the currently detected crack and the historical records. After a successful match, input the historical length and width sequences into the Kalman filter to predict the current value, calculate the residual between the predicted value and the detected value, and adjust the confidence level based on the magnitude of the residual.

[0057] Beneficial effects: This invention introduces a physical field consistency constraint unit to geometrically align acoustic phase distortion with visual edge gradient, establishing an acoustic-optical physical consistency model for cracks. This effectively solves the problem of poor detection accuracy from a single data source. Specifically, the acoustic sensor can detect hidden cracks under surface coverings, while the visual sensor can accurately depict the two-dimensional geometry of the cracks. The two sensors achieve spatial alignment through a geometric constraint loss function, enabling the system to maintain high detection accuracy even in harsh environments such as vegetation cover, mud obstruction, and underwater turbidity.

[0058] This invention constructs a dynamic confidence propagation unit to establish a quantitative correlation between the robot's motion state and sensor data quality, and achieves collaborative propagation of confidence through a spatiotemporal graph neural network, thus solving the technical challenges of weight allocation and spatiotemporal consistency in multimodal fusion. Specifically, when the robot vibrates violently, the confidence of the visual modality automatically decreases, while the confidence of the acoustic modality relatively increases. When high-confidence regions and low-confidence regions are spatiotemporally adjacent, a message passing mechanism is used to achieve smooth propagation of confidence, effectively suppressing interference from isolated noise points.

[0059] This invention, by introducing a crack matching and Kalman filter prediction mechanism, achieves long-term tracking and self-verification of crack evolution, solving the problems of detection result jumps and difficulties in trend prediction. Specifically, by comparing the current detection results with historical trends, the confidence level is increased when the detected value matches the predicted value, and decreased when abnormal jumps occur, enabling the detection system to have self-correcting capabilities.

[0060] In summary, the multimodal dam crack detection system and method for patrol robots provided by this invention significantly improves the accuracy, robustness, and intelligence level of dam crack detection through technological innovations such as acoustic-optical physical consistency constraints, adaptive confidence propagation of motion state, and spatiotemporal graph neural network collaborative optimization, and has important engineering application value.

[0061] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A multimodal dam crack detection system using a dam patrol robot, characterized in that, include: A data acquisition unit, configured on the robot, is used to synchronously acquire multimodal raw data of the dam surface; the multimodal raw data includes visual image data, sonar echo data, and robot motion state data; The physical field consistency constraint unit is used to construct a geometric constraint loss function between the acoustic phase field and the visual gradient field based on the phase delay of the reflected wave caused by the crack interface in the sonar echo data and the gray-level gradient direction of the crack edge in the visual image data, and to generate a cross-modal physical consistency map that characterizes the spatial consistency between acoustic features and visual features. The dynamic confidence propagation unit is used to calculate the instantaneous confidence of the visual and acoustic modes based on the triaxial acceleration amplitude and vibration spectrum energy in the motion state data of the dike patrol robot, and update the confidence of the sensor data at different spatiotemporal locations based on the continuity of the crack in time and space through the message passing mechanism of the spatiotemporal graph neural network. The fusion determination unit is used to input the cross-modal physical consistency map and the updated confidence level into the crack segmentation network after weighted fusion, and output the crack detection result.

2. The system according to claim 1, characterized in that, The physical field consistency constraint unit includes: The acoustic feature extraction module is used to perform short-time Fourier transform on the sonar echo data, extract the reflected wave envelope and phase delay caused by the crack interface, and generate an acoustic phase field. ; The visual feature extraction module is used to perform edge detection on the visual image data, extract the gray-level gradient direction and magnitude of crack edge points, and generate a visual gradient field. ; The geometric constraint loss calculation module is used to map the acoustic phase field to the image coordinate system and calculate the geometric constraint loss L between the spatial gradient of the acoustic phase field and the visual gradient field. ； Where Ω represents the image spatial domain, The grayscale gradient of the current detection frame visual image data. The scale normalization factor is determined using the following calibration method: During the calibration phase, multiple sets of synchronous acoustic phase fields from known crack-free regions were collected. and visual grayscale images Calculate the standard deviation (std) of the gradient fields respectively and take the ratio as follows: ； ； Furthermore, it was found during the research process that, It can also be used as a trainable parameter, automatically learned through backpropagation during the training of the crack segmentation network. The consistency map generation module is used to mark pixels with geometric constraint loss below a preset threshold as acoustic-visual consistent regions, and pixels with geometric constraint loss above a preset threshold as acoustic-visual non-consistent regions, and to generate the cross-modal physical consistency map by using the reciprocal of the geometric constraint loss as the cross-modal consistency weight of each pixel.

3. The system according to claim 2, characterized in that, The physical field consistency constraint unit also includes: The underwater parameter acquisition module is used to acquire water temperature, salinity, and depth parameters, and to calculate water sound speed and refractive index when the robot is in underwater operation mode. The photoacoustic path correction module is used to correct the propagation path of sound waves in water and the refraction path of light in water based on Snell's law and the sound speed and refractive index of the water body, respectively, and generate coordinate mapping correction values ​​for the acoustic phase field and the visual gradient field. The geometric constraint loss calculation module is also used to calculate the geometric constraint loss after performing coordinate registration of the acoustic phase field and the visual gradient field based on the coordinate mapping correction.

4. The system according to claim 1, characterized in that, The dynamic confidence propagation unit includes: The instantaneous confidence calculation module is used to calculate the instantaneous confidence of the visual modality based on the exponential decay combination of the triaxial acceleration amplitude and the visual image signal-to-noise ratio, and to calculate the instantaneous confidence of the acoustic modality based on the logarithmic saturation combination of the triaxial acceleration amplitude and the sonar echo signal-to-noise ratio, and to normalize the calculation results to the 0 to 1 range using the Sigmoid function. The spatiotemporal graph construction module is used to construct a spatiotemporal graph neural network. The nodes of the spatiotemporal graph neural network represent sensor data units with different spatial coordinates and timestamps. The initial confidence of the nodes is the instantaneous confidence. The edge weights between nodes are calculated according to the spatial distance and time interval between nodes using a Gaussian function. The closer the spatial distance and the shorter the time interval, the greater the edge weight. The message passing module is used to propagate the instantaneous confidence of each node along the edge through multi-layer message passing of the spatiotemporal graph neural network. In each layer of message passing, the update amount of the node confidence is the weighted sum of the difference between the confidence of the node and the confidence of the adjacent nodes. After a preset number of iterations, the updated confidence of each node is output.

5. The system according to claim 4, characterized in that, The dynamic confidence propagation unit further includes: The crack evolution association module is used to perform spatiotemporal association matching between the region identified as a crack in the current detection cycle and the crack region in the previous detection cycle. When the spatial overlap of the two regions is greater than the preset overlap threshold, it is determined to be the continuous evolution of the same crack. The confidence time-series propagation module is used to propagate the confidence of the crack region in the previous detection period along the time axis to the current detection period based on the matching results of the crack evolution association module, and to correct the confidence of the crack region in the current detection period. The corrected confidence is the larger value of the confidence of the current detection period and the confidence of the previous detection period after time decay.

6. The system according to claim 1, characterized in that, The fusion determination unit includes: The weighted fusion module is used to use the cross-modal physical consistency map as a spatial weight matrix and perform pixel-by-pixel weighted fusion with the visual modal feature map and the acoustic modal feature map to generate a fused feature map, wherein the acoustic modal feature map is multiplied by the weight of the corresponding position of the cross-modal physical consistency map during fusion; The crack segmentation module is used to input the fused feature map into a crack segmentation network based on an encoder-decoder structure and output a crack probability map for each pixel. The post-processing module is used to perform morphological closing operations and connected component analysis on the crack probability map, remove isolated noise regions with an area smaller than a preset threshold, and output the final crack detection result.

7. The system according to claim 1, characterized in that, The data acquisition unit also includes: The vibration monitoring module is used to calculate the total vibration energy based on the vibration spectrum in the robot's motion state data. The acquisition mode switching module is used to acquire visual image data at a first sampling frequency when the total vibration energy is lower than a first threshold; and to reduce the acquisition frequency of visual image data to a third sampling frequency when the total vibration energy is higher than a second threshold, and to activate image sonar to acquire sonar echo data at a fourth sampling frequency, wherein the fourth sampling frequency is positively correlated with the total vibration energy.

8. The system according to claim 1, characterized in that, The system also includes: The elastic wave excitation unit is used to transmit pulsed elastic waves to the surface of the dam and collect the reflected echoes of the elastic waves in the dam medium. The depth feature extraction module is used to calculate the position and opening width of the crack in the depth direction based on the arrival time difference and amplitude attenuation of the reflected echo, and generate a crack depth feature vector. The fusion determination unit is also used to associate the crack depth feature vector with the crack detection result and output the three-dimensional geometric parameters of the crack.

9. The system according to claim 1, characterized in that, The system also includes: The crack matching module is used to match the center point coordinates, direction angle and cross-modal physical consistency map of the currently detected crack with the records in the historical crack database. When the center point distance is less than the first preset distance and the direction angle difference is less than the first preset angle, it is determined to be the same crack. The Kalman filter prediction module is used to input the historical length and width sequences of successfully matched cracks into the Kalman filter, predict the crack length and width of the current detection cycle, and calculate the residual between the predicted value and the detected value. The confidence correction module is used to correct the confidence level output by the dynamic confidence propagation unit based on the residual. When the residual is less than a preset residual threshold, the confidence level is increased; otherwise, the confidence level is decreased.

10. A method for detecting cracks in a multimodal embankment using a patrol robot based on the system described in any one of claims 1 to 9, characterized in that, Includes the following steps: Step S1: Simultaneously collect visual image data, sonar echo data, and robot motion state data of the dam surface; Step S2: Based on the phase delay of the reflected wave caused by the crack interface in the sonar echo data and the gray-level gradient direction of the crack edge in the visual image data, calculate the geometric constraint loss of the acoustic phase field and the visual gradient field to generate a cross-modal physical consistency map. Step S3: Calculate the instantaneous confidence of the visual and acoustic modes based on the triaxial acceleration amplitude and vibration spectrum energy in the robot motion state data, and update the confidence of the sensor data at different spatiotemporal locations through the message passing mechanism of the spatiotemporal graph neural network. Step S4: The cross-modal physical consistency map and the updated confidence level are weighted and fused, and then input into the crack segmentation network to output the crack detection result; Step S5: Match the crack detection results with the historical crack database, predict the crack development trend through Kalman filtering, and correct the confidence level based on the prediction residual.

Images