Dam measurement system and method based on multi-modal data processing

Abstract

This invention discloses a dam measurement system and method based on multimodal data processing, belonging to the field of dam measurement system data processing technology. It employs a physical model to extract dam physical morphological change features from visual data, sound features associated with dam physical changes from audio data, and micro-displacement features from radar point cloud data. This invention constructs a three-dimensional monitoring network from macroscopic deformation to microscopic displacement through the synergy of visual, audio, and radar three-modal data, overcoming the blind spots of single-sensor monitoring and realizing the correlation analysis between dam surface deformation and internal stress changes. Physically constrained feature fusion embeds finite element model constraints at the feature-level fusion stage. Radar point cloud displacement vectors must conform to geological structural direction constraints, and audio acoustic features must be spatiotemporally synchronized with the visual crack propagation rate to eliminate environmental noise interference. Crack propagation early warning time is significantly advanced; cavitation damage location is more accurate; and the recognition rate of abnormal operating conditions is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology for dam measurement systems, and in particular to a dam measurement system and method based on multimodal data processing. Background Technology

[0002] Current dam monitoring technologies mainly follow these technical approaches:

[0003] Single-modal monitoring system: relies on manual inspection or fixed cameras, and has blind spots (such as the back of the dam body and underwater structures).

[0004] Traditional point cloud technology can only obtain static displacement, lacking dynamic coupling with the physical model (e.g., it does not consider the constraints of geological conditions on the displacement direction).

[0005] Different modal data have independent coordinate systems (e.g., there is no unified benchmark between visual pixel coordinates and radar point cloud coordinates).

[0006] Data-driven models (such as ResNet and LSTM) are separated from finite element models, making it impossible to verify whether the output conforms to the laws of mechanics;

[0007] Therefore, developing a dam measurement system and corresponding data processing method based on multimodal data processing to solve the problems of the traditional single-modal monitoring system is an urgent technical issue that needs to be addressed. Summary of the Invention

[0008] To address the technical problems in partial discharge detection and diagnosis of the aforementioned equipment, this invention provides a dam measurement system and method based on multimodal data processing. The technical solution adopted is as follows:

[0009] A method for processing multimodal data from a dam measurement system includes the following steps:

[0010] Step 1: Collect multimodal data from the dam measurement system. The multimodal data includes visual data, audio data, and radar point cloud data.

[0011] Step 2: Preprocess the multimodal data separately and perform cross-modal spatiotemporal alignment;

[0012] Step 3: Use physical models to extract the physical morphological change features of the dam from the visual data, the sound features associated with the physical changes of the dam from the audio data, and the micro-displacement features from the radar point cloud data.

[0013] Step 4: Multi-level cross-modal fusion of the physical morphological change characteristics of the dam, the sound characteristics associated with the physical changes of the dam, and the micro-displacement characteristics; layered fusion of local features and global state; and physical verification to ensure the rationality of the results.

[0014] Step 5: Integrate multimodal risk assessment, dynamically output dam health index, and output dam risk early warning signal based on dam health index threshold.

[0015] Optionally, in step 1, multiple visual cameras are used to cover the entire dam area; a microphone array is used to cover key parts of the dam, including the gate area, the gap area, and the dam foundation area; and a millimeter-level radar array is used to cover the entire dam area, each corresponding to a collection point of multiple visual cameras.

[0016] By adopting the above technical solutions, cameras cover the entire dam area, enabling real-time visual monitoring of macroscopic defects such as surface cracks, seepage, and deformation. Microphone arrays focus on key areas such as the gate area, gap area, and dam foundation area, accurately capturing concealed acoustic signals such as crack propagation, cavitation, and structural loosening. A millimeter-level radar array provides full dam coverage, corresponding to visual locations, enabling sub-millimeter-level micro-displacement monitoring (such as dam deformation and foundation settlement).

[0017] It provides seamless spatial coverage and synchronous data acquisition in time, completely solving the blind spot problem of traditional single-modal monitoring.

[0018] The same event (such as crack propagation) can be captured simultaneously by vision (deformation images), audio (crack acoustic emissions), and radar (sudden displacement increase), and the multi-source data can be mutually verified, significantly reducing the false alarm rate.

[0019] Visual recognition of cracks + audio detection of crack expansion characteristics + radar detection of abnormal displacement in the area → high confidence warning.

[0020] In complex environments such as rain and fog (visual impairment) and flood discharge noise (audio interference), monitoring continuity is ensured through multimodal complementarity (such as radar data dominance).

[0021] Based on the finite element model, deformation is quantified and correlated with the mechanical response of the dam body, thus avoiding the "black box" risk of traditional image recognition.

[0022] CMM models identify specific physical events (cavitation, crack sound) and directly correlate them with structural damage mechanisms.

[0023] By combining static / flood discharge conditions with dynamic thresholds, normal deformation and abnormal displacement can be distinguished.

[0024] Correlate local features of different modes (e.g., abnormal displacement at a certain point + abnormal noise in the corresponding area).

[0025] The overall stability is assessed by comprehensively evaluating the status of key components (such as the impact of micro-displacement of the dam foundation on overall safety).

[0026] Verify whether the fusion results conform to the laws of mechanics (such as the consistency between displacement direction and load, and the physical correlation between sound, deformation and displacement), and intercept erroneous outputs that violate the laws of physics.

[0027] To avoid the "illusion" risks of purely data-driven models, the output combines data intelligence with physical credibility.

[0028] The dam's real-time status is quantified by integrating multimodal features (such as visual deformation weight of 30%, audio event weight of 20%, and radar displacement weight of 50%), and a health index of 0-100 is output.

[0029] Early warnings are triggered based on health index levels (e.g., ≥80 is normal, 60-80 requires observation, <60 requires emergency repair), supporting tiered responses. This upgrades from passive response to proactive prediction, significantly improving risk management efficiency.

[0030] Optionally, step 2 includes the following sub-steps:

[0031] Step 21: Establish the mapping relationship between the visual coordinate system and the radar coordinate system using a checkerboard calibration board;

[0032] Step 22: Transmit a reference acoustic pulse at a known location on the dam crest, record the time difference of arrival using a microphone array, and calculate the location of the sound source;

[0033] Step 23: Select the dam synchronization event for time alignment and unify the time reference.

[0034] By adopting the above technical solution, using a checkerboard calibration board, the camera's internal and external parameters are calculated through binocular visual calibration. Combined with radar point cloud registration, the rigid transformation matrix (rotation matrix R + translation vector T) between the visual pixel coordinates and the radar 3D point cloud coordinates is solved. This ensures that the coordinates of the same physical location (such as a crack point) are consistent in the visual image and the radar displacement field (error ≤ 1cm), avoiding fusion failure caused by "position shift of the same event in different coordinate systems".

[0035] Supporting cross-modal correlation: It provides a geometric basis for the subsequent feature fusion of "visual crack morphology + radar micro-displacement at the same point", and achieves pixel-level spatial alignment.

[0036] A sound pulse is emitted from a known coordinate point on the top of the dam. The microphone array calculates the time difference of sound wave propagation to each microphone using the time difference of arrival (TDOA), and uses an optimization algorithm (such as the least squares method) to infer the location of the sound source.

[0037] Abstract audio signals are converted into three-dimensional spatial coordinates (such as identifying the source of crack propagation sound in the X area of ​​the dam foundation), enabling spatial correlation between sound events and visual / radar data.

[0038] Pulse signals have a high signal-to-noise ratio, enabling accurate positioning even in the context of flood discharge noise, thus solving the problem of traditional acoustic monitoring being susceptible to environmental interference.

[0039] Focus on abnormal sounds in areas such as gates and dam foundations (e.g., cavitation sound coordinates are close to the flood discharge gate) to improve the accuracy of diagnosis.

[0040] Sensor clock synchronization at the microsecond level is achieved through the GPS / PTP protocol.

[0041] Capture synchronous events of the dam (such as gate opening and closing vibrations, flood discharge flow impacts), and use these events as a benchmark to perform timestamp normalization on the multimodal data.

[0042] Eliminate temporal drift: Ensure that data on visual deformation, audio events, and radar displacement occurring at the same time are analyzed synchronously (e.g., matching the peak value of vibration sound during flood discharge with the peak value of dam displacement).

[0043] It enables millisecond-level correlation analysis of transient events (such as instantaneous crack propagation) to avoid causal misjudgments caused by time asynchrony.

[0044] Optionally, in step 3, the method for extracting the physical morphological change characteristics of the dam is as follows: based on the spatiotemporally aligned visual data and the dam body finite element model, the visual characteristics of the physical morphological change of the dam are quantified, and a visual characteristic change threshold is set. Quantified visual characteristics that are greater than the visual characteristic change threshold are marked as physical morphological change characteristics of the dam.

[0045] Optionally, in step 3, the method for extracting the sound features associated with the physical changes of the dam is to use a speech recognition model (CMM) to detect sound events. The output items of the sound event detection results are: normal sound features, crack propagation sound features, and cavitation sound features.

[0046] Optionally, in step 3, the method for extracting micro-displacement features is as follows: first, point cloud registration is performed, and then displacement field calculation is performed to obtain the tangential and normal displacements of the monitoring points. If the condition is static, a static displacement field threshold is set; if the condition is flood discharge, a flood discharge displacement field threshold is set; if the tangential or normal displacement is greater than the corresponding displacement field threshold, an anomaly is marked.

[0047] By adopting the above technical solutions, the finite element model transforms images into mechanical language, the CMM model translates sound waves into a damage event dictionary, and the working condition adaptive threshold realizes displacement diagnosis based on environmental perception.

[0048] The feature extraction process deeply integrates structural mechanics principles, eliminating the risk of a "black box" approach. The output can be directly used in structural safety assessment reports (e.g., "Cavitation in the gate area caused tangential displacement to exceed the flood discharge threshold by 2.1 mm"), thus constructing a physically interpretable and quantitatively executable intelligent sensing foundation for dam health diagnosis.

[0049] Optionally, in step 4, a cross-modal attention mechanism is used to calculate the intermodal interaction weights, and the radar micro-displacement features, visual morphological features and audio event features are fused at the feature level to output the fused local feature matrix.

[0050] The local feature matrix and the overall physical model of the dam are then input into the graph neural network (GNN) to output the global health state vector.

[0051] By adopting the above technical solution, the input feature matrix includes radar micro-displacement modes, and the feature symbols are... In the visual feature modality, the feature symbol is In radar micro-displacement modes, the audio characteristic modes are: ;

[0052] First, the features are converted into a uniform-dimensional tensor, then cross-modal attention is calculated, and finally the features are concatenated to output a global health state vector. Attention weights reveal feature correlations (e.g., crack direction is highly correlated with tangential displacement), and the output... Includes enhanced features following cross-modal interactions.

[0053] Optionally, in step 4, the physical verification is performed by applying predefined physical rules and a physical simulation model to verify the global health state vector;

[0054] Predefined physical rules include: the displacement direction of the dam foundation must conform to geological conditions and loads;

[0055] The intensity of the sound of crack propagation should match the magnitude of the crack size change observed visually and the magnitude of the displacement in the area monitored by radar. If they do not match, it is determined that it does not conform to the physical rules.

[0056] The vibration audio and radar micro-displacement characteristics under flood discharge conditions should be stronger than those under static conditions; otherwise, it is judged to be inconsistent with physical rules.

[0057] Cavitation sounds should occur in areas of high-speed water flow. If cavitation sounds occur in dry areas, it is determined that they do not conform to physical rules.

[0058] By adopting the above technical solution, unlike the pure data-driven method, the reliability of the results is guaranteed by first principles, and the error is greatly reduced in the strong interference environment; physical rules intercept abnormal data that do not match the sensor noise (such as radar water mist interference) and the operating conditions.

[0059] Optionally, in step 5, the global health state vector output in step 4 is subjected to multimodal feature weighted fusion. The calculation formula for multimodal feature weighted fusion is:

[0060] ;

[0061] in It is the dam health index. These are the eigenvalues ​​of the first mode. These are modal weights, where n is the total number of modes. It represents the theoretical state deviation output by the physical model, and the global state of the graph neural network (GNN). It is the physical factor adjustment coefficient.

[0062] By adopting the above technical solution, and through weighted fusion of multi-source features such as visual morphology, audio events, and radar micro-displacement (as shown in the formula), This approach can effectively compensate for the limitations of a single mode. For example, visual features can capture surface cracks, audio features can detect changes in internal stress, and radar features can quantify displacement magnitudes. The synergy of these three features can reduce the false alarm rate.

[0063] By introducing theoretical state bias (T) and physical factor adjustment coefficient (β), the prediction result (G) of the graph neural network (GNN) is dynamically corrected to the theoretical value of the physical model. For example, in flood discharge conditions, the β coefficient can amplify the weight of structural deformation characteristics caused by water flow impact, making the health index (H) more consistent with actual physical laws.

[0064] A dam measurement system based on multimodal data processing is used to implement a multimodal data processing method for dam measurement. The dam measurement system includes multiple visual cameras, a microphone array, a millimeter-level radar array, a data acquisition module, and a data processing computer. The multiple visual cameras cover the entire dam, and the microphone array covers key parts of the dam. The signal output terminals of the multiple visual cameras, microphone array, and millimeter-level radar array are respectively connected to the signal input terminal of the data acquisition module. The signal output terminal of the data acquisition module is connected to the data processing computer. The data processing computer is pre-installed with a data processing program designed using the multimodal data processing method of the dam measurement system. Running the data processing program outputs a dam health index and outputs a dam risk warning signal based on the dam health index threshold.

[0065] In summary, the present invention has at least one of the following beneficial technical effects:

[0066] This invention provides a dam measurement system and method based on multimodal data processing. By coordinating visual, audio, and radar data, a three-dimensional monitoring network from macroscopic deformation to microscopic displacement is constructed, breaking through the blind spots of single sensor monitoring and realizing the correlation analysis between dam surface deformation and internal stress changes.

[0067] Physically constrained feature fusion embeds finite element model constraints at the feature-level fusion stage. Radar point cloud displacement vectors must conform to geological structural direction constraints, and audio fingerprint features must be spatiotemporally synchronized with visual crack propagation rates to eliminate environmental noise interference and ensure that fused features conform to the physical laws of the dam, avoiding false correlations at the data level. Crack propagation early warning time is significantly advanced; cavitation damage location is more accurate, and the abnormal operating condition recognition rate is greatly improved; operation and maintenance costs are optimized, ineffective inspections are reduced, sensor redundancy is significantly reduced, and the estimated lifespan of the dam's main structure is extended. Attached Figure Description

[0068] Figure 1 This is a schematic diagram of the electrical component connection principle of the dam measurement system based on multimodal data processing according to the present invention; Detailed Implementation

[0069] The present invention will be further described in detail below with reference to the accompanying drawings.

[0070] This invention discloses a dam measurement system and method based on multimodal data processing.

[0071] Reference Figure 1 Example 1, a multimodal data processing method for a dam measurement system, includes the following steps:

[0072] Step 1: Collect multimodal data from the dam measurement system. The multimodal data includes visual data, audio data, and radar point cloud data.

[0073] Step 2: Preprocess the multimodal data separately and perform cross-modal spatiotemporal alignment;

[0074] Step 3: Use physical models to extract the physical morphological change features of the dam from the visual data, the sound features associated with the physical changes of the dam from the audio data, and the micro-displacement features from the radar point cloud data.

[0075] Step 4: Multi-level cross-modal fusion of the physical morphological change characteristics of the dam, the sound characteristics associated with the physical changes of the dam, and the micro-displacement characteristics; layered fusion of local features and global state; and physical verification to ensure the rationality of the results.

[0076] Step 5: Integrate multimodal risk assessment, dynamically output dam health index, and output dam risk early warning signal based on dam health index threshold.

[0077] In Example 2, in step 1, multiple visual cameras are used to cover the entire dam area; a microphone array is used to cover key parts of the dam, including the gate area, the gap area, and the dam foundation area; and a millimeter-level radar array is used to cover the entire dam area, corresponding to the acquisition points of the multiple visual cameras.

[0078] Cameras cover the entire dam area, enabling real-time visual monitoring of macroscopic defects such as surface cracks, seepage, and deformation. Microphone arrays focus on key areas such as the gate area, gap area, and dam foundation area, accurately capturing hidden acoustic signals such as crack propagation, cavitation, and structural loosening. A millimeter-level radar array provides full dam coverage, corresponding to visual locations, enabling sub-millimeter-level micro-displacement monitoring (such as dam deformation and foundation settlement).

[0079] It provides seamless spatial coverage and synchronous data acquisition in time, completely solving the blind spot problem of traditional single-modal monitoring.

[0080] The same event (such as crack propagation) can be captured simultaneously by vision (deformation images), audio (crack acoustic emissions), and radar (sudden displacement increase), and the multi-source data can be mutually verified, significantly reducing the false alarm rate.

[0081] Visual recognition of cracks + audio detection of crack expansion characteristics + radar detection of abnormal displacement in the area → high confidence warning.

[0082] In complex environments such as rain and fog (visual impairment) and flood discharge noise (audio interference), monitoring continuity is ensured through multimodal complementarity (such as radar data dominance).

[0083] Based on the finite element model, deformation is quantified and correlated with the mechanical response of the dam body, thus avoiding the "black box" risk of traditional image recognition.

[0084] CMM models identify specific physical events (cavitation, crack sound) and directly correlate them with structural damage mechanisms.

[0085] By combining static / flood discharge conditions with dynamic thresholds, normal deformation and abnormal displacement can be distinguished.

[0086] Correlate local features of different modes (e.g., abnormal displacement at a certain point + abnormal noise in the corresponding area).

[0087] The overall stability is assessed by comprehensively evaluating the status of key components (such as the impact of micro-displacement of the dam foundation on overall safety).

[0088] Verify whether the fusion results conform to the laws of mechanics (such as the consistency between displacement direction and load, and the physical correlation between sound, deformation and displacement), and intercept erroneous outputs that violate the laws of physics.

[0089] To avoid the "illusion" risks of purely data-driven models, the output combines data intelligence with physical credibility.

[0090] The dam's real-time status is quantified by integrating multimodal features (such as visual deformation weight of 30%, audio event weight of 20%, and radar displacement weight of 50%), and a health index of 0-100 is output.

[0091] Early warnings are triggered based on health index levels (e.g., ≥80 is normal, 60-80 requires observation, <60 requires emergency repair), supporting tiered responses. This upgrades from passive response to proactive prediction, significantly improving risk management efficiency.

[0092] Example 3, step 2 includes the following sub-steps:

[0093] Step 21: Establish the mapping relationship between the visual coordinate system and the radar coordinate system using a checkerboard calibration board;

[0094] Step 22: Transmit a reference acoustic pulse at a known location on the dam crest, record the time difference of arrival using a microphone array, and calculate the location of the sound source;

[0095] Step 23: Select the dam synchronization event for time alignment and unify the time reference.

[0096] Using a checkerboard calibration board, the camera's intrinsic and extrinsic parameters are calculated through binocular visual calibration. Combined with radar point cloud registration, the rigid transformation matrix (rotation matrix R + translation vector T) between the visual pixel coordinates and the radar 3D point cloud coordinates is solved. This ensures that the coordinates of the same physical location (such as a crack point) are consistent in the visual image and the radar displacement field (error ≤ 1cm), avoiding fusion failure caused by "positional shift of the same event in different coordinate systems".

[0097] Supporting cross-modal correlation: It provides a geometric basis for the subsequent feature fusion of "visual crack morphology + radar micro-displacement at the same point", and achieves pixel-level spatial alignment.

[0098] A sound pulse is emitted from a known coordinate point on the top of the dam. The microphone array calculates the time difference of sound wave propagation to each microphone using the time difference of arrival (TDOA), and uses an optimization algorithm (such as the least squares method) to infer the location of the sound source.

[0099] Abstract audio signals are converted into three-dimensional spatial coordinates (such as identifying the source of crack propagation sound in the X area of ​​the dam foundation), enabling spatial correlation between sound events and visual / radar data.

[0100] Pulse signals have a high signal-to-noise ratio, enabling accurate positioning even in the context of flood discharge noise, thus solving the problem of traditional acoustic monitoring being susceptible to environmental interference.

[0101] Focus on abnormal sounds in areas such as gates and dam foundations (e.g., cavitation sound coordinates are close to the flood discharge gate) to improve the accuracy of diagnosis.

[0102] Sensor clock synchronization at the microsecond level is achieved through the GPS / PTP protocol.

[0103] Capture synchronous events of the dam (such as gate opening and closing vibrations, flood discharge flow impact), and use these events as a benchmark to perform timestamp normalization on the multimodal data.

[0104] Eliminate temporal drift: Ensure that data on visual deformation, audio events, and radar displacement occurring at the same time are analyzed synchronously (e.g., matching the peak value of vibration sound during flood discharge with the peak value of dam displacement).

[0105] It enables millisecond-level correlation analysis of transient events (such as instantaneous crack propagation) to avoid causal misjudgments caused by time asynchrony.

[0106] In Example 4, step 3, the method for extracting the physical morphological change features of the dam is as follows: based on the spatiotemporally aligned visual data and the dam body finite element model, the visual features of the physical morphological change of the dam are quantified, and a visual feature change threshold is set. Quantified visual features that are greater than the visual feature change threshold are marked as physical morphological change features of the dam.

[0107] In Example 5, step 3, the method for extracting sound features associated with physical changes in the dam is as follows: sound event detection is performed using a speech recognition model (CMM), and the output items of the sound event detection results are: normal sound features, crack propagation sound features, and cavitation sound features.

[0108] In Example 6, step 3, the method for extracting micro-displacement features is as follows: first, point cloud registration is performed, and then displacement field calculation is performed to obtain the tangential and normal displacements of the monitoring points. If the condition is static, a static displacement field threshold is set; if the condition is flood discharge, a flood discharge displacement field threshold is set; if the tangential or normal displacement is greater than the corresponding displacement field threshold, an anomaly is marked.

[0109] The finite element model transforms images into mechanical language, the CMM model translates sound waves into a dictionary of damage events, and the working condition adaptive threshold enables displacement diagnosis based on environmental perception.

[0110] The feature extraction process deeply integrates structural mechanics principles, eliminating the risk of a "black box" approach. The output can be directly used in structural safety assessment reports (e.g., "Cavitation in the gate area caused tangential displacement to exceed the flood discharge threshold by 2.1 mm"), thus constructing a physically interpretable and quantitatively executable intelligent sensing foundation for dam health diagnosis.

[0111] In Example 7, step 4, a cross-modal attention mechanism is used to calculate the intermodal interaction weights, and the radar micro-displacement features, visual morphological features and audio event features are fused at the feature level to output the fused local feature matrix.

[0112] The local feature matrix and the overall physical model of the dam are then input into the graph neural network (GNN) to output the global health state vector.

[0113] The input feature matrix includes radar micro-displacement modes, and the feature symbols are... In the visual feature modality, the feature symbol is In radar micro-displacement modes, the audio characteristic modes are: ;

[0114] First, the features are converted into a uniform-dimensional tensor, then cross-modal attention is calculated, and finally the features are concatenated to output a global health state vector. Attention weights reveal feature correlations (e.g., crack direction is highly correlated with tangential displacement), and the output... Includes enhanced features following cross-modal interactions.

[0115] In Example 8, the physical verification in step 4 involves using predefined physical rules and a physical simulation model to verify the global health state vector.

[0116] Predefined physical rules include: the displacement direction of the dam foundation must conform to geological conditions and loads;

[0117] The intensity of the sound of crack propagation should match the magnitude of the crack size change observed visually and the magnitude of the displacement in the area monitored by radar. If they do not match, it is determined that it does not conform to the physical rules.

[0118] The vibration audio and radar micro-displacement characteristics under flood discharge conditions should be stronger than those under static conditions; otherwise, it is judged to be inconsistent with physical rules.

[0119] Cavitation sounds should occur in areas of high-speed water flow. If cavitation sounds occur in dry areas, it is determined that they do not conform to physical rules.

[0120] Unlike purely data-driven methods, it ensures the reliability of results through first principles and significantly reduces errors in environments with strong interference; physical rules intercept abnormal data that do not match sensor noise (such as radar water mist interference) and operating conditions.

[0121] In Example 9, step 5 involves performing multimodal feature weighted fusion on the global health state vector output from step 4. The calculation formula for multimodal feature weighted fusion is:

[0122] ;

[0123] in It is the dam health index. These are the eigenvalues ​​of the first mode. These are modal weights, where n is the total number of modes. It represents the theoretical state deviation output by the physical model, and the global state of the graph neural network (GNN). It is the physical factor adjustment coefficient.

[0124] By weighted fusion of multi-source features such as visual morphology, audio events, and radar micro-displacement (as shown in the formula) This approach can effectively compensate for the limitations of a single mode. For example, visual features can capture surface cracks, audio features can detect changes in internal stress, and radar features can quantify displacement magnitudes. The synergy of these three features can reduce the false alarm rate.

[0125] By introducing theoretical state bias (T) and physical factor adjustment coefficient (β), the prediction result (G) of the graph neural network (GNN) is dynamically corrected to the theoretical value of the physical model. For example, in flood discharge conditions, the β coefficient can amplify the weight of structural deformation characteristics caused by water flow impact, making the health index (H) more consistent with actual physical laws.

[0126] Example 10: A dam measurement system based on multimodal data processing is used to implement a multimodal data processing method for dam measurement systems. The dam measurement system includes multiple visual cameras, a microphone array, a millimeter-level radar array, a data acquisition module, and a data processing computer. The multiple visual cameras cover the entire dam, and the microphone array covers key parts of the dam. The signal output terminals of the multiple visual cameras, microphone array, and millimeter-level radar array are respectively communicatively connected to the signal input terminal of the data acquisition module. The signal output terminal of the data acquisition module is communicatively connected to the data processing computer. The data processing computer is pre-installed with a data processing program designed using the multimodal data processing method for dam measurement systems. Running the data processing program outputs a dam health index and outputs a dam risk warning signal based on the dam health index threshold.

[0127] The following specific embodiments illustrate the implementation principle of the present invention:

[0128] System components:

[0129] 1. Sensor Networks:

[0130] Visual perception layer: 128 4K resolution anti-fog cameras, forming a 15m×15m grid covering the upstream and downstream sides of the dam and the interior of the corridor.

[0131] Acoustic perception layer:

[0132] Gate area: 8 sets of noise-canceling industrial microphone arrays (sampling rate 48kHz);

[0133] Crack-sensitive area: 36 resonant acoustic emission sensors (frequency response 30-150kHz);

[0134] Dam foundation area: 12 sets of waterproof hydrophone arrays (sensitivity -180dB);

[0135] Radar perception layer: 77GHz millimeter-wave radar array (50 units), with one-to-one spatial coordinates of visual cameras;

[0136] 2. Data Acquisition Module:

[0137] Multi-channel synchronous acquisition box (supports PTP precise clock protocol);

[0138] Environmental parameter integration unit (real-time acquisition of temperature, humidity, and water level data);

[0139] 3. Data processing computer:

[0140] AI server equipped with NVIDIA A100 GPU;

[0141] Pre-installed multimodal fusion system (including physics rule engine and finite element solver);

[0142] Workflow:

[0143] Step 1: Multimodal data acquisition;

[0144] Example of flood discharge operation (reservoir water level rises suddenly by 3m):

[0145] Visual system: detects abnormal water flow patterns on the overflow surface;

[0146] Audio system: A 120Hz abnormal vibration harmonic was recorded by the microphone array in the gate area;

[0147] Radar system: A sudden increase in tangential displacement of 0.35 mm was detected in the dam foundation area;

[0148] Step 2: Cross-modal spatiotemporal alignment;

[0149] 1. Spatial calibration:

[0150] Through the pre-set calibration plate (number G12) in the corridor.

[0151] Calculate the visual radar transformation matrix with an alignment error ≤ 8mm;

[0152] 2. Sound source localization:

[0153] A sound pulse is emitted at the control point on the dam crest (coordinates X=235.6, Y=118.2);

[0154] The source of the abnormal vibration was located in the gate hinge area (coordinates X=201.5, Y=97.3).

[0155] 3. Time synchronization:

[0156] The moment the floodgate opens is taken as the baseline event;

[0157] Dynamic time warping eliminates timing drift (synchronization accuracy ±2ms);

[0158] Step 3: Physical Feature Extraction

[0159] Visual features:

[0160] U-Net identifies overflow surface cracks (0.28 mm wide, 1.2 m long) and marks those exceeding the threshold (0.2 mm) as abnormal.

[0161] Audio characteristics:

[0162] CMM model identifies features;

[0163] Radar characteristics;

[0164] The tangential displacement of the dam foundation is 0.35 mm (flood discharge threshold = 0.3 mm).

[0165] Normal displacement 0.18mm (normal range);

[0166] Step 4: Multi-level cross-modal fusion:

[0167] 1. Feature-level fusion:

[0168] Multi-head attention mechanism calculates association weights:

[0169] The correlation weight between crack direction (125°) and tangential displacement is 0.92;

[0170] The correlation weight between cavitation sound and gate area displacement is 0.87;

[0171] 2. Global Fusion of GNNs:

[0172] Construct the dam diagram structure (2,385 nodes);

[0173] Output health state vector: `[0.73,0.15,0.08,...]` (128 dimensions);

[0174] 3. Physical verification:

[0175] Rule 1: Displacement direction (82°) conforms to geological structure (80±5°) → Pass;

[0176] Rule 2: Cavitation sounds occur in the wet area of ​​the flood discharge zone → Pass;

[0177] Rule 3: Discharge vibration intensity > 3.2 times that of static conditions → Pass;

[0178] Step 5: Safety assessment and early warning;

[0179] The health index is calculated to be 76.4.

[0180] Early warning decision: Health index 76.4, risk level is Level 1 warning, response measures are to increase sampling frequency to 5Hz and initiate FEM simulation verification;

[0181] Technical effects:

[0182] 1. False alarm interception:

[0183] Physical rules block two error alerts:

[0184] Visual misjudgment of water stains as cracks (sound energy mismatch);

[0185] False displacement of radar due to interference from birds (direction does not conform to geology).

[0186] 2. Early detection of potential hazards:

[0187] The integrated system provides an early warning of gate hinge abnormalities 48 hours in advance.

[0188] Tangential displacement 0.28mm (not exceeding the threshold);

[0189] However, the combination of sound characteristics and visual wear marks forms a high-weighted correlation.

[0190] Engineering value: In a certain water conservancy project, the system successfully predicted the risk of dam foundation sliding during the 2025 flood season and avoided direct economic losses of hundreds of millions of yuan through pre-grouting reinforcement.

[0191] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. Dam survey system multi-modal data processing method, characterized in that: Includes the following steps Step 1: Collect multimodal data from the dam measurement system. The multimodal data includes visual data, audio data, and radar point cloud data. Step 2: Preprocess the multimodal data separately and perform cross-modal spatiotemporal alignment; Step 3: Use physical models to extract the physical morphological change features of the dam from the visual data, the sound features associated with the physical changes of the dam from the audio data, and the micro-displacement features from the radar point cloud data. Step 4: Multi-level cross-modal fusion of the physical morphological change characteristics of the dam, the sound characteristics associated with the physical changes of the dam, and the micro-displacement characteristics; layered fusion of local features and global state; and physical verification to ensure the rationality of the results. Step 5: Integrate multimodal risk assessment, dynamically output dam health index, and output dam risk early warning signal based on dam health index threshold; In step 1, multiple visual cameras are used to cover the entire dam area; a microphone array is used to cover key parts of the dam, including the gate area, the gap area, and the dam foundation area. A millimeter-level radar array is used to cover the entire dam area, and each location corresponds to a collection point of multiple visual cameras. In step 4, a cross-modal attention mechanism is used to calculate the intermodal interaction weights, and the radar micro-displacement features, visual morphological features and audio event features are fused at the feature level to output the fused local feature matrix. Then, the local feature matrix and the overall physical model of the dam are input into the graph neural network (GNN) to output the global health state vector. In step 4, the physical verification involves using predefined physical rules and a physical simulation model to verify the global health state vector. Predefined physical rules include: the displacement direction of the dam foundation must conform to geological conditions and loads; The intensity of the sound of crack propagation should match the magnitude of the crack size change observed visually and the magnitude of the regional displacement detected by radar. If they do not match, it is determined that it does not conform to the physical rules. The vibration audio and radar micro-displacement characteristics under flood discharge conditions should be stronger than those under static conditions; otherwise, it is judged to be inconsistent with physical rules. Cavitation sounds should occur in areas of high-speed water flow. If cavitation sounds occur in dry areas, it is determined that they do not conform to physical rules. In step 5, the global health state vector output in step 4 is subjected to multimodal feature weighted fusion. The calculation formula for multimodal feature weighted fusion is: ； in It is the dam health index. These are the eigenvalues ​​of the first mode. These are modal weights, where n is the total number of modes. It represents the theoretical state deviation output by the physical model, and the global state of the graph neural network (GNN). It is the physical factor adjustment coefficient.

2. The method of claim 1, wherein: Step 2 includes the following sub-steps: Step 21: Establish the mapping relationship between the visual coordinate system and the radar coordinate system using a checkerboard calibration board; Step 22: Transmit a reference acoustic pulse at a known location on the dam crest, record the time difference of arrival using a microphone array, and calculate the location of the sound source; Step 23: Select the dam synchronization event for time alignment and unify the time reference.

3. The dam survey system multi-modal data processing method of claim 2, wherein, In step 3, the method for extracting the physical morphological change characteristics of the dam is as follows: based on the spatiotemporally aligned visual data and the dam body finite element model, the visual characteristics of the physical morphological change of the dam are quantified, and a visual characteristic change threshold is set. Quantified visual characteristics that are greater than the visual characteristic change threshold are marked as physical morphological change characteristics of the dam.

4. The dam survey system multi-modal data processing method of claim 3, wherein, In step 3, the method for extracting sound features associated with physical changes in the dam is as follows: sound event detection is performed using a speech recognition model (CMM). The output items of the sound event detection results are: normal sound features, crack propagation sound features, and cavitation sound features.

5. The dam survey system multi-modal data processing method of claim 4, wherein, In step 3, the method for extracting micro-displacement features is as follows: first, point cloud registration is performed, and then displacement field calculation is performed to obtain the tangential and normal displacements of the monitoring points. If the condition is static, a static displacement field threshold is set; if the condition is flood discharge, a flood discharge displacement field threshold is set; if the tangential or normal displacement is greater than the corresponding displacement field threshold, an anomaly is marked.

6. Dam measurement system based on processing of multi-modal data, characterized in that: To implement the multimodal data processing method for the dam measurement system as described in claim 5, the dam measurement system includes multiple visual cameras, a microphone array, a millimeter-level radar array, a data acquisition module, and a data processing computer. The multiple visual cameras cover the entire dam, the microphone array covers key parts of the dam, and the signal output terminals of the multiple visual cameras, microphone array, and millimeter-level radar array are respectively communicatively connected to the signal input terminal of the data acquisition module. The signal output terminal of the data acquisition module is communicatively connected to the data processing computer. The data processing computer is pre-installed with a data processing program designed using the multimodal data processing method for the dam measurement system as described in claim 5. The data processing program outputs a dam health index and outputs a dam risk warning signal based on the dam health index threshold.

Images