An intelligent floating stone risk elimination method and system based on multi-modal data alignment

Abstract

The present application belongs to the technical field of floatstone risk elimination, and particularly relates to an intelligent floatstone risk elimination method and system based on multi-modal data alignment. In the multi-modal perception and collection step, three types of sensing devices, i.e. images, voiceprints and vibrations, are used to synchronously collect rock mass surface topography, knocking acoustic response and impact vibration timing data. In the cross-modal physical alignment and feature extraction step, multi-modal data space-time constraint relationships are established based on rock mass mechanical transmission rules, and image crack features, voiceprint damage features and vibration structure features are extracted respectively. In the physical driving multi-modal fusion step, a fusion network is constructed based on rock fracture mechanics and rock mass damage evolution causality rules, each modal fusion weight is dynamically and adaptively distributed, and a stability quantitative index representing the rock mass loosening degree is calculated by introducing a physical information neural network.

Description

Technical Field

[0001] This invention belongs to the field of floating rock hazard removal technology, and particularly relates to an intelligent floating rock hazard removal method and system based on multimodal data alignment. Background Technology

[0002] Long-term geological stress disturbance in mine roadways and tunnels easily leads to loose rocks, surface cracks, and internal cavities. Traditional methods of loose rock detection rely on manual tapping with hand tools and listening for sounds, resulting in low detection efficiency, safety hazards from falling rocks during close-range operations, and a high risk of missed detection due to the reliance on operator experience and subjectivity. Existing automated loose rock detection equipment mostly uses single visual or single vibration detection methods, relying solely on fixed weights to simply fuse multi-source data without establishing physical constraints based on rock fracture mechanics. This makes it prone to numerous misjudgments in complex underground environments with interference such as dust, water accumulation, and on-site mechanical vibration. Furthermore, it lacks logical verification of the evolution of rock damage, making it impossible to distinguish between actual rock damage and environmental noise signals. Risk assessment lacks physical explanation, resulting in insufficient accuracy in graded treatment.

[0003] Existing AI-based multimodal rock detection solutions generally employ data-driven blind alignment and pure statistical neural network classification architectures. They fail to establish physical transmission constraints between vibration and acoustic signatures, and modal fusion only performs feature splicing, failing to capture the inherent mechanical causal relationships between cracks, cavities, and loosening. Furthermore, the modal fusion weights are pre-fixed and cannot be adaptively adjusted according to on-site lighting, dust, and noise intensity. There is also no three-dimensional consistency verification mechanism for space, time, and depth. When conflicts arise between image, acoustic signature, and vibration detection results, there is no sound arbitration logic, making it difficult to output stable and reliable risk assessment results. The supporting hazard mitigation execution mechanism lacks stable support and a folding and rotating structure, resulting in poor overall operational stability and mobility, making it difficult to adapt to the narrow and variable working conditions of underground tunnels. Summary of the Invention

[0004] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide an intelligent floating rock hazard removal method based on multimodal data alignment, comprising the following steps: The multimodal sensing and acquisition process involves simultaneously acquiring rock surface morphology, impact acoustic response, and impact vibration time-series data using three types of sensing devices: image, acoustic signature, and vibration. The cross-modal physical alignment and feature extraction steps establish a spatiotemporal constraint relationship for multimodal data based on the transmission law of rock mechanics, and extract image crack features, acoustic damage features, and vibration structure features respectively; The physical-driven multimodal fusion process constructs a fusion network based on rock fracture mechanics and the causal law of rock mass damage evolution, dynamically and adaptively allocates the fusion weights of each mode, and introduces a physical information neural network to calculate the stability quantification index characterizing the degree of rock mass loosening. The multimodal conflict consistency verification step combines evidence theory, rock mass damage time-series evolution logic, and spatial / temporal / depth three-dimensional consistency rules to jointly arbitrate the multimodal detection results and output a unified anomaly confidence level. The graded autonomous risk mitigation and disposal steps are based on the dual criteria of rock mass stability quantitative indicators and anomaly confidence level to classify risk levels, and the automatic control actuators complete the vibration removal of loose rocks or the relocation and inspection of equipment.

[0005] Preferably, the multimodal sensing and acquisition step specifically includes: the work machine equipped with a wire-controlled walking chassis travels to the detection point and completes the fixed support of the machine body; Control the retractable electric vibrating arm to reciprocate and impact the rock mass; simultaneously activate the night vision camera to collect images of the rock surface, the sound sensor to collect impact sound patterns, and the IEPE vibration sensor to collect vibration timing signals; Samples of various types of intact rock masses and rock masses containing fissures / cavities / loose pumice were collected to complete the model pre-training, and the vibration-acoustic physical transfer function was obtained through standard tapping calibration experiments.

[0006] Preferably, the cross-modal physical alignment and feature extraction step includes: using a convolutional neural network and U-Net to segment and extract visual features of crack density, length, and width from the image, and generating an image feature vector; Wavelet packet decomposition combined with principal component analysis is used to reduce the dimensionality of the voiceprint and extract the time-frequency attenuation features to generate a voiceprint feature vector. A long short-term memory network is used to extract the damping ratio and natural frequency offset time-series frequency domain features of the vibration signal to generate a vibration feature vector. The vibration-acoustic transfer function obtained by calibration is used to perform causal convolution time alignment. The theoretical acoustic text is predicted by the vibration signal and the deviation score between the theoretical acoustic text and the measured acoustic text is calculated. When the deviation score exceeds the preset threshold, it is determined that there are cavities or delamination anomalies inside the rock mass. Establish the spatial-frequency domain mapping relationship between the spatial distribution of cracks and the vibration propagation path, and complete the spatiotemporal alignment of three types of physical features.

[0007] Preferably, the physical-driven multimodal fusion step includes: constructing a causal cross-attention interaction network, establishing three pairs of modal physical coupling attention weights for vibration-voiceprint, voiceprint-image, and image-vibration respectively, and generating fusion features with mechanical consistency; Calculate the Mahalanobis distance anomaly deviation of images, acoustic signatures, and vibrations based on the baseline characteristics of the complete rock mass; Real-time acquisition of parameters such as illumination, dust, environmental noise, and mechanical interference; calculation of confidence scores for each mode through fuzzy inference; allocation of initial fusion weights based on confidence scores; locking the upper limit of the weight for a single mode when its confidence score is below a threshold; and correcting the mode weights by combining a secondary penalty based on the temporal causal consistency of rock mass damage. The stress intensity factor of image feature mapping, the internal damage index of acoustic feature mapping, and the structural integrity coefficient of vibration feature mapping are input into the physical information neural network. The network loss function is constructed using the rock fracture mechanics equation, and the normalized rock mass stability index (RSI) is output.

[0008] Preferably, the fixed time sequence of rock mass damage evolution is as follows: surface cracks appear first, followed by internal cavities, and finally structural loosening and abnormal vibration. The causal consistency is 1 only when the timing of the detected events conforms to this rule, and 0 when the timing is disordered. A forced attenuation penalty is applied to the fusion weight of the disordered mode.

[0009] Preferably, the multimodal conflict consistency verification step includes: The Dempster-Shafer evidence theory is used to fuse the anomaly probabilities of each mode and calculate the mode conflict coefficient; If the conflict coefficient is below the threshold, it is directly fused and judged; if the conflict coefficient is above the threshold, environmental interference is identified and weighted, or the equipment is controlled to change the tapping point for secondary collection. The three-fold verification of spatial consistency, temporal consistency, and depth consistency are performed sequentially: Spatial consistency calculates the intersection-union ratio (IoU) between the acoustic cavity projection area and the image crack area; if the IoU is lower than the threshold, the verification is deemed to have failed; Temporal consistency verifies whether the vibration anomaly peak is later than the crack detection time; if the vibration signal is earlier, it is determined to be external mechanical interference. Depth consistency comparison of stereo vision gap depth and voiceprint inversion hole depth; if the relative error exceeds the threshold, re-detection is triggered. The inconsistency penalty factor is calculated based on the triple verification results. The final anomaly confidence S is then calculated by combining the dynamically corrected modal weights and the local anomaly probabilities of each modality.

[0010] Preferably, the tiered autonomous risk mitigation and disposal steps are equipped with a dual risk assessment system: The basic risk level is divided into three levels based on the rock mass stability index (RSI): RSI < 0.3 is high risk, 0.3 ≤ RSI < 0.6 is medium risk, and RSI ≥ 0.6 is low risk. The abnormal confidence level S is used to classify the warning into four levels of fine-grained warning: S < 0.3 is normal level, 0.3 ≤ S < 0.6 is attention level, 0.6 ≤ S < 0.85 is warning level, and S ≥ 0.85 is danger level; When a medium-risk, high-risk, or dangerous level warning is triggered, the electric vibrating arm is driven to continuously impact and remove loose rocks, and a visual abnormality heat map, a timeline record of rock mass damage evolution, and a quantitative risk assessment basis are pushed to the human-machine terminal. When the risk level is determined to be low and normal, retract the vibrating arm and support legs, and control the entire machine to move to the next inspection station.

[0011] A smart floating rock removal system based on multimodal data alignment includes hardware operating tools and a layered software control system; The hardware operating equipment includes a heavy-duty drive-by-wire chassis module, a multimodal sensing and acquisition module, an electric vibration execution module, and an ESP drive control module. The hierarchical software control system includes a data acquisition module, an intelligent recognition module, and a human-computer interaction terminal; the data acquisition module is used for multimodal raw signal acquisition and basic feature extraction. The intelligent recognition module incorporates a physical alignment unit, a causal cross-attention fusion unit, a PINN rock mass stability calculation unit, a conflict and consistency verification unit, and a risk classification output unit. The human-computer interaction terminal is used to display risk data, visualize abnormal heat maps, and store and export records of rock mass hazards.

[0012] Preferably, the heavy-duty drive-by-wire chassis module includes a rear-drive front-steering wheel chassis, an ARM architecture VCU electronic control unit, a CAN communication bus, a quick-release lithium battery pack, a laser navigation and IMU obstacle avoidance unit, and a retractable electric outrigger for work stabilization at the rear of the chassis. The electric vibration actuator module includes a 360° rotating turntable, an L-shaped connecting base, a folding electric push rod, and a two-section sleeve-type electric vibration arm. The outer sleeve of the vibrating arm is made of titanium alloy, and the inner layer is a reciprocating impact rod. The matching servo motor provides an impact thrust of 200N to 400N and an impact frequency of 30 to 50 times / minute. The folding electric push rod can store the entire vibrating arm in the chassis.

[0013] Preferably, the multimodal sensing and acquisition module includes a 360° waterproof night vision high-definition camera, an array of sound sensors, and multiple sets of IEPE triaxial vibration sensors; The camera is equipped with a U-Net slit segmentation acceleration unit, and vibration sensors are respectively arranged at the bottom of the vibration arm and the chassis body; The intelligent recognition module is equipped with an AI acceleration unit, which has built-in causal convolutional network, LSTM, 1D-CNN, U-Net, and physical information neural network PINN. It stores a complete rock mass benchmark feature library, vibration-acoustic transfer function parameter library, and rock fracture mechanics constraint parameter set. The intelligent recognition module also integrates a fuzzy inference dynamic weight adjustment subunit, and simultaneously corrects the fusion weights of each mode based on environmental interference parameters and the causal consistency of rock mass damage.

[0014] Based on the above aspects, Attached Figure Description

[0015] Figure 1This is a data acquisition system diagram of an intelligent floating rock hazard removal method and system based on multimodal data alignment provided in an embodiment of the present invention; Figure 2 This is a diagram of an intelligent detection system for an intelligent floating rock hazard removal method and system based on multimodal data alignment provided in an embodiment of the present invention. Figure 3 This is a circuit control diagram of an intelligent floating rock hazard removal method and system based on multimodal data alignment provided in an embodiment of the present invention; Figure 4 This is a structural diagram of an intelligent floating rock hazard removal method and system based on multimodal data alignment provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an intelligent floating rock hazard removal method and system based on multimodal data alignment provided in an embodiment of the present invention; Figure 6 This is the workflow of an intelligent floating rock hazard removal method and system based on multimodal data alignment provided in this embodiment of the invention; Figure 7 This is a flowchart of a smart floating rock hazard removal method and system based on multimodal data alignment provided by an embodiment of the present invention.

[0016] Numbering in the diagram: 1-Wheel chassis, 2-Electric vibrating arm (section 1 201, section 2 202), 3-Electric inclined outrigger, 4-Searchlight, 5-Night vision camera, 6-Vibration sensor, 7-Sound sensor, 8-ESP control board (including 801 ESP mainboard, 802 push rod servo motor, 803 walking servo motor), 9-Lithium battery, 11-Heavy-duty wire-controlled electric control system for chassis, 12-Laser navigation system, 13-Turntable structure, 14-L-shaped connecting base, 15-Folding electric push rod.

[0017] The system comprises a control system 100, a data acquisition module 110, an intelligent recognition module 120, a video data acquisition module 111, an audio data acquisition module 112, a vibration data acquisition module 113, a core hardware module 121, an AI acceleration module 122, and an operating system and software module 123. Detailed Implementation

[0018] The present invention will now be described in detail with reference to the accompanying drawings.

[0019] The simple machine has mechanical learning capabilities and can perform image recognition, voiceprint recognition, and vibration frequency recognition to collaboratively determine deep fractures and fissures in the rock mass. Under the control of the ESP intelligent platform, it can automatically vibrate and remove loose rocks, so that cracks can be accurately detected and loose rocks can be removed when the simple machine is in motion.

[0020] The video data acquisition module 111 of the simple machine: extracts visual features of rock surface and cracks from image information through convolutional neural network, and uses a trained model as the base network to extract features from the input rock image.

[0021] The video data acquisition module uses an enhanced rock sample dataset to fine-tune the model, improving its adaptability to complex environments. The base model outputs an m-dimensional feature vector to characterize rock surface morphology indicators, such as crack density and edge sharpness.

[0022] The audio data acquisition module extracts the time-frequency features of the voiceprint information, uses principal component analysis to reduce the dimensionality of the high-dimensional voiceprint data, retains key frequency band information that reflects the internal integrity of the rock, and generates an n-dimensional feature vector.

[0023] The vibration data acquisition module extracts frequency domain features from vibration information, combines long short-term memory networks to process the temporal characteristics of vibration signals, and outputs a k-dimensional feature vector.

[0024] When integrating three types of feature vectors, this invention breaks through the existing three-step underlying paradigm of feature concatenation, statistical classifier, and threshold judgment, and proposes a five-level physical causal-driven deep fusion mechanism. Each level has made explicit improvements to the underlying logic of the existing model: Underlying alignment logic: The alignment is changed from data-driven blind alignment to alignment constrained by physical causal transfer functions. Existing technologies directly align feature vectors of different modes on the time axis according to the sampling rate (data-driven blind alignment), ignoring the physical time delay and transmission rules between the propagation of vibration signals and the generation of acoustic responses.

[0025] First, this invention establishes a cross-modal physical causal transmission model: vibration signals are transmitted through the rock mass structure to generate acoustic responses, and their transfer function is constrained by the geometric morphology and material properties of rock mass cracks; simultaneously, the spatial distribution of cracks in the image directly determines the propagation path and attenuation mode of the vibration wave. Based on this physical causal relationship, a causal convolutional network is used to causally align the vibration signal and acoustic signal, ensuring that vibration features strictly precede acoustic features in time sequence; for image crack features and vibration frequency domain features, a spatial-frequency domain mapping table based on structural mechanical response is established to achieve physically consistent spatiotemporal correlation.

[0026] The key difference between this invention and existing technologies lies in the following: a physical transfer function H(f) = P(f) / A(f) between vibration acceleration signal a(t) and sound pressure signal p(t) is established through on-site calibration experiments, and this function is embedded as a hard constraint in the alignment preprocessing layer; when the residual between the measured acoustic pattern and the acoustic pattern predicted by vibration via H(f) exceeds the threshold of 0.3, the system determines that there are abnormal structures such as cavities or delamination inside the rock mass, and this physical residual directly participates in the subsequent RSI calculation—this judgment basis is completely different from the alignment method in existing technologies that only rely on statistical feature similarity.

[0027] Existing technologies directly concatenate or weight the feature vectors of each modality, treating each modality as an independent channel during fusion, thus failing to capture the physical coupling and interdependence between modalities.

[0028] This invention constructs a three-modal feature interaction network: in the feature fusion layer, a causal cross-attention mechanism is employed. Using vibration features as the query and voiceprint features as the key and value, the vibration-voiceprint causal attention weight is calculated to capture the physical coupling relationship between vibration excitation and voiceprint response. Using voiceprint features as the query and image crack features as the key and value, the voiceprint-image attention weight is calculated to reflect the correlation between "internal damage and surface morphology". Using image crack features as the query and vibration features as the key and value, the image-vibration attention weight is calculated to characterize the mapping between "surface cracks and structural dynamic response".

[0029] Through multi-layered causal cross-attention interactions, a physically consistent fusion feature representation is generated. This design ensures that the association weights between each pair of modalities are determined by physical causality, rather than being freely determined by statistical learning, fundamentally different from blind fusion techniques in the past.

[0030] Existing classifiers are trained using statistical loss functions such as cross-entropy. The output classification probabilities only reflect the statistical distribution of training samples and have no correspondence with the rock fracture mechanics mechanism. Furthermore, the judgment indicators (such as classification probability values) lack physical interpretability. This invention further introduces a Physics-Informed Neural Network (PINN) to construct a Rock Stability Index (RSI) calculation model. The three-modal fusion features are mapped to fracture mechanics physical parameters: crack length, width, and density extracted from image features are mapped to the stress intensity factor K; the spectral attenuation slope and resonance peak shift extracted from acoustic features are mapped to the internal damage index η and the fracture development index; and the damping ratio change and natural frequency shift extracted from vibration features are mapped to the structural integrity coefficient ζ. The PINN model uses rock fracture mechanics equations (such as the relationship between stress intensity factor and crack propagation rate) as physical constraints, fusing the above physical parameters into a normalized rock stability index RSI∈0,1, where the closer the RSI is to 0, the more unstable the rock mass.

[0031] Compared with the pure statistical classifiers in the prior art, this invention embeds the rock fracture mechanics equation into the neural network loss function, namely: L = L_data + λ × L_physics, where L_physics is the residual penalty term of the mechanics equation. This enables the model to make informed judgments based on physical laws even in extreme working conditions where training samples are lacking, rather than relying on extrapolation from data distribution. This is the fundamental difference from the underlying training logic of the classifiers in the prior art.

[0032] Existing technologies use preset fixed weights (e.g., image 0.4, voiceprint 0.3, vibration 0.3) for three-modal fusion. However, these weights cannot adaptively adjust in complex underground environments with interference (dust obstruction, water accumulation, mechanical vibration noise), resulting in insufficient robustness (anti-interference capability). In contrast, this invention employs a dynamic adaptive weighting mechanism: real-time monitoring of the data quality for each modality, including image signal-to-noise ratio (SNR) (affected by dust / light), voiceprint SNR (affected by environmental noise), and vibration SNR (affected by mechanical interference). Based on the real-time SNR and historical prediction confidence of each modality, a fuzzy inference system dynamically adjusts the contribution weights of the three modalities in the RSI calculation. For example, in high-dust, low-light environments, the image modality weight is automatically reduced while the voiceprint and vibration modality weights are increased; in environments with high mechanical vibration interference, the vibration modality weight is reduced while the image and voiceprint modality weights are increased, ensuring the robustness of the fusion judgment.

[0033] The fundamental difference from the fixed weights in existing technologies lies in the fact that this invention divides the adjustment of weights into two dimensions: the first dimension is the modal confidence C of environmental perception (calculated in real time based on environmental parameters such as signal-to-noise ratio, illumination, and dust); the second dimension is the causal consistency C based on the physical evolution chain of damage (see 0017). When a temporal event of a certain modality violates the physical causal chain, even if its confidence is high, its weight is forcibly penalized to the upper limit of 20%, thereby avoiding false alarms of "high-quality noise signals disguised as anomalies". This dual-dimensional dynamic weight has not been applied in existing technologies.

[0034] Existing technologies compare the output probability of a neural network classifier with a fixed threshold to obtain a binary classification result of "dangerous" or "safe," failing to distinguish between "real rock mass anomalies" and "multimodal conflict noise," and the output lacks physically interpretable evidence. This invention establishes a multimodal inconsistency detection and arbitration mechanism: when a single modality judgment result conflicts (e.g., image recognition indicates high risk while soundprint / vibration recognition indicates low risk), the Dempster-Shafer evidence theory fusion framework is activated to calculate the basic probability allocation function and conflict coefficient for each modality. If the conflict coefficient is below the threshold, a fusion judgment is obtained through evidence synthesis rules; if the conflict coefficient is above the threshold, it indicates the presence of abnormal interference or an unknown damage mode, and the system marks this location as requiring secondary verification, controlling the equipment to adjust the detection angle or change the impact point for re-collection, thus avoiding misjudgment by a single modality. Furthermore, based on DS fusion, a temporal causal verification based on the damage physical evolution chain (cracks, cavities, loosening, vibration anomalies) is superimposed: only when the temporal sequence of the detection events in the three modes strictly conforms to the physical laws of rock mechanics fracture is the anomaly confidence S assigned a high value; otherwise, even if the local anomaly probability of each mode is high, the final confidence S will be suppressed by the inconsistency penalty factor δ. This mechanism enables "high-resolution anomaly signals" from external interferences such as mechanical vibration and wind noise to be identified and filtered by the physical evolution chain verification, which is significantly different from the fixed threshold judgment logic of existing technologies.

[0035] In the physical causal transmission model, the vibration-acoustic transfer function H(f) is determined through field calibration experiments: under the conditions of known intact rock mass and standard impact force, the vibration acceleration signal a(t) and sound pressure signal p(t) are recorded, and H(f) is estimated through a system identification method. ; in: The frequency domain representation (Fourier transform) of the sound pressure signal; This represents the frequency domain of the vibration acceleration signal. For frequency variables.

[0036] In normal, intact rock masses, there is a stable correspondence between vibration signals and acoustic signatures (i.e., the reference transfer function H(f)). During actual testing, the measured vibration characteristics are used to predict acoustic signature characteristics using H(f), and the residual between the predicted and measured acoustic signatures is calculated. If the residual exceeds a preset threshold, it indicates the presence of abnormal structures (such as cavities or delamination) within the rock mass. This residual serves as an auxiliary indicator for anomaly detection in RSI calculation.

[0037] The predicted voiceprint is subtracted from the actual voiceprint point by point, and the absolute values ​​are averaged to obtain a deviation score: ; If the score exceeds the threshold of 0.3, it indicates that the actual sound pattern deviates significantly from the expected one. This is a typical manifestation when there are cavities, fissures, or delaminations inside the rock mass, because abnormal structures will change the propagation path of sound waves, causing the sound pattern to lose its normal regularity.

[0038] The calculation of the rock mass stability index (RSI) follows the following physical-data fusion logic: ; in: This is the baseline stability function (neural network approximation). The stress intensity factor (calculated from the geometric features of the crack in the image); The internal damage index (estimated by acoustic spectrum attenuation characteristics); The structural integrity coefficient (extracted from vibration mode parameters); For dynamic adaptive weights ( ); (1) Stress intensity factor ; ; (2) Internal damage index

[0039] ; (3) Structural integrity coefficient ; ; Change in damping ratio offset from natural frequency Together, these factors determine the degree of degradation of the rock mass structure relative to its intact state. When the rock mass is struck, the accelerometer records the vibration decay rate (damping ratio) and the dominant frequency. Compared to an intact rock mass benchmark, if the damping ratio increases significantly and the dominant frequency shifts significantly, it indicates a decrease in rock mass stiffness and damage to the internal structure. It will be too low.

[0040] Physical consistency constraints: ; Only when the image determines the degree of crack expansion ( The degree of internal damage is determined by voiceprint. Additional penalties are only triggered when both indicators exceed their respective thresholds. If only one indicator exceeds the limit, there is no penalty; both must exceed the limit simultaneously to indicate that both independent sensing channels have confirmed an anomaly, at which point the system increases the risk score.

[0041] When stress intensity factor And internal damage index When the physical constraint term is greater than zero, a penalty is triggered. That is, when two physical indicators exceed the limit at the same time, the system applies an additional constraint correction to prevent the model from making overly optimistic judgments in high-risk areas.

[0042] Comprehensive anomaly correction items: ; Constraint: Subtract the weighted sum of the deviations of the three sensors from 1 to obtain a "normality score". The greater the deviation of the three sensors, the lower the score, and the lower the final rock mass stability index (RSI). Anomaly Correction Term (If the calculation result exceeds this range, it will be forcibly truncated to the boundary value).

[0043] The RSI ranges and risk levels correspond as follows: ; Anomaly deviation calculation, i.e., Mahalanobis distance: ; in: The feature vector of the sample to be tested (image / voiceprint / vibration); This is the baseline characteristic vector of normal rock mass; The inverse of the covariance matrix (simplified to the identity matrix in the code). ) Three-modal anomaly deviation: ; ; ; →The sample is close to the normal baseline and shows no significant abnormalities; Increased deviation indicates a greater degree of abnormality compared to the normal baseline. Imagine the characteristics of a normal rock mass as a "standard template." The Mahalanobis distance is a measure of how far the current detection result differs from this template; a larger difference indicates a more abnormal rock mass.

[0044] The dynamic adaptive weight adjustment adopts fuzzy inference rules based on modal confidence: Image confidence level (affected by dust and lighting): ; in: ; ; ; Voiceprint confidence level (affected by ambient noise): ; Vibration confidence level (affected by mechanical disturbance): ; Under normal circumstances, the three weights are allocated according to the confidence level: ; constraint: ; When any confidence level falls below 0.3, that sensor enters a degraded state, its fusion weight cap is locked at 0.2, and the remaining weights are redistributed according to the confidence levels of the other two sensors.

[0045] ; In the multimodal inconsistency detection, a three-modal anomaly basic probability allocation function is defined: Each sensor first independently provides an "anomaly probability": ; ; ; Accordingly: ,in It serves as the adaptive judgment threshold for each sensor, thus "adjusting sensitivity".

[0046] Low conflict situation (conflict coefficient) The three sensors make largely consistent judgments, and the Dempster combination rule is used to combine and calculate the results. ; High conflict situation (conflict coefficient) If the three sensors make conflicting judgments, the following strategy is executed: 1. Check if there is any known interference (mechanical vibration, airflow noise); 2. If there is known interference, reduce the weight of the affected sensor and recalculate; 3. If there is no known interference, mark the location as "unknown abnormal mode" and upgrade it to a high-risk level, and record the conflict sample for subsequent model learning.

[0047] Based on the aforementioned cross-modal fusion judgment, a dynamic assessment mechanism for modal credibility and a cross-modal consistency verification mechanism based on the damage physical evolution chain are further introduced. This mechanism uses physical causal constraints to collaboratively verify and correct the final anomaly confidence level. The specific implementation is as follows: Based on prior knowledge of rock mechanics, the evolution of rock mass damage typically follows this physical sequence: surface crack appearance → internal cavity expansion → structural loosening → abnormal vibration. This invention mandates that the time series of the three-modal detection events conform to this physical causal chain; otherwise, it is judged as sensor noise or a false alarm. Specifically, the detection results of each modality are transformed into a discrete event sequence with a time order: ; in: For the event timestamp (in seconds), For event type, The feature vectors are (image U-Net segmentation results, acoustic wavelet packet energy, and vibration frequency domain features). Traditional methods directly concatenate three-modal feature vectors for static fusion, ignoring the physical temporal patterns of rock damage. This invention transforms the detection results into a time-ordered discrete event sequence, forcing subsequent causal chain verification.

[0048] ; in It is a d-dimensional eigenvector.

[0049] The image modality outputs crack events (using U-Net segmentation to extract crack width and length directions), the acoustic modality extracts energy for each frequency band through wavelet packet decomposition and then inputs it into a 1D-CNN to output hole events, and the vibration modality calculates the time-domain peak value and frequency-domain dominant frequency offset using a triaxial accelerometer to output loosening events. After aligning the three modal data by timestamps, a multimodal event sequence E is constructed within a time window ΔT.

[0050] Based on rock mechanics fracture theory, surface cracks are the macroscopic manifestation of internal damage. Crack propagation leads to the formation of internal cavities, and the further propagation of these cavities causes structural loosening. This is an irreversible physical causal chain.

[0051] Therefore, a damage evolution coefficient can be defined: ; ; ; The prior probability that reflects the evolution of damage The closer the value is to 1, the more reliable the physical causal chain. If this occurs, it indicates that there are many violations of the physical chain in the sample, and recalibration is required.

[0052] Causal consistency C: ; ; ; This is a hard constraint criterion. If loosening is detected before cracking (e.g., mechanical vibration interference), C=0 is directly determined, and the modal weight is penalized. This differs from traditional soft constraints (e.g., reducing weight) and avoids the false alarm chain of "vibration noise → misjudgment as loosening → triggering hazard mitigation".

[0053] ; ; ; Traditional fuzzy inference only considers environmental noise (dust, light), while this invention adds physical causal consistency as a second dimension. When C=0, even if a certain mode has a high confidence level (such as a vibration sensor detecting a strong signal), its weight is penalized to 20% as long as its timing violates the physical chain.

[0054] Design three paired consistency checkers to perform adversarial verification of the multimodal detection results within each time window: Spatial consistency check: Project the acoustic hole probability map onto the image coordinate system and calculate the intersection-over-union (IoU) between the image gap region and the acoustic hole location. If the IoU is lower than a preset threshold, the acoustic modality confidence is automatically reduced, triggering the downgrade weighting mechanism in 0015.

[0055] ; ; ; The location of the cavity detected by the acoustic signature should be projected onto the image coordinate system and spatially overlap with the crack area in the image. If the cavity location is far from the crack (IoU << 0.3), it indicates that the acoustic signature is detecting other interference sources (such as wind tunnels or water flow), rather than the internal damage corresponding to the crack.

[0056] Time consistency check: This checks the chronological relationship between the vibration anomaly time point and the appearance time of the crack / void. If the vibration peak occurs before the crack detection in time, the vibration signal is determined to be an external interference, the corresponding vibration mode weight is reset to zero, and the basic probability allocation function is adjusted accordingly in the DS conflict detection.

[0057] ; Abnormal vibration is a result of structural loosening, which is caused by cracks / voids. If the vibration peak occurs before the cracks appear ( It cannot be due to loosening of the surrounding rock structure; it must be due to mechanical interference or environmental vibration.

[0058] ; in For indicator functions, This represents the minimum physical delay (stress wave propagation time).

[0059] Depth consistency check: The crack depth is estimated from stereo vision and compared with the cavity depth obtained from acoustic inversion. If the depth error between the two exceeds 30%, a re-inspection flag is triggered, and the equipment is controlled to adjust the tapping angle and re-acquire data to avoid misjudgment caused by depth estimation deviation.

[0060] ; ; Where B is the reference distance and f is the focal length. For parallax.

[0061] ; in The sound velocity of the rock mass. This is the transmit-receive time difference.

[0062] ; ; This is the average failure rate of the three-dimensional consistency test. When all three tests fail... (Maximum penalty) At this point, even if the local anomaly probability of each modality is high, the final confidence S is suppressed to 0.

[0063] Physical meaning: Reflects the "degree of multimodal contradiction". High This means that the system should make a conservative judgment regarding sensor noise, environmental interference, or unknown damage modes.

[0064] Modal dynamic weights constrained by the physical evolution chain Integrating the results of the three consistency tests above, the final anomaly confidence level S is calculated: ; ; ; This is a dual-gating mechanism: the first layer of gating: dynamic weighting. Filtering low-quality modes; Second gating: Inconsistency penalty Filter out multimodal contradictions.

[0065] in For each modality, the local anomaly probability is... The inconsistency penalty factor is determined by the number of times the consistency test fails (each failure is penalized proportionally), ensuring that only anomalies that pass the consistency test can generate high-confidence warnings. Used in conjunction with RSI in 0014: RSI reflects the quantitative stability of fracture mechanics physical parameters. It reflects the integrity and consistency of the multimodal temporal causal chain, and the two together determine the final risk level and disposal instructions.

[0066] Based on the S-value, a refined four-level early warning judgment is output: ; This four-level early warning system complements the three-level classification system based on RSI. The former focuses on the integrity of the physical evolution chain, while the latter focuses on the quantification of fracture mechanics parameters. Together, they constitute a dual-redundant safety decision mechanism.

[0067] While outputting the final warning level, the system automatically generates an interpretable report, which includes: (1) an abnormal heat map, with the probability distribution of acoustic voids and the vibration intensity distribution superimposed on the image; (2) a physical evolution chain trigger record, indicating the detection timestamp and sequence of each event; and (3) a description of the main criteria, such as "the image detected a crack area A, an acoustic void was found 0.2m below the corresponding area, the vibration peak increased by 30% within 0.5s after the void appeared, which satisfies the physical evolution chain, and the IoU=0.78 passes the spatial consistency test, and is judged to be a dangerous pumice." This interpretable output makes it easier for operators to understand the warning basis, supports accurate on-site handling decisions, and provides high-quality labeled samples for subsequent incremental model learning.

[0068] Triggering text template: “The image detected a crack region A (width {w} mm, depth {d_image} m). A hole was found in the voiceprint at a position {d_offset}m below the corresponding area (confidence level {p_voiceprint}%). The peak vibration increases by {rise}% (dominant frequency shift {df}Hz) after the appearance of the cavity. The physical evolutionary chain is satisfied: crack → cavity → loosening. Spatial IoU = {iou}, Depth Error = {Depth Error Rate}%. This is classified as a level {level} pumice; action {recommendation} is recommended. Anomaly heatmap matrix: ; in For U-Net segmentation mask, This is a probability map for voiceprint inversion. This is a vibration intensity diagram.

[0069] The simple machine has a heavy-duty drive-by-wire chassis 1. The overall chassis structure is rear-drive and front-steering. It is powered by two high-power brushless DC servo motors, and the steering is powered by a steering motor.

[0070] The simple machine is powered by a lithium battery (2-10kWh), which is installed on the chassis and has a range of 8 hours. It can be quickly detached.

[0071] The chassis's heavy-duty drive-by-wire electronic control system, 11VCU, is based on an ARM core control board, enabling precise control of chassis speed and steering angle, and providing real-time feedback on linear speed and steering angle. The entire chassis utilizes CAN communication for drive and control, allowing for autonomous driving and intelligent obstacle avoidance.

[0072] The chassis is equipped with a laser navigation system 12, which includes: a navigation algorithm motherboard (software), an array obstacle avoidance laser IMU module, and a 16-line laser radar.

[0073] The simple machine chassis is equipped with a retractable electric inclined leg 3 at the rear. Under vibration conditions, the retractable electric inclined leg extends to overcome the recoil force from the vibrating arm and maintain the stability of the chassis when identifying and mitigating vibration hazards.

[0074] A waterproof searchlight 4 and a 360-degree rotating, waterproof, high-resolution camera 5 with night vision function are installed at the front of the small chassis to detect cracks on the surface of the rock mass (limestone patterns or 2-10mm cracks).

[0075] First, the high-pixel camera device 5 is used, and a control motherboard is configured. The high-pixel camera captures the crack image and marks the location of the hidden danger. The image information is then transmitted to the video data acquisition module 111 for judgment. A retractable electric vibrating arm 2 (2-3m long) is mounted on the small chassis. It has a two-section sleeve structure. The outer sleeve of the first section 201 does not participate in the vibration and is made of titanium alloy tubing (density 4.51g / cm³). The second section 202 is an electric metal rod that can move up and down linearly and rapidly. The second section is fitted onto the support plate inside the cavity at the top of the first section. The electric actuator is connected to the electric actuator controller and is connected to the main ESP board 801, which can realize remote control.

[0076] The metal rod vibrating arm 2 uses a servo motor or a reluctance motor 802 with a power of 55W or more and a voltage of 24V. It can quickly extend and retract 30 to 50 times per minute, with a thrust of about 200N (20kg) to 400N, within the range of 1 to 10MPa.

[0077] A turntable structure 14 is installed between the electric vibrating arm 2 and the chassis 1, which allows the vibrating arm 2 to rotate 360 ​​degrees. The first outer sleeve 201 is installed at an angle of 20° to 90° with the ground plane. There is an "L"-shaped connecting base 15 between the bottom of the sleeve and the turntable. The "L"-shaped connecting base 15 and the first outer sleeve 201 are connected by a movable pin.

[0078] The "L"-shaped connecting base 15 and the lower part of the first outer sleeve 201 are provided with a folding electric push rod 16. When the electric push rod reaches its maximum stroke, the first outer sleeve 201 rises to its highest position. When the electric push rod 16 reaches its minimum stroke, the first outer sleeve 201 falls back and folds onto the chassis 1.

[0079] On-site, traditional methods were used to tap the roof or sidewalls of the surrounding rock. Acoustic fingerprint data acquisition cards GX8003 or TLV320ADC3101 were used to record the acoustic fingerprint characteristics of areas with delamination, interlayer cracks, or voids. This data was then transmitted to the audio data acquisition module 112 for fine-tuning and training of the system model. The training dataset should include various typical "cross-modal contradiction" samples (i.e., scenarios where one or two modes are abnormal while the others are normal) to enhance the intermodal consistency analysis submodule's ability to identify hidden cracks.

[0080] After the simple equipment is in place, when the electric vibrating arm 2 reciprocates and impacts the surrounding rock, the "hollow" or "solid" impact sound is transmitted to the sound sensor 18 through the microphone, and the vibration information is transmitted to the intelligent recognition module 120. After the core hardware module 121 and the AI ​​acceleration module 122 complete the cross-modal cross-validation anomaly scoring, they send an electrical signal to the operating system and software module 123 for further processing.

[0081] On-site, traditional methods were used to strike the roof or sidewalls of the surrounding rock. IEPE vibration sensors 19 and a high-precision vibration signal data acquisition card recorded the "large" and "small" vibration frequency characteristics of fractures that were likely to cause damage. These were converted into data signals and transmitted to the vibration data acquisition module 113 for fine-tuning and training of the judgment system. During training, physical coupling relationship labels corresponding to each frequency characteristic were simultaneously labeled, enabling the vibration sub-network to learn feature expressions consistent with the physical coupling of acoustic and image modalities.

[0082] The bottom of the second push-pull rod 202 of the vibrating arm 2 is equipped with a vibration sensor 19 receiver port. The vibration sensor 19 is installed on the chassis 1. When the vibrating arm 2 reciprocates and impacts the surrounding rock, the core hardware module 121 and the AI ​​acceleration module 122 complete the cross-modal cross-verification anomaly scoring, output the progressive graded anomaly score S and the contribution ratio of each mode, analyze and judge whether there are any faults or fissures in the deep rock mass that can cause harm, transmit the location and risk level of the hidden danger, and send an electrical signal for further handling to the operating system and software module 123.

[0083] The chassis is equipped with an ESP execution system 8 (including ESP main board 801, push rod servo motor 802, and walking servo motor 803). After the intelligent identification module 120 receives the identification signal of the hidden danger location and the risk level ("warning" or "danger"), it issues an action command, which is transmitted to the vibrating arm 2 through the ESP execution system 8.

[0084] Furthermore, this invention also mentions a simple tool for identifying and removing floating rocks based on intelligent identification. This method employs an intelligent detection and processing device for floating rocks in surrounding rock, as described above, and includes the following steps: Step 1: The data acquisition module 110 creates datasets of acoustic signatures, images, and vibration frequencies. On-site, the top or side of the surrounding rock is struck using traditional methods to collect images, acoustic signatures, and vibration signals of normal, intact rock masses and rock masses with fractures. Multimodal response characteristics are recorded under various fracture scenarios (including surface cracks, internal cavities, structural loosening, etc.). Vibration acceleration and sound pressure signals are measured under intact rock mass conditions through on-site calibration experiments. A vibration-acoustic signature reference transfer function H(f) is established, and the model is fine-tuned and trained.

[0085] Step 2: The heavy-duty wire-controlled electric control system 11 and the laser navigation system 12 control the walking servo motor 803 to move the chassis 1 to the work position, open and fix the electric inclined support leg 3 at the rear of the chassis to support the ground, start the turntable structure 14 and the folding electric push rod 16 to move the first section 201 of the vibrating arm 2 to the work position, and start the push rod 202 to impact the rock.

[0086] Step 3: The core hardware module 121 collects images of the rock surface and vibration and sound data of the surrounding rock under impact through the high-definition camera 5, vibration sensor 19, and sound sensor 18. The intelligent recognition module 120 executes the following judgment process: S3-1: Data Preprocessing and Physical Causal Alignment. Crack detection and geometric feature extraction are performed on images; spectral analysis and attenuation feature extraction are performed on acoustic signatures; and modal parameter identification and frequency domain feature extraction are performed on vibrations. Based on a pre-calibrated vibration-acoustic signature physical transfer function and a crack-vibration spatial mapping table, cross-modal causal alignment is performed to establish a spatiotemporal correlated dataset.

[0087] S3-2: Trimodal Feature Interaction and Anomaly Deviation Calculation. Based on the physical coupling relationship between vibration excitation generating acoustic response, internal damage affecting surface morphology, and surface cracks affecting structural dynamic response, a pairwise cross-correlation analysis is performed on the three modal features to capture the physical dependencies between modes and generate a fused feature representation. Based on the normal baseline features of the intact rock mass, the image anomaly deviation D_image (crack geometric parameter deviation), acoustic anomaly deviation D_acoustic (spectral attenuation slope deviation), and vibration anomaly deviation D_vibration (modal parameter offset) are calculated respectively to quantify the degree of deviation of each mode relative to the normal rock mass.

[0088] S3-3: Dynamic Weight Allocation and RSI Calculation. Real-time evaluation of data quality and confidence levels for each modality is performed, and dynamic adaptive weights α, β, and γ are calculated using a fuzzy inference system. The three-modal anomaly deviation and fracture mechanics physical parameters (stress intensity factor K, internal damage index η, and structural integrity coefficient ζ) are input into a physical information neural network to calculate the rock mass stability index (RSI).

[0089] S3-4: Multimodal Inconsistency Arbitration and Decision-Making. Based on the anomaly probabilities of each modality, DS evidence fusion is performed to calculate the conflict coefficient. If the conflict coefficient is below a threshold, risk classification is based primarily on RSI: RSI < 0.3 is "High Risk - Immediate Action," 0.3 ≤ RSI < 0.6 is "Medium Risk - Key Monitoring," and RSI ≥ 0.6 is "Low Risk - Normal Passage." If the conflict coefficient is above the threshold, a conflict resolution strategy is implemented: the affected modality is identified and suppressed, or marked as "Requires Secondary Verification," and the equipment is adjusted to re-collect data. Based on this, the damage physical evolution chain verification and three-category cross-modal consistency checks in section 0017 are further performed to calculate the final anomaly confidence S. Combined with RSI, a four-level refined early warning system (Normal / Attention / Warning / Dangerous) is output, and an interpretable anomaly heatmap and physical evolution chain trigger record are generated for operators to use for precise handling.

[0090] S3-5: Based on the hierarchical decision in S3-4, the operating system and software module 123 generate corresponding control instructions and proceed to step 4.

[0091] Step 4: Based on the graded decision in Step 3 (S3-4), the operating system and software module 123 issue instructions to the ESP control center 801. If RSI < 0.6 ("medium risk" or "high risk" level), the top rod servo motor 802 is immediately started to activate the electric vibrating arm 2, generating impact vibration to treat the loose stones on the top plate. At the same time, the RSI value, risk level, and contribution ratio of each mode are pushed to the operator interface. If RSI ≥ 0.6 ("low risk" level), the vibrating arm 2 is moved to the next work station via the rotary table structure 14, or the electric inclined support leg 3 of the chassis is retracted, and the vibrating arm 2 and the walking servo motor 803 move the chassis 1 to the next work station.

[0092] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

[0093] The embodiments of this application have been described above with reference to the accompanying drawings. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. This application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A smart floating rock hazard mitigation method based on multimodal data alignment, characterized in that: Includes the following steps: The multimodal sensing and acquisition process involves simultaneously acquiring rock surface morphology, impact acoustic response, and impact vibration time-series data using three types of sensing devices: image, acoustic signature, and vibration. The cross-modal physical alignment and feature extraction steps establish a spatiotemporal constraint relationship for multimodal data based on the transmission law of rock mechanics, and extract image crack features, acoustic damage features, and vibration structure features respectively; The physical-driven multimodal fusion process constructs a fusion network based on rock fracture mechanics and the causal law of rock mass damage evolution, dynamically and adaptively allocates the fusion weights of each mode, and introduces a physical information neural network to calculate the stability quantification index characterizing the degree of rock mass loosening. The multimodal conflict consistency verification step combines evidence theory, rock mass damage time-series evolution logic, and spatial / temporal / depth three-dimensional consistency rules to jointly arbitrate the multimodal detection results and output a unified anomaly confidence level. The graded autonomous risk mitigation and disposal steps are based on the dual criteria of rock mass stability quantitative indicators and anomaly confidence level to classify risk levels, and the automatic control actuators complete the vibration removal of loose rocks or the relocation and inspection of equipment.

2. The intelligent floating rock hazard removal method based on multimodal data alignment according to claim 1, characterized in that: The multimodal sensing and acquisition steps specifically include: the work machine equipped with a wire-controlled walking chassis travels to the detection point and completes the fixed support of the machine body; Control the retractable electric vibrating arm to reciprocate and impact the rock mass; simultaneously activate the night vision camera to collect images of the rock surface, the sound sensor to collect impact sound patterns, and the IEPE vibration sensor to collect vibration timing signals. Samples of various types of intact rock masses and rock masses containing fissures / cavities / loose pumice were collected to complete the model pre-training, and the vibration-acoustic physical transfer function was obtained through standard tapping calibration experiments.

3. The intelligent floating rock hazard removal method based on multimodal data alignment according to claim 2, characterized in that: The cross-modal physical alignment and feature extraction steps include: using a convolutional neural network and U-Net to segment and extract visual features of crack density, length, and width from the image, and generating an image feature vector; Wavelet packet decomposition combined with principal component analysis is used to reduce the dimensionality of the voiceprint and extract the time-frequency attenuation features to generate a voiceprint feature vector. A long short-term memory network is used to extract the damping ratio and natural frequency offset time-series frequency domain features of the vibration signal to generate a vibration feature vector. The vibration-acoustic transfer function obtained by calibration is used to perform causal convolution time alignment. The theoretical acoustic text is predicted by the vibration signal and the deviation score between the theoretical acoustic text and the measured acoustic text is calculated. When the deviation score exceeds the preset threshold, it is determined that there are cavities or delamination anomalies inside the rock mass. Establish the spatial-frequency domain mapping relationship between the spatial distribution of cracks and the vibration propagation path, and complete the spatiotemporal alignment of three types of physical features.

4. The intelligent floating rock hazard removal method based on multimodal data alignment according to claim 1, characterized in that: The physical-driven multimodal fusion step includes: constructing a causal cross-attention interaction network, establishing three pairs of modal physical coupling attention weights for vibration-voiceprint, voiceprint-image, and image-vibration respectively, and generating fusion features with mechanical consistency; Calculate the Mahalanobis distance anomaly deviation of images, acoustic signatures, and vibrations based on the baseline characteristics of the complete rock mass; Real-time acquisition of parameters such as illumination, dust, environmental noise, and mechanical interference; calculation of confidence scores for each mode through fuzzy inference; allocation of initial fusion weights based on confidence scores; locking the upper limit of the weight for a single mode when its confidence score is below a threshold; and correcting the mode weights by combining a secondary penalty based on the temporal causal consistency of rock mass damage. The stress intensity factor of image feature mapping, the internal damage index of acoustic feature mapping, and the structural integrity coefficient of vibration feature mapping are input into the physical information neural network. The network loss function is constructed using the rock fracture mechanics equation, and the normalized rock mass stability index (RSI) is output.

5. The intelligent floating rock hazard removal method based on multimodal data alignment according to claim 4, characterized in that: The fixed time sequence of rock mass damage evolution is as follows: surface cracks appear first, followed by internal cavities, and finally structural loosening and abnormal vibration. The causal consistency is 1 only when the timing of the detected events conforms to this rule, and 0 when the timing is disordered. A forced attenuation penalty is applied to the fusion weight of the disordered mode.

6. The intelligent floating rock hazard removal method based on multimodal data alignment according to claim 1, characterized in that: The multimodal conflict consistency verification steps include: The Dempster-Shafer evidence theory is used to fuse the anomaly probabilities of each mode and calculate the mode conflict coefficient; If the conflict coefficient is below the threshold, it is directly fused and judged; if the conflict coefficient is above the threshold, environmental interference is identified and weighted down, or the control equipment is changed to a second collection point. The three-fold verification of spatial consistency, temporal consistency, and depth consistency are performed sequentially: Spatial consistency calculates the intersection-union ratio (IoU) between the acoustic cavity projection area and the image crack area; if the IoU is lower than the threshold, the verification is deemed to have failed; Temporal consistency verifies whether the vibration anomaly peak is later than the crack detection time; if the vibration signal is earlier, it is determined to be external mechanical interference. Depth consistency comparison of stereo vision gap depth and voiceprint inversion hole depth; if the relative error exceeds the threshold, re-detection is triggered. The inconsistency penalty factor is calculated based on the triple verification results. The final anomaly confidence S is then calculated by combining the dynamically corrected modal weights and the local anomaly probabilities of each modality.

7. The intelligent floating rock hazard removal method based on multimodal data alignment according to claim 1, characterized in that: The tiered autonomous risk mitigation and disposal process establishes a dual risk assessment system: The basic risk level is divided into three levels based on the rock mass stability index (RSI): RSI < 0.3 is high risk, 0.3 ≤ RSI < 0.6 is medium risk, and RSI ≥ 0.6 is low risk. The abnormal confidence level S is used to classify the warning into four levels of fine-grained warning: S < 0.3 is normal level, 0.3 ≤ S < 0.6 is attention level, 0.6 ≤ S < 0.85 is warning level, and S ≥ 0.85 is danger level; When a medium-risk, high-risk, or dangerous level warning is triggered, the electric vibrating arm is driven to continuously impact and remove loose rocks, and a visual abnormality heat map, a timeline record of rock mass damage evolution, and a quantitative risk assessment basis are pushed to the human-machine terminal. When the risk level is determined to be low and normal, retract the vibrating arm and support legs, and control the entire machine to move to the next inspection station.

8. A smart floating rock hazard mitigation system based on multimodal data alignment, characterized in that: The method for implementing the intelligent floating rock removal method based on multimodal data alignment as described in any one of claims 1 to 7 includes hardware operating equipment and a layered software control system. The hardware operating equipment includes a heavy-duty drive-by-wire chassis module, a multimodal sensing and acquisition module, an electric vibration execution module, and an ESP drive control module. The hierarchical software control system includes a data acquisition module, an intelligent recognition module, and a human-computer interaction terminal; the data acquisition module is used for multimodal raw signal acquisition and basic feature extraction. The intelligent recognition module incorporates a physical alignment unit, a causal cross-attention fusion unit, a PINN rock mass stability calculation unit, a conflict and consistency verification unit, and a risk classification output unit. The human-computer interaction terminal is used to display risk data, visualize abnormal heat maps, and store and export records of rock mass hazards.

9. The intelligent floating rock hazard removal system based on multimodal data alignment according to claim 8, characterized in that: The heavy-duty drive-by-wire chassis module includes a rear-drive front-steering wheel chassis, an ARM architecture VCU electronic control unit, a CAN communication bus, a quick-release lithium battery pack, a laser navigation and IMU obstacle avoidance unit, and a retractable electric outrigger for work stabilization at the rear of the chassis. The electric vibration actuator module includes a 360° rotating turntable, an L-shaped connecting base, a folding electric push rod, and a two-section sleeve-type electric vibration arm. The outer sleeve of the vibrating arm is made of titanium alloy, and the inner layer is a reciprocating impact rod. The matching servo motor provides an impact thrust of 200N to 400N and an impact frequency of 30 to 50 times / minute. The folding electric push rod can store the entire vibrating arm in the chassis.

10. The intelligent floating rock hazard removal system based on multimodal data alignment according to claim 8, characterized in that: The multimodal sensing and acquisition module includes a 360° waterproof night vision high-definition camera, an array of sound sensors, and multiple sets of IEPE triaxial vibration sensors. The camera is equipped with a U-Net slit segmentation acceleration unit, and vibration sensors are respectively arranged at the bottom of the vibration arm and the chassis body; The intelligent recognition module is equipped with an AI acceleration unit, which has built-in causal convolutional network, LSTM, 1D-CNN, U-Net, and physical information neural network PINN. It stores a complete rock mass benchmark feature library, vibration-acoustic transfer function parameter library, and rock fracture mechanics constraint parameter set. The intelligent recognition module also integrates a fuzzy inference dynamic weight adjustment subunit, and simultaneously corrects the fusion weights of each mode based on environmental interference parameters and the causal consistency of rock mass damage.

Images