Rock mass anomaly detection method based on vibration and visual data multi-modal fusion

Abstract

The invention relates to the technical field of geological survey and measurement, in particular to a rock mass anomaly detection method based on vibration and visual data multi-modal fusion, which comprises the following steps: firstly, respectively acquiring vibration data and image data, and then performing data feature extraction and fusion through a deep learning model; specifically, a combined model of a long short-term memory network LSTM and a convolutional neural network CNN is used to extract time sequence features and frequency information of vibration data, a Swinin-T model is used to extract spatial features of image data, and then a shared Transform encoder and a cross attention mechanism are used to realize multi-modal data alignment and fusion. And inputting the fused features into an anomaly detection module for training and tuning, and finally deploying the trained model to edge equipment for anomaly detection of the rock mass. According to the invention, through multi-modal data fusion, abnormity can be accurately identified in various complex environments, so that the probability of false alarm and missing alarm is reduced, and meanwhile, by combining with an edge computing architecture, field deployment and real-time monitoring are facilitated, and the method is suitable for various application scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of geological exploration and surveying, and particularly relates to a method for detecting rock mass anomalies based on multimodal fusion of vibration and visual data. Background Art

[0002] Traditional rock mass monitoring methods mainly rely on means such as vibration monitoring, displacement monitoring, and optical imaging. These technologies provide basic data for rock mass anomaly detection. However, there are still a series of problems to be solved urgently in the actual application of existing detection systems, which affect the accuracy, real-time performance, and reliability of anomaly detection.

[0003] Vibration monitoring technology is widely used in rock mass anomaly detection. Rock mass vibration data is collected through vibration sensors (such as accelerometers, seismographs). Vibration signals can reflect the dynamic changes of the rock mass. In particular, the tiny vibrations before the instability of the rock mass are usually important precursors of rock mass anomalies. Therefore, vibration monitoring plays an important role in rock mass anomaly detection. However, vibration data is greatly interfered by environmental noise, and the complex changes inside the rock mass often lead to the vibration signals showing non-linearity and diversity. Traditional signal processing and analysis methods often have difficulty in effectively identifying these complex anomaly patterns, resulting in false alarms or missed detections from time to time.

[0004] Displacement monitoring technology is also commonly used in rock mass anomaly detection, especially to obtain the displacement information on the surface of the rock mass through devices such as laser scanning, total station, or GPS. The change in displacement directly reflects the stability of the rock mass. Anomaly phenomena such as crack expansion, settlement, and sliding are usually accompanied by obvious displacement changes. However, the displacement monitoring method reacts relatively slowly in a dynamic environment and cannot timely capture the tiny dynamic changes of the rock mass. Especially in the case of the coupling of vibration and displacement changes, traditional displacement monitoring methods often cannot independently judge the abnormal state of the rock mass.

[0005] Optical imaging and infrared thermal imaging technologies provide another way to monitor the surface condition of the rock mass through image data. Optical imaging technology can capture anomalies such as cracks, deformations, and fissures on the surface of the rock mass in real time, while infrared imaging can identify anomalies caused by internal deformation or temperature changes. However, image data is greatly affected by environmental factors. For example, factors such as weather and light may cause the image quality to deteriorate, thereby affecting the detection effect. In addition, most existing image processing methods rely on traditional image analysis technologies and lack deep learning models for abnormal changes in rock masses, resulting in the accuracy and robustness of the system being unable to meet the requirements in complex environments.

[0006] Although current rock mass monitoring systems employ various monitoring means such as vibration, displacement, and imaging, their main problems are concentrated in the deficiencies of data fusion and anomaly detection capabilities. Most existing systems still use traditional signal processing and simple fusion methods, such as weighted average and threshold determination. These methods cannot fully consider the complex relationships and spatio-temporal characteristics between different data modalities. Especially in the combination of vibration data and image data, existing systems have failed to deeply explore their potential interactions, resulting in limited ability to accurately identify and early warn of rock mass anomalies. Therefore, although traditional methods have improved the accuracy of rock mass monitoring to a certain extent, they are still inadequate for detecting rock mass anomalies in complex environments.

[0007] In this context, the existing rock mass anomaly detection systems face the following problems:

[0008] 1. Single data source, unable to comprehensively reflect rock mass changes. Most current rock mass monitoring systems usually rely on a single data source (such as only using vibration or displacement data) for anomaly detection. This single data source cannot fully reveal the changes in the rock mass. Especially in complex and dynamically changing environments, a single data source may not be able to timely capture the small changes or complex anomaly patterns in the rock mass.

[0009] 2. Insufficient multi-modal data fusion ability. Although modern rock mass monitoring systems already have the ability to collect data from multiple sensors, the research on data fusion and intelligent analysis is still relatively weak. Most systems only rely on simple weighted average or threshold methods for data fusion and cannot fully explore the potential correlations between different modal data. Especially in complex environments, the spatio-temporal relationships between vibration data and image data may not be fully considered, resulting in the system's difficulty in accurately judging the abnormal state of the rock mass.

[0010] 3. Insufficient detection accuracy and real-time performance. Existing rock mass anomaly detection methods generally use fixed thresholds and simple signal processing techniques, which limit the detection accuracy and real-time performance of the system. The stability changes of the rock mass often have complex non-linear characteristics, and traditional anomaly detection methods cannot effectively identify these complex changes. In addition, existing methods have poor robustness to environmental noise and data loss, resulting in insufficient reliability of anomaly detection results.

[0011] 4. High false alarm and missed alarm rates. Since rock mass anomalies usually show relatively weak dynamic changes, existing detection methods are easily interfered by noise, resulting in frequent false alarms and missed alarms. False alarms not only waste system resources but may also cause overreactions, while missed alarms will lead to potential risks not being warned in time, increasing the risk of accidents.

[0012] 5. High dependence on manual intervention and slow response. In the existing rock mass monitoring system, during the analysis and decision-making process, it still relies on manual judgment or simple rule engines, resulting in a slow response speed of the system to emergencies. Especially in high-risk areas, the dependence on manual intervention is strong, which may miss the early warning opportunity and lead to disasters.

[0013] 6. High equipment and technology costs. Although rock mass monitoring technology has been gradually developing, the costs of monitoring equipment such as high-precision vibration sensors, laser scanners, and infrared imaging devices are relatively high. At the same time, the installation, maintenance, and data processing of the equipment also require a large amount of capital and human support. Such high costs limit the application of rock mass monitoring technology in some small-scale projects or high-risk areas. Summary of the Invention

[0014] The purpose of the present invention is to provide a rock mass anomaly detection method based on multi-modal fusion of vibration and visual data. By combining a deep learning model and fusing the features of vibration signals and image data, the accuracy and real-time performance of rock mass anomaly detection are improved.

[0015] To achieve the above purpose, the present invention provides a rock mass anomaly detection method based on multi-modal fusion of vibration and visual data, including the following steps:

[0016] Step 1: Collect vibration data and image data respectively to form data sets.

[0017] Step 2: After data preprocessing, use a combined model of long short-term memory network LSTM and convolutional neural network CNN to extract the features of vibration data, and use the Swin-T model to extract the spatial features in the image data.

[0018] Step 3: Use a shared Transformer encoder alignment network to align the extracted features, and adopt a cross-attention mechanism for further fusion.

[0019] Step 4: Input the fused features into the anomaly detection model for training and optimization.

[0020] Step 5: Evaluate the training process and deploy the trained model to edge devices for application.

[0021] Optionally, the vibration data in Step 1 is obtained by real-time collection through vibration sensors installed at key parts of the rock mass, which is presented as time-series data and contains multiple vibration features such as acceleration, displacement, and frequency; the image data is obtained by shooting with high-definition cameras or drones around the rock mass, and the image data is two-dimensional spatial data.

[0022] Optionally, in step 2, the long short-term memory network (LSTM) in the combined model is used to capture the temporal characteristics of the vibration signal, and the convolutional neural network (CNN) is used to extract local features and frequency information from the features output by the LSTM, and the combined model extracts a vibration feature representation with temporal dependence and frequency information.

[0023] Optionally, the Swin-T model first learns the fine-grained information of the image within a local region through the self-attention mechanism of the local window, and further gradually expands the receptive field as the network depth increases to learn the spatial correlation between different regions on the rock mass surface.

[0024] Optionally, the shared Transformer encoder alignment network in step 3 ensures that data from different modalities can be aligned on the same time scale by learning the spatio-temporal relationship between the vibration data and the image data, and obtains a new global joint feature.

[0025] Optionally, the fusion process of the cross-attention mechanism is specifically to calculate the global joint feature as the query (Q), the image feature as the key (K) and the value (V) to obtain the attention output; calculate the vibration feature as the key (K) and the value (V), and the global joint feature as the query (Q) to obtain the attention output; finally, splice the two attention outputs to obtain the fused feature.

[0026] Optionally, in step 4, the fused feature is input into the MLP for training, and the cross-entropy loss function is used to measure the difference between the predicted value and the actual label; the early stopping method is used to avoid overfitting during the tuning process.

[0027] The present invention provides a method for detecting rock mass anomalies based on multi-modal fusion of vibration and visual data. First, vibration data and image data are collected separately, and then data feature extraction and fusion are performed through a deep learning model. Specifically, a combined model of a long short-term memory network (LSTM) and a convolutional neural network (CNN) is used to extract the temporal features and frequency information of the vibration data, and the spatial features of the image data are extracted through the Swin-T model. Then, a shared Transformer encoder and a cross-attention mechanism are used to achieve multi-modal data alignment and fusion. Then, the fused feature is input into the anomaly detection module for training and tuning. Finally, the trained model is deployed to edge devices for detecting rock mass anomalies. Through multi-modal data fusion, the present invention can accurately identify anomalies in various complex environments, thereby reducing the probability of false alarms and missed detections. At the same time, combined with the edge computing architecture, it is also convenient for on-site deployment and real-time monitoring, and can adapt to a variety of application scenarios. Description of the Drawings

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0029] Figure 1 It is a schematic flow chart of the steps of a rock mass anomaly detection method based on multi-modal fusion of vibration and visual data of the present invention. Detailed implementation manners

[0030] The following details the embodiments of the present invention. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present invention, and should not be construed as a limitation of the present invention.

[0031] Please refer to Figure 1 , the present invention provides a rock mass anomaly detection method based on multi-modal fusion of vibration and visual data, including the following steps:

[0032] S1: Collect vibration data and image data to form data sets respectively;

[0033] S2: After data preprocessing, use a combined model of long short-term memory network LSTM and convolutional neural network CNN to extract the features of vibration data, and use the Swin-T model to extract the spatial features in the image data;

[0034] S3: Use a shared Transformer encoder alignment network to align the extracted features, and adopt a cross-attention mechanism for further fusion;

[0035] S4: Input the fused features into the anomaly detection model for training and optimization;

[0036] S5: Evaluate the training process, and deploy the trained model to edge devices for application.

[0037] The following further explains in combination with specific embodiments and implementation steps:

[0038] In this embodiment, the Three Gorges rock mass data set is adopted. In order to ensure the accurate monitoring of rock mass anomalies by the system, data collection is first carried out by deploying multiple sensors.

[0039] Step S1: Data collection process

[0040] Vibration data acquisition: Vibration sensors (accelerometers) installed at key parts of the rock mass are used to collect vibration signals of the rock mass in real time. The vibration signals are mainly used to monitor the dynamic changes of the rock mass, such as crack propagation, displacement, and microscopic changes in the internal structure of the rock mass. The signals generated by the vibration sensors usually appear as time-series data, containing multiple vibration characteristics (such as acceleration, displacement, frequency, etc.). To effectively train the model, the collected vibration data needs to be divided into a training set and a test set. The vibration data is generally stored in the form of time-series data, and each data point contains characteristic information such as acceleration, displacement, and frequency. The training set and the test set are divided in a ratio of 8:2.

[0041] Image data acquisition: Install high-definition cameras or drones around the rock mass to take pictures. By capturing the video stream of the rock mass surface in real time, surface changes such as cracks, displacement, and weathering of the rock mass can be captured. The image data is two-dimensional spatial data and can display the surface details of the rock mass. The image data set is also divided into a training set and a test set in a ratio of 8:2. The size of each image is 224*224 pixels, containing changes such as cracks and displacement on the rock mass surface.

[0042] Step S2: Preprocessing and feature extraction

[0043] Vibration data preprocessing: Denoise the vibration signal, and remove low-frequency noise through filtering methods; convert the vibration signal into a time-frequency diagram through short-time Fourier transform (STFT) to capture the frequency-domain characteristics of the vibration signal; standardize the time-frequency diagram to ensure that the mean of all data is 0 and the standard deviation is 1 to meet the input requirements of the neural network.

[0044] Image data preprocessing: Grayscale and normalize the image, and adjust it to a unified size; perform data augmentation (such as rotation, translation, scaling, etc.) on the image data to enhance the robustness of the model.

[0045] Vibration data feature extraction: Vibration data is time-series data collected by accelerometer sensors. Since the vibration signal contains rich dynamic information and the vibration pattern usually shows complex changes in time and frequency, a deep learning model suitable for time-series data needs to be used to extract features. In this embodiment, a combined model of long short-term memory network LSTM and convolutional neural network CNN is used to process vibration data.

[0046] LSTM is used to capture the time-series characteristics of the vibration signal, can model the long-term dependence relationship in the vibration signal, and identify the dynamic pattern when changes occur inside the rock mass. CNN is used to extract local features and frequency information from the features output by LSTM, especially the fast-changing part of the signal, which can help identify short-term anomalies and local vibration patterns in the vibration data. Through the LSTM+CNN combined model, a vibration feature representation with time-series dependence and frequency information can be obtained.

[0047] Image data feature extraction: Image data usually contains information such as cracks and displacements on the rock mass surface, which is crucial for rock mass stability detection. To more effectively extract the spatial features in the image data, this embodiment selects the Swin-T (Swin Transformer - Tiny) model. Compared with traditional convolutional neural networks (CNNs), Swin-T is better at processing high-dimensional image data, especially in detecting minor changes and cracks on the rock mass surface, and can capture details more precisely.

[0048] Through its self-attention mechanism of local windows, Swin-T first learns the fine-grained information of the image within the local area, such as the shape of cracks and surface deformation; then, as the network depth increases, the window attention mechanism can gradually expand the receptive field to capture more extensive context information, further enhancing the expression ability of image features. This model can learn the spatial correlation between different regions on the rock mass surface, thereby accurately identifying abnormal changes in the rock mass.

[0049] Step S3: Multimodal data alignment and fusion

[0050] After the feature extraction of vibration data and image data, the next step is to align and fuse the features of these two different modalities. Since there are differences in the time scale and data structure between vibration data and image data, this embodiment uses a shared Transformer encoder alignment network to align the two modalities of data.

[0051] The shared Transformer encoder alignment network ensures that data from different modalities can be aligned on the same time scale by learning the spatio-temporal relationship between vibration data and image data. The multi-head self-attention mechanism of the Transformer can guarantee the consistency of vibration features and image features in time and space by calculating the relationship between different modalities, further enhancing their complementarity, and finally obtaining new global joint features.

[0052] After aligning and fusing the vibration data features and image data features, this embodiment adopts a cross-attention mechanism to further fuse data from different modalities. The cross-attention mechanism can adjust the weights according to the correlation between different modality features to achieve more precise feature fusion. By calculating the attention weights between image features, vibration features and the global joint features respectively, a weighted representation of image features with respect to the global joint features and a weighted representation of vibration features with respect to the global joint features are generated.

[0053] Specifically, shared Transformer encoder alignment: The features of vibration data and the features of image data are aligned and fused through a shared Transformer alignment network. This network ensures the consistency of vibration data and image data on the time axis, enabling effective fusion of data from different modalities.

[0054] Cross-attention mechanism: Apply the cross-attention mechanism to the aligned features, calculate the global joint features as the query (Q), the image features as the key (K) and value (V), and obtain the attention output. Similarly, calculate the vibration features as the key (K) and value (V), and the global joint features as the query (Q) to obtain the attention output.

[0055] Step S4: Anomaly detection and training optimization

[0056] Concatenate the two attention outputs to obtain the final fused features. This fused feature contains the common information of vibration data, image data, and the global joint features obtained through the alignment network, providing richer input for subsequent anomaly detection.

[0057] Training process: Input the processed multi-modal data into the MLP for training. During the training process, use the cross-entropy loss function to measure the difference between the predicted value and the actual label.

[0058] Every 5 batches of training, perform a validation on the test set, calculate the accuracy of the model on the test set as the test metric, and the calculation formula is as follows:

[0059]

[0060] Training optimization: Optimize the model performance by adjusting hyperparameters such as the learning rate, batch size, and number of training epochs. Use early stopping to avoid overfitting and stop training when the performance of the model on the validation set no longer improves.

[0061] Step S5: Model evaluation and deployment

[0062] During the entire training process, after every 5 iterations, save the current optimal model parameters in the model checkpoint, and evaluate the accuracy and other metrics (precision, recall, F1-score) of the model on the test set.

[0063] After the training is completed, select the model with the best performance on the test set and save its parameter file.

[0064] Deploy the trained model to edge devices to ensure that the system can perform real-time anomaly detection of rock masses on edge devices. Through the communication between the device and the cloud platform, the system can send early warning information in real time to ensure that the staff can take emergency measures in time.

[0065] In summary, the present invention demonstrates the following significant advantages:

[0066] 1. Multi-modal data fusion to improve the accuracy of anomaly detection

[0067] Traditional rock mass anomaly detection usually relies on a single data source, such as vibration data or image data, and has problems such as low accuracy and poor real-time performance. The present invention can comprehensively and meticulously capture the abnormal changes of the rock mass by fusing vibration data and visual data and combining the advantages of both. Vibration data can reflect the internal dynamic changes of the rock mass, while image data can reveal the visual features such as surface cracks and displacements of the rock mass. Through multi-modal fusion, anomalies caused by both internal mechanical changes and surface deformations can be detected simultaneously, thereby improving the accuracy and comprehensiveness of detection.

[0068] 2. Adopt a deep learning model to enhance the ability of anomaly recognition

[0069] The present invention uses deep learning methods for data feature extraction and fusion. Specifically, the LSTM+CNN combined model is used to extract time series features and frequency information from vibration data, and the Swin-T model is used to extract spatial features from image data. Deep learning models can automatically learn the complex patterns in the data and have stronger feature expression capabilities, especially for small changes and anomalies in complex environments. Through these models, valuable anomaly signals can be extracted from a large amount of monitoring data, avoiding manual intervention and empirical limitations in traditional methods.

[0070] 3. Edge computing support to improve real-time performance

[0071] To ensure the real-time performance of rock mass anomaly detection, the present invention adopts an edge computing architecture and deploys the computing tasks of data processing and feature extraction on the on-site edge devices. This design greatly reduces the data transmission delay, enabling real-time response to the dynamic changes of the rock mass, timely discovery of anomalies and issuance of early warnings. This edge computing solution can effectively handle high-frequency vibration and image data streams to ensure real-time monitoring in complex environments.

[0072] 4. High robustness and adaptability to adapt to different environments

[0073] The method of the present invention can still maintain high precision and robustness in complex environments. By fusing vibration and image data, it can effectively cope with environmental interference, noise, and dynamic changes. Through multi-modal data fusion, it can accurately identify anomalies in various complex environments, thereby reducing the probabilities of false alarms and missed detections. Coupled with the adaptive learning ability of the deep learning model, it can adjust its own parameters under different environmental conditions to improve the accuracy of anomaly detection. It not only has a high level of intelligence but also can adapt to a variety of application scenarios, having broad practical application values.

[0074] The above-disclosed are only one or more preferred embodiments of the present invention. Of course, the scope of rights of the present invention cannot be limited thereby. Those of ordinary skill in the art can understand all or part of the processes of implementing the above embodiments, and the equivalent changes made according to the claims of the present invention still fall within the scope covered by the invention.

Claims

1. A rock anomaly detection method based on multimodal fusion of vibration and visual data, characterized in that: The following steps are involved: Step 1: Collect vibration data and image data to form data sets respectively; Step 2: After data preprocessing, a combined model of long short-term memory network (LSTM) and convolutional neural network (CNN) is used to extract the features of vibration data, and the Swin-T model is used to extract spatial features in image data; Step 3: Use the shared Transformer encoder alignment network to align the extracted features and use the cross-attention mechanism to further fuse them; Step 4: Input the fused features into the anomaly detection model training and tuning; Step 5: Evaluate the training process and deploy the trained model to edge devices for application.

2. The rock mass anomaly detection method based on multimodal fusion of vibration and visual data according to claim 1, characterized in that: The vibration data in step 1 is acquired in real time through vibration sensors installed at key locations of the rock mass, and is expressed as time series data, including multiple vibration characteristics such as acceleration, displacement, and frequency. The image data is acquired through high-definition cameras or drones around the rock mass, and the image data is two-dimensional spatial data.

3. The rock mass anomaly detection method based on multimodal fusion of vibration and visual data according to claim 2, characterized in that: In step 2, the long short-term memory network LSTM in the combined model is used to capture the timing characteristics of the vibration signal, and the convolutional neural network CNN is used to extract local features and frequency information from the features output by the long short-term memory network LSTM. The combined model extracts and obtains a vibration feature representation with timing dependency and frequency information.

4. The rock mass anomaly detection method based on multimodal fusion of vibration and visual data according to claim 3 is characterized in that: The Swin-T model first learns the fine-grained information of the image in the local area through the self-attention mechanism of the local window, and then gradually expands the receptive field as the network depth increases, and learns the spatial correlation between various areas on the rock surface.

5. The rock mass anomaly detection method based on multimodal fusion of vibration and visual data according to claim 4, characterized in that: The shared Transformer encoder alignment network in step 3 learns the spatiotemporal relationship between vibration data and image data to ensure that data from different modalities can be aligned on the same time scale to obtain new global joint features.

6. The rock mass anomaly detection method based on multimodal fusion of vibration and visual data according to claim 5, characterized in that: The fusion process of the cross-attention mechanism is specifically to use the global joint feature as the query (Q), the image feature as the key (K) and the value (V) to calculate and obtain the attention output; the vibration feature as the key (K) and the value (V), and the global joint feature as the query (Q) to calculate and obtain the attention output; finally, the two attention outputs are spliced ​​to obtain the fused feature.

7. The rock mass anomaly detection method based on multimodal fusion of vibration and visual data according to claim 6, characterized in that: In step 4, the fused features are input into the MLP for training, and the cross entropy loss function is used to measure the difference between the predicted value and the actual label; the early stopping method is used in the tuning process to avoid overfitting.

Images