Mine earthquake intelligent identification method based on multi-mode deep learning and signal processing

Abstract

The invention discloses a mine earthquake intelligent identification method based on multi-mode deep learning and signal processing, and belongs to the technical field of mine earthquake identification. Aiming at the problems that a traditional mine earthquake signal processing method cannot fully utilize spatial relevance and frequency domain characteristics of signals in a sensor network, the automation level is relatively low, the response speed is slow, and the emergency processing effect is influenced, seismic waveform data and multi-modal characteristic input are adopted, so that the emergency processing effect is influenced. Based on text semantic features generated by a large language model (LLMs), frequency domain features extracted by Fourier transform and time sequence features after obspy processing, a dynamic graph recognition model of a fusion graph neural network (GNN) and Transform is constructed; graph structure edge weights are dynamically updated through cross correlation between signals, and the structure change of complex environments such as mine roadways is self-adapted.

Description

Technical Field

[0001] The present invention belongs to the technical field of mine earthquake recognition, and specifically relates to an intelligent mine earthquake recognition method based on multimodal deep learning and signal processing. Background Art

[0002] Mine safety monitoring is a crucial component in ensuring both production safety and the safety of personnel. With the expansion of mining operations and the increasing demand for monitoring, real-time detection and accurate identification of mine earthquake signals have become a research priority. Traditional mine earthquake detection methods rely primarily on threshold triggering or single-mode signal analysis. These methods are susceptible to interference from environmental noise, have a high false alarm rate, and struggle to effectively distinguish complex noise from actual vibration events, compromising monitoring accuracy and reliability.

[0003] In recent years, deep learning technology has garnered widespread attention in the field of mine-seismic signal processing. However, existing solutions often focus on single time series models, such as convolutional neural networks (CNNs) or long-short-term memory networks (LSTMs). These approaches fail to fully exploit the spatial correlation and frequency domain characteristics of signals in sensor networks, limiting the performance of detection models. Furthermore, the interpretation and reporting of mine-seismic events often rely on manual labor, resulting in a low level of automation, which slows response times and hinders the effectiveness of emergency response. Summary of the Invention

[0004] To address the problems of traditional mine-seismic signal processing methods, which fail to fully utilize the spatial correlation and frequency domain characteristics of signals in sensor networks and have a low level of automation, resulting in slow response and impacting emergency response effectiveness, this paper proposes an intelligent mine-seismic signal recognition method based on multimodal deep learning and signal processing. This method integrates large language models (LLMs), Fourier transforms, a seismic data processing tool (obspy), graph neural networks (GNNs), and Transformer technologies. This method integrates multimodal data processing and combines the spatiotemporal and frequency domain characteristics of signals to achieve high-precision, real-time detection, classification, and early warning of mine-seismic events, effectively improving monitoring accuracy and response efficiency.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for intelligent identification of mine earthquakes based on multimodal deep learning and signal processing, the method comprising the following steps:

[0007] Step 1: Deploy multiple vibration sensors in a mine environment to form a mine sensor network. The mine sensor network collects seismic waveform data in real time. The seismic waveform data includes acceleration signals of P and S waves in the X, Y, and Z axes, and synchronizes the timestamp information.

[0008] Step 2: Perform noise suppression on the collected seismic waveform data. Use the open source seismology toolkit Obspy to read the raw acceleration waveform data and call the BandpassFilter module in Obspy to perform bandpass filtering on the data signal to filter out low-frequency terrain drift and high-frequency mechanical noise.

[0009] The original acceleration waveform data signal collected in step 2 is , bandpass filtering The processing formula is:

[0010]

[0011] in, Represents the feature vector after pooling.

[0012] Step 3: Segment the continuous waveform data signal into segments according to fixed time windows, annotate the sliced ​​data by combining historical event records or manual annotation methods, and construct a supervised learning dataset;

[0013] Step 4: Call the STA / LTA algorithm in Obspy to quickly scan the processed waveform data. By analyzing the short-time average and long-time average energy ratio, candidate abnormal vibration events are detected to form an event candidate set.

[0014] The short-time average in step 4 / Long-term average The formula for energy ratio is:

[0015]

[0016]

[0017]

[0018] in, express The short-term energy of a moment is used to calculate the signal energy in a short window near the current moment and capture the instantaneous change of the signal; express The long-term energy at the moment is used to calculate the signal energy in a long window near the current moment, reflecting the average energy level of the background noise; Indicates the short-term window length; Indicates the long window length.

[0019] Step 5: Build a graph structure for the seismic waveform data based on the graph neural network, extract the frequency domain statistical features and text feature representation of the seismic waveform data in the event candidate set, fuse the frequency domain statistical features and text feature representation, and input the fused multimodal feature representation into the graph structure;

[0020] Step 5.1: Build a graph structure by treating each sensor as a graph node ,According to the physical distance between the vibration sensors and the mutual correlation between the corresponding signals, the edge weights are calculated to achieve the construction of the adaptive graph structure. The formula is:

[0021]

[0022] in, represents the edge weight between nodes i and j; represents the mutual correlation, is the distance between nodes, is the weighting parameter; Represents the feature vector of node i; represents the feature vector of node j;

[0023] Step 5.2: Perform short-time Fourier transform on the seismic waveform data of each event candidate set to generate the corresponding time-frequency spectrum. The formula of short-time Fourier transform (STFT) is:

[0024]

[0025] in, represents the spectral intensity at time t and frequency f; Indicates time signal; Indicates the center is Window function at time instant; represents a complex exponential basis; Indicates frequency; represents the integral variable;

[0026] Step 5.3: Extract frequency domain statistical features based on the time-spectrum graph , mean frequency The calculation formula is:

[0027]

[0028] Main frequency The calculation formula is:

[0029]

[0030] Step 5.4: Obtain metadata information related to the vibration sensor and input it into large language models (LLMs) to generate a semantic description of the current signal environment;

[0031] Step 5.5: Input the semantic description text into the embedding layer of the large language model for vectorized encoding to generate semantic description text , through the embedding model Get 128-dimensional text feature representation , the formula is:

[0032]

[0033] Step 5.6: Frequency domain statistical features and text feature representation Splicing to form a complete multimodal feature representation of each vibration sensor node , and input the graph structure, the formula for multimodal feature representation is:

[0034]

[0035] in, The frequency domain statistical features of the i-th vibration sensor node are extracted from the frequency domain analysis of the vibration signal to characterize the frequency-related characteristics of the vibration of the node; Represents the text feature representation of the i-th shock sensor node; Represents the dimension of the concatenated multimodal feature Vi.

[0036] Step 6: Construct a spatiotemporal joint feature matrix. The temporal embedding sequence (time step × feature dimension) of each vibration sensor is passed through a four-layer Transformer. A multi-head attention mechanism is used to capture long-term dependencies. Timestamp information (accurate to milliseconds) is injected into the positional encoding to obtain a spatiotemporal joint feature matrix.

[0037] The specific operations of step 6 are:

[0038] Step 6.1: The feature representation of each vibration sensor node in the continuous time window (time step × feature dimension), including the node features after frequency domain and semantic fusion, is used to form a time series input sequence. The formula is:

[0039]

[0040] in, represents the temporal feature sequence of node i; represents the characteristics of sensor node i at time t; Indicates the length of the time window;

[0041] Step 6.2: Use a 4-layer Transformer encoder to process the temporal input sequence. Each layer of the Transformer contains a multi-head attention mechanism and a feedforward neural network to capture the long-term dependencies and feature interactions in the vibration signal. Position encoding is added to each layer of the Transformer encoder. The formula is:

[0042]

[0043] in, Represents the encoded features; Indicates positional encoding;

[0044] The update process of each layer of Transformer encoder is:

[0045]

[0046] in, Represents the layer structure in the l-th layer Transformer encoder;

[0047] Step 6.3: The millisecond-accurate timestamp is embedded in the position code and used together with the input to calculate the attention weight. Finally, a spatiotemporal joint feature matrix is ​​output to provide temporal feature support for subsequent event classification and report generation. The formula for the output spatiotemporal joint feature matrix is:

[0048]

[0049] in, represents the spatiotemporal joint feature matrix corresponding to the i-th vibration sensor node; Represents the final feature representation after processing by the L-layer Transformer encoder.

[0050] Step 7: Event classification and report generation: The spatiotemporal joint feature matrix is ​​passed through a fully connected layer to output the event probability (mine tremors / blasting / mechanical noise), which is then input into the classifier. The classification results and key features are input into LLMs to generate a structured report.

[0051] The specific operations of step 7 are:

[0052] The spatiotemporal joint feature matrix is ​​pooled and then input into the classifier:

[0053]

[0054]

[0055] in, Represents the spatiotemporal joint feature matrix The pooled feature vector obtained after the pooling operation, represents the pooling operation, represents the predicted probability distribution of the classifier output, represents the Softmax activation function, represents the weight matrix of the linear transformation in the classifier, Represents the bias term of the linear transformation in the classifier.

[0056] Combine the classification results with the features and input :

[0057]

[0058] in, It represents the predicted probability distribution output by the previous classifier; Represents the complete multimodal feature representation of the i-th shock sensor node; hi is the feature vector after the previous pooling.

[0059] Compared with the prior art, the present invention has the following advantages:

[0060] By fusing multimodal features to construct a dynamic graph recognition model, the robustness of recognition in complex environments is improved. By introducing a large language model to extract semantic information, the ability to understand the semantics of mine earthquake scenes is effectively enhanced. By dynamically adjusting the graph structure through inter-signal correlation, the model can adapt to changes in mine structure. At the same time, an end-to-end full-process automated recognition method is adopted to improve processing efficiency and response speed. The present invention can realize the fusion processing of multimodal data, combine the spatiotemporal characteristics and frequency domain characteristics of the signal, and achieve high-precision real-time detection, classification and early warning of mine earthquake events, effectively improving the detection accuracy and response efficiency of monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 : Method architecture diagram;

[0062] Figure 2 : Data processing flow chart;

[0063] Figure 3 : GNN-Transformer joint model structure diagram;

[0064] Figure 4 : Result diagram of the model picking up seismic phases of mine earthquake data. DETAILED DESCRIPTION

[0065] To gain a deeper understanding of the present invention, we will provide a comprehensive and detailed description thereof. However, the present invention has various implementations and is not limited to the specific examples listed herein. These examples are presented to enhance a comprehensive understanding of the present disclosure.

[0066] A method for intelligent identification of mine earthquakes based on multimodal deep learning and signal processing, the method comprising the following steps:

[0067] Step 1: Deploy multiple vibration sensors in a mine environment to form a mine sensor network. The mine sensor network collects seismic waveform data in real time. The seismic waveform data includes acceleration signals of P and S waves in the X, Y, and Z axes, and synchronizes the timestamp information.

[0068] Step 2: Perform noise suppression on the collected seismic waveform data. Use the open source seismology toolkit Obspy to read the raw acceleration waveform data and call the BandpassFilter module in Obspy to perform bandpass filtering on the data signal to filter out low-frequency terrain drift and high-frequency mechanical noise.

[0069] The original acceleration waveform data signal collected in step 2 is , bandpass filtering The processing formula is:

[0070]

[0071] in, Represents the feature vector after pooling.

[0072] Step 3: Segment the continuous waveform data signal into segments according to fixed time windows, annotate the sliced ​​data by combining historical event records or manual annotation methods, and construct a supervised learning dataset;

[0073] Step 4: Call the STA / LTA algorithm in Obspy to quickly scan the processed waveform data. By analyzing the short-time average and long-time average energy ratio, candidate abnormal vibration events are detected to form an event candidate set.

[0074] The short-time average in step 4 / Long-term average The formula for energy ratio is:

[0075]

[0076]

[0077]

[0078] in, express The short-term energy of a moment is used to calculate the signal energy in a short window near the current moment and capture the instantaneous change of the signal; express The long-term energy at the moment is used to calculate the signal energy in the long window near the current moment, reflecting the average energy level of the background noise; Indicates the short-term window length; Indicates the long window length.

[0079] Step 5: Build a graph structure for the seismic waveform data based on the graph neural network, extract the frequency domain statistical features and text feature representation of the seismic waveform data in the event candidate set, fuse the frequency domain statistical features and text feature representation, and input the fused multimodal feature representation into the graph structure;

[0080] Step 5.1: Build a graph structure by treating each sensor as a graph node ,According to the physical distance between the vibration sensors and the mutual correlation between the corresponding signals, the edge weights are calculated to achieve the construction of the adaptive graph structure. The formula is:

[0081]

[0082] in, represents the edge weight between nodes i and j; represents the mutual correlation, is the distance between nodes, is the weighting parameter; Represents the feature vector of node i; represents the feature vector of node j;

[0083] Step 5.2: Perform short-time Fourier transform on the seismic waveform data of each event candidate set to generate the corresponding time-frequency spectrum. The formula of short-time Fourier transform (STFT) is:

[0084]

[0085] in, represents the spectral intensity at time t and frequency f; Indicates time signal; represents the window function centered at t; represents a complex exponential basis; Indicates frequency; represents the integral variable;

[0086] Step 5.3: Extract frequency domain statistical features based on the time-spectrum graph , mean frequency The calculation formula is:

[0087]

[0088] Main frequency The calculation formula is:

[0089]

[0090] Step 5.4: Obtain metadata information related to the vibration sensor and input it into large language models (LLMs) to generate a semantic description of the current signal environment;

[0091] Step 5.5: Input the semantic description text into the embedding layer of the large language model for vectorized encoding to generate semantic description text , through the embedding model Get 128-dimensional text feature representation , the formula is:

[0092]

[0093] Step 5.6: Frequency domain statistical features and text feature representation Splicing to form a complete multimodal feature representation of each vibration sensor node , and input the graph structure, the formula for multimodal feature representation is:

[0094]

[0095] in, The frequency domain statistical features of the i-th vibration sensor node are extracted from the frequency domain analysis of the vibration signal to characterize the frequency-related characteristics of the vibration of the node; Represents the text feature representation of the i-th shock sensor node; Represents the dimension of the concatenated multimodal feature Vi.

[0096] Step 6: Construct a spatiotemporal joint feature matrix. The temporal embedding sequence (time step × feature dimension) of each vibration sensor is passed through a four-layer Transformer. A multi-head attention mechanism is used to capture long-term dependencies. Timestamp information (accurate to milliseconds) is injected into the positional encoding to obtain a spatiotemporal joint feature matrix.

[0097] The specific operations of step 6 are:

[0098] Step 6.1: The feature representation of each vibration sensor node in the continuous time window (time step × feature dimension), including the node features after frequency domain and semantic fusion, is used to form a time series input sequence. The formula is:

[0099]

[0100] in, represents the temporal feature sequence of node i; represents the characteristics of sensor node i at time t; Indicates the length of the time window;

[0101] Step 6.2: Use a 4-layer Transformer encoder to process the temporal input sequence. Each layer of the Transformer contains a multi-head attention mechanism and a feedforward neural network to capture the long-term dependencies and feature interactions in the vibration signal. Position encoding is added to each layer of the Transformer encoder. The formula is:

[0102]

[0103] in, Represents the encoded features; Represents the original features of the i-th time series input sequence; Indicates positional encoding;

[0104] The update process of each layer of Transformer encoder is:

[0105]

[0106] in, Represents the layer structure in the l-th layer Transformer encoder;

[0107] Step 6.3: The millisecond-accurate timestamp is embedded in the position code and used together with the input to calculate the attention weight. Finally, a spatiotemporal joint feature matrix is ​​output to provide temporal feature support for subsequent event classification and report generation. The formula for the output spatiotemporal joint feature matrix is:

[0108]

[0109] in, represents the spatiotemporal joint feature matrix corresponding to the i-th vibration sensor node; Represents the final feature representation after processing by the L-layer Transformer encoder.

[0110] Step 7: Event classification and report generation: The spatiotemporal joint feature matrix is ​​passed through a fully connected layer to output the event probability (mine tremors / blasting / mechanical noise), which is then input into the classifier. The classification results and key features are input into LLMs to generate a structured report.

[0111] The specific operations of step 7 are:

[0112] The spatiotemporal joint feature matrix is ​​pooled and then input into the classifier:

[0113]

[0114]

[0115] in, Represents the spatiotemporal joint feature matrix The pooled feature vector obtained after the pooling operation, represents the pooling operation, represents the predicted probability distribution of the classifier output, represents the Softmax activation function, represents the weight matrix of the linear transformation in the classifier, Represents the bias term of the linear transformation in the classifier.

[0116] Combine the classification results with the features and input :

[0117]

[0118] in, It represents the predicted probability distribution output by the previous classifier; represents the complete multimodal feature representation of the i-th shock sensor node; That is the feature vector after the previous pooling.

[0119] To address the need for intelligent identification of vibration events in mining environments, this invention deploys multiple vibration sensing nodes across the monitoring area, forming a distributed sensor network. Each sensor continuously collects raw triaxial acceleration signals (including P-wave and S-wave components), generating continuous vibration waveform data. This collected data undergoes a multi-stage processing process, sequentially completing steps including signal denoising, event detection, feature extraction, graph construction, spatiotemporal feature modeling, and classification decisions. This process, combined with semantic auxiliary information generated by a large language model, further enhances the model's ability to discriminate complex vibration events.

[0120] Table 1 Comparison of performance parameters before and after model improvement

[0121]

[0122] As shown in Table 1, compared with the traditional STA / LTA method, the accuracy of the present invention is improved from 82% to 96%, the false alarm rate is reduced from 25% to 5%, and the response delay is shortened from 2 seconds to 0.8 seconds.

[0123] Any matters not described in detail in this specification are prior art known to those skilled in the art. Although the above description of the present invention is based on specific embodiments to facilitate understanding of the present invention by those skilled in the art, it should be understood that the present invention is not limited to the scope of the specific embodiments. As long as various modifications are within the spirit and scope of the present invention as defined and determined by the appended claims, such modifications will be obvious to those skilled in the art, and all inventions and creations utilizing the concepts of the present invention are protected.

Claims

1. A method for intelligent identification of mine earthquakes based on multimodal deep learning and signal processing, characterized in that: The method comprises the following steps: Step 1: Deploy multiple vibration sensors in a mine environment to form a mine sensor network. The mine sensor network collects seismic waveform data in real time. The seismic waveform data includes acceleration signals of P and S waves in the X, Y, and Z axes, and synchronizes the timestamp information. Step 2: Perform noise suppression on the collected seismic waveform data. Use the open source seismology toolkit Obspy to read the raw acceleration waveform data and call the BandpassFilter module in Obspy to perform bandpass filtering on the data signal to filter out low-frequency terrain drift and high-frequency mechanical noise. Step 3: Segment the continuous waveform data signal into segments according to fixed time windows, annotate the sliced ​​data by combining historical event records or manual annotation methods, and construct a supervised learning dataset; Step 4: Call the STA / LTA algorithm in Obspy to quickly scan the processed waveform data. By analyzing the short-time average and long-time average energy ratio, candidate abnormal vibration events are detected to form an event candidate set. Step 5: Build a graph structure for the seismic waveform data based on the graph neural network, extract the frequency domain statistical features and text feature representation of the seismic waveform data in the event candidate set, fuse the frequency domain statistical features and text feature representation, and input the fused multimodal feature representation into the graph structure; Step 6: Construct a spatiotemporal joint feature matrix. The time-series embedding sequence of each vibration sensor passes through a four-layer Transformer, using a multi-head attention mechanism to capture long-term dependencies and inject timestamp information into the positional encoding to obtain a spatiotemporal joint feature matrix. Step 7: Event classification and report generation: The spatiotemporal joint feature matrix is ​​pooled through a fully connected layer and then input into the classifier. The classification results and key features are input into LLMs to generate a structured report.

2. The method for intelligent identification of mine earthquakes based on multimodal deep learning and signal processing according to claim 1, characterized in that: The original acceleration waveform data signal collected in step 2 is , bandpass filtering The processing formula is: ， Among them, h(t) represents the feature vector after pooling.

3. The method for intelligent identification of mine earthquakes based on multimodal deep learning and signal processing according to claim 2, characterized in that: The short-time average in step 4 / Long-term average Energy ratio The formula is: ， ， ， in, express The short-term energy of a moment is used to calculate the signal energy in a short window near the current moment and capture the instantaneous change of the signal; express The long-term energy at the moment is used to calculate the signal energy in a long window near the current moment, reflecting the average energy level of the background noise; Indicates the short-term window length; Indicates the long window length.

4. The method for intelligent identification of mine earthquakes based on multimodal deep learning and signal processing according to claim 3 is characterized in that: The specific operations of step 5 are: Step 5.1: Build a graph structure by treating each sensor as a graph node ,According to the physical distance between the vibration sensors and the mutual correlation between the corresponding signals, the edge weights are calculated to achieve the construction of the adaptive graph structure. The formula is: ， in, represents the edge weight between nodes i and j; represents the mutual correlation, is the distance between nodes, is the weighting parameter; Represents the feature vector of node i; represents the feature vector of node j; Step 5.2: Perform short-time Fourier transform on the seismic waveform data of each event candidate set to generate the corresponding time-frequency spectrum. The formula of short-time Fourier transform is: ， in, represents the spectral intensity at time t and frequency f; Indicates time signal; represents the window function centered at time t; represents a complex exponential basis; Indicates frequency; represents the integral variable; Step 5.3: Extract frequency domain statistical features based on the time-spectrum graph , mean frequency The calculation formula is: ， Main frequency The calculation formula is: ； Step 5.4: Obtain metadata information related to the vibration sensor and input it into the large language model to generate a semantic description of the current signal environment; Step 5.5: Input the semantic description text into the embedding layer of the large language model for vectorized encoding to generate semantic description text , through the embedding model Get 128-dimensional text feature representation , the formula is: ， Step 5.6: Frequency domain statistical features and text feature representation Splicing to form a complete multimodal feature representation of each vibration sensor node , and input the graph structure, the formula for multimodal feature representation is: ， in, The frequency domain statistical features of the i-th vibration sensor node are extracted from the frequency domain analysis of the vibration signal to characterize the frequency-related characteristics of the vibration of the node; Represents the text feature representation of the i-th shock sensor node; Represents the multimodal features after splicing dimension.

5. The method for intelligent identification of mine earthquakes based on multimodal deep learning and signal processing according to claim 4 is characterized in that: The specific operations of step 6 are: Step 6.1: The feature representation of each vibration sensor node in the continuous time window constitutes a time series input sequence, the formula is: ， in, represents the temporal feature sequence of node i; represents the characteristics of sensor node i at time t; Indicates the length of the time window; Step 6.2: Use a 4-layer Transformer encoder to process the temporal input sequence. Add position encoding to each layer of Transformer encoder. The formula is: ， in, Represents the encoded features; Indicates positional encoding; The update process of each layer of Transformer encoder is: ， in, Represents the layer structure in the l-th layer Transformer encoder; Step 6.3: The millisecond-accurate timestamp is embedded in the position code and used together with the input to calculate the attention weight. Finally, a spatiotemporal joint feature matrix is ​​output to provide temporal feature support for subsequent event classification and report generation. The formula for the output spatiotemporal joint feature matrix is: ， in, represents the spatiotemporal joint feature matrix corresponding to the i-th vibration sensor node; Represents the final feature representation after processing by the L-layer Transformer encoder.

6. The method for intelligent identification of mine tremors based on multimodal deep learning and signal processing according to claim 5, characterized in that: The specific operations of step 7 are: The spatiotemporal joint feature matrix is ​​pooled and then input into the classifier: ， ， in, Represents the pooled feature vector obtained after pooling the spatiotemporal joint feature matrix Hi, represents the pooling operation, represents the predicted probability distribution of the classifier output, represents the Softmax activation function, represents the weight matrix of the linear transformation in the classifier, Represents the bias term of the linear transformation in the classifier; Combine the classification results with the features and input : ， in, It represents the predicted probability distribution output by the previous classifier; Represents the complete multimodal feature representation of the i-th shock sensor node; hi is the feature vector after the previous pooling.

Images