A mine earthquake intelligent identification method based on multi-modal deep learning and signal processing

Abstract

This invention discloses an intelligent method for identifying mine seismic events based on multimodal deep learning and signal processing, belonging to the field of mine seismic identification technology. Addressing the problems of traditional mine seismic signal processing methods failing to fully utilize the spatial correlation and frequency domain features of signals in sensor networks, and their low level of automation leading to slow response speeds and impacting emergency response effectiveness, this invention constructs a dynamic graph recognition model that integrates graph neural networks (GNNs) and Transformers. This model uses seismic waveform data and multimodal feature inputs, based on textual semantic features generated by Large Language Models (LLMs), frequency domain features extracted by Fourier transform, and temporal features processed by Obspy. The model dynamically updates the edge weights of the graph structure through the cross-correlation between signals, adapting to structural changes in complex environments such as mine roadways.

Description

Technical Field

[0001] This invention belongs to the field of mine seismic identification technology, specifically relating to an intelligent mine seismic identification method based on multimodal deep learning and signal processing. Background Technology

[0002] Mine safety monitoring is a crucial link in ensuring the safety of mine production and the lives of personnel. With the expansion of mining scale and the increasing demand for monitoring, the real-time detection and accurate identification of seismic signals has become a research focus. Traditional seismic detection methods mainly rely on threshold triggering or single-mode signal analysis. These methods are easily affected by environmental noise, have a high false alarm rate, and struggle to effectively distinguish between complex noise and real vibration events, thus impacting the accuracy and reliability of monitoring.

[0003] In recent years, deep learning technology has received widespread attention in the field of mine seismic signal processing. However, existing solutions are mostly focused on single time-series models, such as convolutional neural networks (CNNs) or long short-term memory networks (LSTMs), which cannot fully utilize the spatial correlation and frequency domain features of signals in sensor networks, thus limiting the performance improvement of detection models. In addition, the interpretation and report generation of mine seismic events often rely on manual work, with low levels of automation, resulting in slow response speeds and affecting the effectiveness of emergency response. Summary of the Invention

[0004] To address the shortcomings of traditional mine seismic signal processing methods, which fail to fully utilize the spatial correlation and frequency domain features of signals in sensor networks and suffer from low automation levels leading to slow response times and impacting emergency response effectiveness, this invention provides a mine seismic intelligent identification method based on multimodal deep learning and signal processing. This method integrates Large Language Models (LLMs), Fourier Transform, seismic data processing tools (obspy), Graph Neural Networks (GNNs), and Transformer technology for intelligent mine seismic signal identification. This method enables the fusion processing of multimodal data, combining the spatiotemporal characteristics and frequency domain features of signals to achieve high-precision real-time detection, classification, and early warning of mine seismic events, effectively improving the detection accuracy and response efficiency of monitoring.

[0005] To achieve the above objectives, the present invention employs the following technical solutions:

[0006] A method for intelligent identification of mine seismic activity based on multimodal deep learning and signal processing, the method comprising the following steps:

[0007] Step 1: Deploy multiple vibration sensors in the mining environment to form a mining sensor network. Collect seismic waveform data in real time through the mining sensor network. The seismic waveform data includes acceleration signals of P-waves and S-waves in the X, Y, and Z axes, and synchronizes timestamp information.

[0008] Step 2: Perform noise suppression processing on the acquired seismic waveform data. Use the open-source seismology toolkit Obspy to read the raw acceleration waveform data and call the BandpassFilter module in Obspy to perform bandpass filtering on the data signal to filter out low-frequency terrain drift and high-frequency mechanical noise.

[0009] The raw acceleration waveform data signal acquired in step 2 is bandpass filtering The formula for processing is: ,in, This represents the feature vector after pooling.

[0010] Step 3: Divide the continuous waveform data signal into segments according to fixed time windows, and label the segmented data by combining historical event records or manual annotation methods to construct a supervised learning dataset;

[0011] Step 4: Call the STA / LTA algorithm in Obspy to quickly scan the processed waveform data. Analyze the energy ratio between short-time average and long-time average to detect candidate abnormal vibration events and form a candidate event set.

[0012] short-time average in step 4 Long-term average The formula for the energy ratio is: , , ,in, express The short-time energy at a given moment is used to calculate the signal energy within a short window near the current moment, capturing instantaneous changes in the signal. express The long-term energy at a given time is calculated by measuring the signal energy within a long window near the current time, reflecting the average energy level of the background noise. Indicates the short-time window length; Indicates the length of the long-term window.

[0013] Step 5: Construct a graph structure for the seismic waveform data based on the graph neural network, extract the frequency domain statistical features and textual feature representations of the seismic waveform data in the event candidate set, fuse the frequency domain statistical features and textual feature representations, and input the fused multimodal feature representation into the graph structure;

[0014] Step 5.1: Construct a graph structure by treating each sensor as a graph node. Based on the physical distance between vibration sensors and the cross-correlation between their corresponding signals, the weights of the edges are calculated to construct the adaptive graph structure. The formula is as follows: ,in, This represents the edge weight between nodes i and j; Indicates cross-correlation. The distance between nodes. For weighted parameters; Represents the feature vector of node i; This represents the feature vector of node j;

[0015] Step 5.2: Perform a short-time Fourier transform on the seismic waveform data of each event candidate set to generate the corresponding time-frequency spectrum. The formula for the short-time Fourier transform (STFT) is: ,in, This represents the spectral intensity at time t and frequency f; Indicates a time signal; Indicates the center is The window function at time t; Indicates a complex exponential basis; Indicates frequency; Represents the integral variable;

[0016] Step 5.3: Extract frequency domain statistical features based on time-spectrum graph Mean frequency The calculation formula is: ;

[0017] clock speed The calculation formula is: ;

[0018] Step 5.4: Obtain metadata information related to the vibration sensor and input it into Large Language Models (LLMs) to generate a semantic description of the current signal environment;

[0019] Step 5.5: Input the semantic description text into the embedding layer of the large language model for vectorized encoding to generate the semantic description text. By embedding model A 128-dimensional text feature representation is obtained. The formula is: ;

[0020] Step 5.6: Analyze the frequency domain statistical characteristics Text feature representation By splicing the data, a complete multimodal feature representation of each vibration sensor node is formed. And input the graph structure, the formula for multimodal feature representation is: ,in, The frequency domain statistical characteristics of the i-th vibration sensor node are extracted from the frequency domain analysis of the vibration signal to characterize the frequency correlation properties of the node's vibration. The text feature representation of the i-th vibration sensor node; Indicates the multimodal features after splicing Dimensions.

[0021] Step 6: Construct the spatiotemporal joint feature matrix; The temporal embedding sequence (time step × feature dimension) of each vibration sensor is processed through 4 layers of Transformer, and the multi-head attention mechanism is used to capture long-term dependencies. Timestamp information (accurate to milliseconds) is injected into the position encoding to obtain the spatiotemporal joint feature matrix;

[0022] The specific operation of step 6 is as follows:

[0023] Step 6.1: Construct a time-series input sequence by representing the features of each vibration sensor node within a continuous time window (time step × feature dimension), including the node features after frequency domain and semantic fusion, using the following formula: ,in, Represents the temporal feature sequence of node i; Indicates that sensor node i at time i Features of the location; Indicates the length of the time window;

[0024] Step 6.2: Process the temporal input sequence using a 4-layer Transformer encoder. Each Transformer layer includes a multi-head attention mechanism and a feedforward neural network to capture long-term dependencies and feature interactions in the vibration signal. Position encoding is added to each Transformer encoder layer, as shown in the formula: ,in, This represents the encoded features; Indicates positional encoding;

[0025] The update process for each Transformer encoder layer is as follows: ,in, This represents the layer structure in the l-th layer Transformer encoder;

[0026] Step 6.3: Embed the timestamp accurate to milliseconds into the location encoding, and use it together with the input to participate in the attention weight calculation. The final output is a spatiotemporal joint feature matrix, providing temporal feature support for subsequent event classification and report generation. The formula for the output spatiotemporal joint feature matrix is: ,in, This represents the spatiotemporal joint feature matrix corresponding to the i-th vibration sensor node; This represents the final feature representation obtained after processing by the L-layer Transformer encoder.

[0027] Step 7: Event Classification and Report Generation; The spatiotemporal joint feature matrix is ​​passed through a fully connected layer to output the event probabilities (mine tremors / blasting / mechanical noise), which are then input into a classifier. The classification results and key features are input into LLMs to generate a structured report.

[0028] The specific operation of step 7 is as follows:

[0029] The spatiotemporal joint feature matrix is ​​pooled and then input into the classifier: , ,in, Represents the spatiotemporal joint characteristic matrix The pooled feature vector obtained after performing the pooling operation. This indicates a pooling operation. This represents the predicted probability distribution of the classifier output. This represents the Softmax activation function. This represents the weight matrix of the linear transformation in the classifier. This represents the bias term of the linear transformation in the classifier.

[0030] Input the concatenated classification results and features. : ,

[0031] in, This represents the predicted probability distribution output by the previous classifier; This represents the complete multimodal feature representation of the i-th vibration sensor node; That is, the feature vector after pooling.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] By fusing multimodal features to construct a dynamic graph recognition model, the robustness of recognition in complex environments is improved. By introducing a large language model to extract semantic information, the ability to understand the semantics of mining seismic scenarios is effectively enhanced. The graph structure is dynamically adjusted through the correlation between signals, enabling the model to adapt to changes in mine structure. Simultaneously, an end-to-end fully automated recognition approach is adopted, improving processing efficiency and response speed. This invention enables the fusion processing of multimodal data, combining the spatiotemporal characteristics and frequency domain features of signals to achieve high-precision real-time detection, classification, and early warning of mining seismic events, effectively improving the detection accuracy and response efficiency of monitoring. Attached Figure Description

[0034] Figure 1 Method architecture diagram;

[0035] Figure 2 Data processing flowchart;

[0036] Figure 3: GNN-Transformer joint model structure diagram;

[0037] Figure 4 Figure: Seismic phase picking results of the model on seismic data. Detailed Implementation

[0038] To gain a deeper understanding of this invention, we will provide a comprehensive and detailed description. However, this invention has various implementations and is not limited to the specific examples listed herein. These examples are presented to enhance a full understanding of the disclosure of this invention.

[0039] A method for intelligent identification of mine seismic activity based on multimodal deep learning and signal processing, the method comprising the following steps:

[0040] Step 1: Deploy multiple vibration sensors in the mining environment to form a mining sensor network. Collect seismic waveform data in real time through the mining sensor network. The seismic waveform data includes acceleration signals of P-waves and S-waves in the X, Y, and Z axes, and synchronizes timestamp information.

[0041] Step 2: Perform noise suppression processing on the acquired seismic waveform data. Use the open-source seismology toolkit Obspy to read the raw acceleration waveform data and call the BandpassFilter module in Obspy to perform bandpass filtering on the data signal to filter out low-frequency terrain drift and high-frequency mechanical noise.

[0042] The raw acceleration waveform data signal acquired in step 2 is bandpass filtering The formula for processing is: ,in, This represents the feature vector after pooling.

[0043] Step 3: Divide the continuous waveform data signal into segments according to fixed time windows, and label the segmented data by combining historical event records or manual annotation methods to construct a supervised learning dataset;

[0044] Step 4: Call the STA / LTA algorithm in Obspy to quickly scan the processed waveform data. Analyze the energy ratio between short-time average and long-time average to detect candidate abnormal vibration events and form a candidate event set.

[0045] short-time average in step 4 Long-term average The formula for the energy ratio is: , , ,in, express The short-time energy at a given moment is used to calculate the signal energy within a short window near the current moment, capturing instantaneous changes in the signal. express The long-term energy at a given time is calculated by measuring the signal energy within a long window near the current time, reflecting the average energy level of the background noise. Indicates the short-time window length; Indicates the length of the long-term window.

[0046] Step 5: Construct a graph structure for the seismic waveform data based on the graph neural network, extract the frequency domain statistical features and textual feature representations of the seismic waveform data in the event candidate set, fuse the frequency domain statistical features and textual feature representations, and input the fused multimodal feature representation into the graph structure;

[0047] Step 5.1: Construct a graph structure by treating each sensor as a graph node. Based on the physical distance between vibration sensors and the cross-correlation between their corresponding signals, the weights of the edges are calculated to construct the adaptive graph structure. The formula is as follows: ,in, This represents the edge weight between nodes i and j; Indicates cross-correlation. The distance between nodes. For weighted parameters; Represents the feature vector of node i; This represents the feature vector of node j;

[0048] Step 5.2: Perform a short-time Fourier transform on the seismic waveform data of each event candidate set to generate the corresponding time-frequency spectrum. The formula for the short-time Fourier transform (STFT) is: ,in, This represents the spectral intensity at time t and frequency f; Indicates a time signal; This represents a window function centered at point t. Indicates a complex exponential basis; Indicates frequency; Represents the integral variable;

[0049] Step 5.3: Extract frequency domain statistical features based on time-spectrum graph Mean frequency The calculation formula is: ;

[0050] clock speed The calculation formula is: ;

[0051] Step 5.4: Obtain metadata information related to the vibration sensor and input it into Large Language Models (LLMs) to generate a semantic description of the current signal environment;

[0052] Step 5.5: Input the semantic description text into the embedding layer of the large language model for vectorized encoding to generate the semantic description text. By embedding model A 128-dimensional text feature representation is obtained. The formula is: ;

[0053] Step 5.6: Analyze the frequency domain statistical characteristics Text feature representation By splicing the data, a complete multimodal feature representation of each vibration sensor node is formed. And input the graph structure, the formula for multimodal feature representation is: ,in, The frequency domain statistical characteristics of the i-th vibration sensor node are extracted from the frequency domain analysis of the vibration signal to characterize the frequency correlation properties of the node's vibration. The text feature representation of the i-th vibration sensor node; Indicates the multimodal features after splicing Dimensions.

[0054] Step 6: Construct the spatiotemporal joint feature matrix; The temporal embedding sequence (time step × feature dimension) of each vibration sensor is processed through 4 layers of Transformer, and the multi-head attention mechanism is used to capture long-term dependencies. Timestamp information (accurate to milliseconds) is injected into the position encoding to obtain the spatiotemporal joint feature matrix;

[0055] The specific operation of step 6 is as follows:

[0056] Step 6.1: Construct a time-series input sequence by representing the features of each vibration sensor node within a continuous time window (time step × feature dimension), including the node features after frequency domain and semantic fusion, using the following formula: ,in, Represents the temporal feature sequence of node i; Indicates that sensor node i at time i Features of the location; Indicates the length of the time window;

[0057] Step 6.2: Process the temporal input sequence using a 4-layer Transformer encoder. Each Transformer layer includes a multi-head attention mechanism and a feedforward neural network to capture long-term dependencies and feature interactions in the vibration signal. Position encoding is added to each Transformer encoder layer, as shown in the formula: ,in, This represents the encoded features; This represents the original features of the i-th time-series input sequence; Indicates positional encoding;

[0058] The update process for each Transformer encoder layer is as follows: ,in, This represents the layer structure in the l-th layer Transformer encoder;

[0059] Step 6.3: Embed the timestamp accurate to milliseconds into the location encoding, and use it together with the input to participate in the attention weight calculation. The final output is a spatiotemporal joint feature matrix, providing temporal feature support for subsequent event classification and report generation. The formula for the output spatiotemporal joint feature matrix is: ,in, This represents the spatiotemporal joint feature matrix corresponding to the i-th vibration sensor node; This represents the final feature representation obtained after processing by the L-layer Transformer encoder.

[0060] Step 7: Event Classification and Report Generation; The spatiotemporal joint feature matrix is ​​passed through a fully connected layer to output the event probabilities (mine tremors / blasting / mechanical noise), which are then input into a classifier. The classification results and key features are input into LLMs to generate a structured report.

[0061] The specific operation of step 7 is as follows:

[0062] The spatiotemporal joint feature matrix is ​​pooled and then input into the classifier: , ,in, Represents the spatiotemporal joint characteristic matrix The pooled feature vector obtained after performing the pooling operation. This indicates a pooling operation. This represents the predicted probability distribution of the classifier output. This represents the Softmax activation function. This represents the weight matrix of the linear transformation in the classifier. This represents the bias term of the linear transformation in the classifier.

[0063] Input the concatenated classification results and features. : ,in, This represents the predicted probability distribution output by the previous classifier; This represents the complete multimodal feature representation of the i-th vibration sensor node; That is, the feature vector after pooling.

[0064] This invention addresses the need for intelligent vibration event identification in mining environments by deploying multiple vibration sensing nodes in the monitoring area to form a distributed sensor network. Each sensor continuously collects raw triaxial acceleration signals (including P-wave and S-wave components), generating continuous vibration waveform data. The collected data undergoes a multi-stage processing flow, sequentially completing steps such as signal denoising, event detection, feature extraction, graph structure construction, spatiotemporal feature modeling, and classification decision-making. Based on this, semantic auxiliary information generated by a large language model is combined to further enhance the model's ability to discriminate complex vibration events.

[0065] Table 1 Comparison of performance parameters before and after model improvement

[0066]

[0067] As shown in Table 1, compared with the traditional STA / LTA method, the present invention improves the accuracy from 82% to 96%, reduces the false alarm rate from 25% to 5%, and shortens the response delay from 2 seconds to 0.8 seconds.

[0068] Contents not described in detail in this specification are prior art known to those skilled in the art. Although illustrative specific embodiments of the invention have been described above to facilitate understanding by those skilled in the art, it should be understood that the invention is not limited to the scope of the specific embodiments. Various modifications are readily apparent to those skilled in the art as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of this invention are protected.

Claims

1. A method for intelligent identification of mine seismic activity based on multimodal deep learning and signal processing, characterized in that, The method includes the following steps: Step 1: Deploy multiple vibration sensors in the mining environment to form a mining sensor network. Collect seismic waveform data in real time through the mining sensor network. The seismic waveform data includes acceleration signals of P-waves and S-waves in the X, Y, and Z axes, and synchronizes timestamp information. Step 2: Perform noise suppression processing on the acquired seismic waveform data. Use the open-source seismology toolkit Obspy to read the raw acceleration waveform data and call the BandpassFilter module in Obspy to perform bandpass filtering on the data signal to filter out low-frequency terrain drift and high-frequency mechanical noise. Step 3: Divide the continuous waveform data signal into segments according to fixed time windows, and label the segmented data by combining historical event records or manual annotation methods to construct a supervised learning dataset; Step 4: Call the STA / LTA algorithm in Obspy to quickly scan the processed waveform data. Analyze the energy ratio between short-time average and long-time average to detect candidate abnormal vibration events and form a candidate event set. Step 5: Construct a graph structure for the seismic waveform data based on the graph neural network, extract the frequency domain statistical features and textual feature representations of the seismic waveform data in the event candidate set, fuse the frequency domain statistical features and textual feature representations, and input the fused multimodal feature representation into the graph structure; Step 5.1: Construct a graph structure by treating each sensor as a graph node. Based on the physical distance between vibration sensors and the cross-correlation between their corresponding signals, the weights of the edges are calculated to construct the adaptive graph structure. The formula is as follows: ,in, This represents the edge weight between nodes i and j; Indicates cross-correlation. The distance between nodes. These are weighted parameters; Represents the feature vector of node i; Represents the feature vector of node j; Step 5.2: Perform a short-time Fourier transform on the seismic waveform data of each event candidate set to generate the corresponding time-frequency spectrum. The formula for the short-time Fourier transform is: ,in, This represents the spectral intensity at time t and frequency f; Indicates a time signal; This represents a window function centered at time t. Indicates a complex exponential basis; Indicates frequency; Represents the integral variable; Step 5.3: Extract frequency domain statistical features based on time-spectrum graph Mean frequency The calculation formula is: ; clock speed The calculation formula is: ; Step 5.4: Obtain metadata information related to the vibration sensor and input it into the large language model to generate a semantic description of the current signal environment; Step 5.5: Input the semantic description text into the embedding layer of the large language model for vectorized encoding to generate the semantic description text. By embedding model A 128-dimensional text feature representation is obtained. The formula is: ; Step 5.6: Analyze the frequency domain statistical characteristics. Text feature representation By splicing the data, a complete multimodal feature representation of each vibration sensor node is formed. And input the graph structure, the formula for multimodal feature representation is: ,in, The frequency domain statistical characteristics of the i-th vibration sensor node are extracted from the frequency domain analysis of the vibration signal to characterize the frequency correlation properties of the node's vibration. The text feature representation of the i-th vibration sensor node; Indicates the multimodal features after splicing The dimension; Step 6: Construct the spatiotemporal joint feature matrix; The temporal embedding sequence of each vibration sensor is processed through 4 layers of Transformer, and the multi-head attention mechanism is used to capture long-term dependencies. Timestamp information is injected into the position encoding to obtain the spatiotemporal joint feature matrix. Step 7: Event Classification and Report Generation; The spatiotemporal joint feature matrix is ​​pooled through a fully connected layer and then input into the classifier. The classification results and key features are input into LLMs to generate a structured report.

2. The intelligent mine seismic identification method based on multimodal deep learning and signal processing according to claim 1, characterized in that, The raw acceleration waveform data signal acquired in step 2 is bandpass filtering The formula for processing is: , where h(t) represents the feature vector after pooling.

3. The intelligent mine seismic identification method based on multimodal deep learning and signal processing according to claim 2, characterized in that, short-time average in step 4 Long-term average Energy ratio The formula is: , , ,in, express The short-time energy at a given moment is calculated to capture the signal energy within a short window near the current moment and to capture instantaneous changes in the signal. express The long-term energy at a given time is calculated by measuring the signal energy within a long window near the current time, reflecting the average energy level of the background noise. Indicates the short-time window length; Indicates the length of the long-term window.

4. The intelligent mine seismic identification method based on multimodal deep learning and signal processing according to claim 3, characterized in that, The specific operation of step 6 is as follows: Step 6.1: Construct a time-series input sequence from the feature representations of each vibration sensor node within a continuous time window, using the following formula: ,in, Represents the temporal feature sequence of node i; Indicates that sensor node i at time i Features of the location; Indicates the length of the time window; Step 6.2: Process the timing input sequence using a 4-layer Transformer encoder, adding position encoding to each layer of the Transformer encoder, with the following formula: ,in, Represents the encoded features; Indicates positional encoding; The update process for each Transformer encoder layer is as follows: ,in, This represents the layer structure in the l-th layer Transformer encoder; Step 6.3: Embed the timestamp accurate to milliseconds into the location encoding, and use it together with the input to participate in the attention weight calculation. The final output is a spatiotemporal joint feature matrix, providing temporal feature support for subsequent event classification and report generation. The formula for the output spatiotemporal joint feature matrix is: ,in, This represents the spatiotemporal joint feature matrix corresponding to the i-th vibration sensor node; This represents the final feature representation obtained after processing by the L-layer Transformer encoder.

5. The intelligent mine seismic identification method based on multimodal deep learning and signal processing according to claim 4, characterized in that, The specific operation of step 7 is as follows: The spatiotemporal joint feature matrix is ​​pooled and then input into the classifier: , ,in, Represents the spatiotemporal joint characteristic matrix The pooled feature vector obtained after performing the pooling operation. This indicates a pooling operation. This represents the predicted probability distribution of the classifier output. This represents the Softmax activation function. This represents the weight matrix of the linear transformation in the classifier. This represents the bias term of the linear transformation in the classifier; Input the concatenated classification results and features. : ,in, This represents the predicted probability distribution output by the previous classifier; This represents the complete multimodal feature representation of the i-th vibration sensor node; That is, the feature vector after pooling.

Images