Micro-seismic event pickup method based on deep reinforcement learning
Through the microseismic event pickup method based on deep reinforcement learning, including data preprocessing and deep Q network segmentation module, the problem of low accuracy in traditional methods under complex geological conditions is solved, and high accuracy and high efficiency microseismic events are achieved Identify.
Patent Information
- Application Number
- CN202411358401.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2025-02-18
AI Technical Summary
Traditional microseismic event picking methods are difficult to accurately identify microseismic events under complex geological conditions. They are affected by underground noise interference and signal amplitude changes, and their identification accuracy is not high.
A microseismic event pickup method based on deep reinforcement learning is adopted, including data preprocessing, deep Q network (DQN) enhanced model segmentation module and deep refinement inference module. Data preprocessing is done through noise filtering, signal enhancement and feature extraction, and the DQN module performs preliminary identification and segmentation, and the deep refinement inference module further refines and confirms the exact moment of the microseismic event.
It significantly improves the accuracy and efficiency of identification of microseismic events, can accurately identify microseismic events under complex geological conditions, and enhances the support capabilities of geological disaster warning and underground resource development.
Smart Images
Figure CN119474708A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of seismic data processing, and in particular relates to a microseismic event picking method based on deep reinforcement learning. Background Art
[0002] Microseismic events refer to seismic activities caused by tiny displacements of underground rocks or rock formations under the action of geomechanics. They are common in underground engineering construction, such as tunnel excavation. With the continuous expansion of underground space development and utilization, the stability of rock layers has received more and more attention. Interference factors in the process of tunnel construction, such as construction vibration and mechanical operation, may cause a series of geological problems, such as local instability of rock formations and microseismic activities, which may affect engineering safety and construction efficiency. Therefore, effective monitoring of microseismic events is of great significance for the safety management and efficiency improvement of underground projects.
[0003] Phase picking of microseismic events is a very challenging problem in microseismic monitoring. However, traditional methods for picking microseismic events are mainly based on threshold setting and signal feature extraction. These methods are often affected by factors such as underground noise interference and signal amplitude changes, resulting in low recognition accuracy. These methods usually require manual setting of thresholds, and their applicability is limited by the geological environment. At the same time, the amplitude of underground noise is similar to that of microseismic signals, and traditional methods are difficult to accurately identify microseismic events. In addition, traditional methods usually rely on empirical feature extraction, which may lead to poor extraction results under complex geological conditions. Therefore, it is particularly urgent to develop a new method that can accurately pick up microseismic events and adapt to complex geological conditions. Summary of the invention
[0004] Purpose of the invention: In order to solve the problems existing in the above-mentioned prior art, the present invention provides a microseismic event picking method based on deep reinforcement learning.
[0005] Technical solution: The present invention provides a method for picking microseismic events based on deep reinforcement learning, which specifically includes the following steps:
[0006] Step 1: Obtain microseismic signals, construct a microseismic signal dataset, and preprocess each microseismic signal in the dataset;
[0007] Step 2: Use the enhanced segmentation module to perform preliminary recognition and segmentation on the preprocessed signal to identify the potential start and end points of the microseismic event;
[0008] Step 3: Based on the potential start and end points of the microseismic events in step 2, the deep refinement reasoning module is used to refine and confirm the exact time of the microseismic events.
[0009] Furthermore, the preprocessing of each microseismic signal in step 1 specifically includes: noise filtering, signal enhancement and feature extraction.
[0010] Furthermore, the step 1 is specifically as follows:
[0011] Step 1.1: Randomly select k microseismic signals in the data set and add synthetic noise of varying lengths to each selected microseismic signal;
[0012] Step 1.2: Standardize the microseismic signals in the data set by removing the mean of the signal and dividing it by its standard deviation to limit the dynamic range of the signal to a certain threshold;
[0013] Step 1.3: Extract the envelope of the microseismic signal to enhance the instantaneous features in the microseismic signal to highlight the instantaneous amplitude changes of the signal;
[0014] Step 1.4: Perform low-pass filtering on the microseismic signal to weaken high-frequency noise and retain the typical frequency characteristics of the microseismic event.
[0015] Furthermore, in step 1.3, Hilbert transform is used to extract the envelope of the microseismic signal; and in step 1.4, Butterworth low-pass filter is used to perform low-pass filtering on the microseismic signal.
[0016] Furthermore, in step 2, the enhanced segmentation module uses a deep Q network to perform preliminary recognition and segmentation on the preprocessed signal.
[0017] Furthermore, the deep Q network uses the ε-greedy strategy for action selection during training, and the ε-greedy strategy is expressed as:
[0018]
[0019] Among them, s represents the state, represents the set of all actions in state s, ε represents the parameter of the ε-greedy strategy, ξ represents a random number, and Q(s,a) represents the expected return that the agent can obtain by taking action a in state s and following the optimal strategy;
[0020] The deep Q network uses a prioritized experience replay mechanism to sort and sample experience tuples according to the priority of experience during training;
[0021] The deep Q network uses importance sampling weights during training to correct for the bias caused by non-uniform probability sampling. These weights are adjusted according to the inverse of the probability of being drawn empirically and then normalized.
[0022] The deep Q network uses a three-layer fully connected neural network to calculate the Q value:
[0023] Q(s,a)=RELU(W3·RELU(W2·RELU(W1·s+b1)+b2)+b3)
[0024] Among them, RELU is the activation function, W1, W2, W3 are weight matrices, and b1, b2, b3 are bias vectors.
[0025] Furthermore, the deep refinement reasoning module uses a deep neural network, specifically:
[0026] Step 3.1: Use a convolutional neural network to extract features of the input time series data, where the convolutional neural network includes L convolutional layers, and each convolutional layer in the first L-1 convolutional layers is connected to the corresponding maximum pooling layer;
[0027] Step 3.2: Convolutional neural network output X = (x1, x2, ..., x t ,…,x T' ) is input to the encoder, and the encoder outputs the feature sequence H = (h1,h2,…,h t ,…,h T' ), T' represents the total length of the encoder output sequence, and introduces the attention mechanism. At each decoding step, the attention mechanism calculates a weighted context vector c t As a weighted representation of the decoder input sequence:
[0028]
[0029] Among them, S t represents the hidden state of the decoder, T represents the transpose, d represents the scaling factor, and the context vector is used to update the hidden state of the decoder;
[0030] Step 3.4: Set up independent first and second branches, the first branch is used to predict the first arrival time of the microseismic event, and the second branch is used to predict the end time of the microseismic event; both branches are composed of two fully connected layers.
[0031] Furthermore, the loss function of the depth refinement reasoning module is:
[0032]
[0033] Among them C A Represents Z A The number of channels, Z A is the first arrival time extracted from the first branch, C B Represents Z B The number of channels, Z B is the end time of the second branch extraction, y A,a1 represents the actual label of the A1th channel at the first arrival time, p A,a1represents the predicted label of the a1th channel at the first arrival time, y A,b1 represents the actual label of the b1th channel at the end time, p A,b1 represents the predicted label for the b1th channel in the end time.
[0034] Furthermore, when training the microseismic event picking model composed of the enhanced segmentation module and the deep refinement reasoning module, all network parameters are initialized using He, and the learning rate scheduling adopts the exponential decay method.
[0035] Beneficial effects:
[0036] (1) The data preprocessing process significantly reduced the impact of background noise and enhanced the significance of microseismic event characteristics. This process includes noise filtering, signal enhancement, and feature extraction, laying a solid foundation for subsequent high-quality feature extraction and event identification, and ensuring the high quality of model input data.
[0037] (2) The present invention adopts a DQN enhanced model segmentation module and a deep refinement reasoning module. This method effectively improves the accuracy and efficiency of event picking through hierarchical processing, and makes the entire process more systematic and modular. The data preprocessing module is responsible for cleaning and standardizing input data, the DQN enhanced model segmentation module is used for preliminary identification and segmentation of microseismic events, and the deep refinement reasoning module further refines and confirms the picking results.
[0038] (3) The deep reinforcement learning method is introduced to the problem of automatic picking of microseismic events. Through this method, the event features are deeply learned and accurate recognition of microseismic events is achieved. This method uses the powerful feature extraction and nonlinear mapping capabilities of deep neural networks to capture subtle changes in microseismic signals, thereby improving the accuracy and robustness of recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 It is the overall framework diagram of the microseismic event picking method of the present invention;
[0040] Figure 2 It is a schematic diagram of the deep Q network;
[0041] Figure 3 This is a training flow chart of the microseismic event picking method of the present invention;
[0042] Figure 4 This is an output result diagram generated by the refined model in this example. DETAILED DESCRIPTION
[0043] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0044] This application is mainly to overcome the limitations of traditional microseismic event picking methods and propose a microseismic event picking method based on deep reinforcement learning. This application is implemented in the Pycharm environment using the Python language. Figure 1 is a framework diagram of a microseismic event picking method based on deep reinforcement learning in an embodiment of the present application, Figure 1 The system includes a data preprocessing module, a DQN reinforcement model segmentation module, and a deep refinement reasoning module. The DQN reinforcement model segmentation module uses a deep Q network (DQN) for preliminary recognition and event segmentation, and uses the strategy adaptability of reinforcement learning to automatically adjust the recognition strategy. In the deep refinement reasoning module, the deep neural network structure, especially the convolutional shared layer and the attention mechanism, is used to refine the preliminary segmented signals and accurately extract the start and end points of the microseismic events.
[0045] like Figure 1 As shown, the microseismic event picking method based on deep reinforcement learning described in this application includes:
[0046] Step S1: Data preprocessing module, including noise filtering, signal enhancement and feature extraction, to improve the signal-to-noise ratio and feature significance of microseismic signals. The data preprocessing module includes standardization, Hilbert transform and Butterworth low-pass filter.
[0047] Specifically, step S1 includes:
[0048] Step S11: randomly select k signals in the data set and add synthetic noise of different lengths to each signal. The randomness of the noise length ensures the robustness of the processing process and can simulate the uncertainty in field monitoring.
[0049] Step S12: Standardize the synthesized signal, remove the mean of the signal and divide it by its standard deviation, and limit the dynamic range of the signal to a certain threshold to improve the comparability between different signals and the generalization ability of the model:
[0050]
[0051] Z is the signal after standardization, x is the processed signal, μ and σ are the mean and standard deviation of the signal respectively;
[0052] Step S13: Hilbert transform is used to extract the envelope of the signal and to enhance the instantaneous features in the microseismic signal to highlight the instantaneous amplitude change of the signal, which is expressed as:
[0053]
[0054] f(t) is the signal before Hilbert transform, that is, the signal before Hilbert transform, and f(s) represents the signal after transform. is the Hilbert transform, s represents the signal variable after the transform, and t represents the signal variable before the transform.
[0055] Step S14: Use a Butterworth low-pass filter to perform low-pass filtering on the signal to weaken high-frequency noise and retain the typical frequency characteristics of microseismic events.
[0056] Step S2: Use the enhanced model segmentation module of the deep Q network (DQN) to perform preliminary recognition and segmentation on the preprocessed signal to identify the potential start and end points of microseismic events.
[0057] Specifically, Figure 2 The step S2 shown includes:
[0058] Step S21: Use the ε-greedy strategy to select actions, and use a parameter ε (a small probability value) to balance exploration and utilization. The ε-greedy strategy is expressed as:
[0059]
[0060] A(s) represents the set of all possible actions in state s, Q(s,a) represents the expected reward that the agent can obtain by taking action a in state s and following the optimal strategy. ξ is a random number, and random action from A(S) means that the agent randomly selects an action from the set of all possible actions A(S) in state s.
[0061] Step S22: The priority experience playback mechanism sorts and samples according to the priority of the experience, and assigns a priority P(i) = |δ i |+ε, where r is the reward, Q(s',a') is the maximum estimated value of the Q value for all possible actions a' in the new state s', i represents the i-th experience tuple, and γ is a discount factor used to measure the value of future rewards relative to immediate rewards.
[0062] Step S23: Use importance sampling weights (ISW) to correct the bias caused by non-uniform probability sampling. These weights are adjusted according to the inverse of the probability of being drawn empirically and then normalized:
[0063]
[0064] Where N is the size of the replay buffer, β is a parameter that can be slowly increased to 1 during the training process to gradually reduce the impact of the bias, and ω represents the normalized weight.
[0065] Step S24: A three-layer fully connected neural network is used to approximate the Q value function to achieve the learning of the optimal action strategy in a complex environment. The neural network structure is expressed as:
[0066] Q(s,a)=RELU(W3·RELU(W2·RELU(W1·s+b1)+b2)+b3)
[0067] Among them, W1, W2, W3 are weight matrices, b1, b2, b3 are bias vectors, and RELU is the activation function.
[0068] Step S3: The deep refinement reasoning module further processes the potential events output by the DQN, refines and confirms the exact moment of the microseismic event through a deep neural network, and improves the accuracy and reliability of recognition.
[0069] Specifically, step S3 includes:
[0070] Step S31: Use convolutional neural network to extract the features of time series data. The convolution operation is expressed as:
[0071] Y (l) =RELU(Conv(X (l-1) ,W (l) ))
[0072] Among them, X (l-1) is the output of the l-1th layer of the convolutional neural network, W (l) is the convolution kernel of this layer, Y (l) It is the output of this convolutional layer.
[0073] At the same time, in order to retain the most significant features and improve the robustness and computational efficiency of the model, a maximum pooling operation is introduced after each convolution layer in the first L-1 convolution layers:
[0074] p (l-1) =MaxPool(Y (l-1) )
[0075] Among them, Y (l-1) is the output of the upper convolution, p (l-1) is the pooling result.
[0076] Step S32: The attention layer uses the encoder to receive the output of the convolutional neural network X = (x1, x2, ..., x T' ), generating hidden state H = (h1,h2,…,h T' ), T' represents the time series length of x; at each decoding step, the attention mechanism calculates a set of weights, and after calculating the attention weights, a weighted context vector c is calculated t As the encoder input sequence weighted representation:
[0077]
[0078] Among them, S t represents the hidden state of the decoder, T represents the transpose, d represents the scaling factor, and the context vector is used to update the hidden state s of the decoder t :
[0079] s t =g(s t-1 ,y t-1 ,c t )
[0080] y t =softmax(h1(s t ,c t ))
[0081] Among them, h1 is the output layer function of the decoder, y t is the output of the decoder, and g(.) is the state update function.
[0082] Step S33: After attention weighting, the features output by the attention layer are transmitted to two independent branches, each of which consists of two fully connected layers, for identifying and predicting the first arrival time and end time of microseismic events.
[0083]
[0084] Z A is the first arrival time extracted from the first branch, Z B is the end time of the second branch extraction, represents the parameters of the fully connected network, Indicates bias.
[0085] like Figure 3 As shown, in this embodiment, the preliminary signal segmentation intelligent agent is trained by a deep Q network (DQN) algorithm, and the training process includes:
[0086] The first real data set is divided into training set, validation set and test set according to the ratio of 8:1:1, and then training, validation and testing are started.
[0087] The intelligent agent includes an input layer, multiple hidden units (such as fully connected layers or convolutional layers), and an output layer. The number of nodes in the output layer corresponds to the possible action space. In order to avoid the correlation between samples affecting the stability of training, the experience replay mechanism is used to randomly extract historical interaction data for training, and the target network is introduced to regularly update the weights to stabilize the Q value estimation.
[0088] At each time step, the intelligent agent selects an action based on the current observed state, and receives rewards and the next state from the environment after execution. The Q value is updated according to the Bellman equation, and the network parameters are adjusted through back propagation to minimize the gap between the Q value prediction and the actual reward. By setting appropriate hyperparameters such as learning rate, discount factor, exploration rate, and using early stopping strategy to monitor the performance of the validation set, overfitting can be effectively prevented to ensure the robustness and generalization ability of the model.
[0089] In the refined signal processing stage, a convolutional neural network (Attention-CNN) with an attention mechanism is used to capture local features in the signal and accurately classify or reconstruct it. The model architecture includes multiple convolutional layers for feature extraction, an attention mechanism for highlighting important information, and a dual-branch fully connected layer for classification or reconstruction.
[0090] The loss function of the deep refinement reasoning module is expressed as:
[0091]
[0092] Among them, C A Represents Z A The number of channels, C B Represents Z B The number of channels, y A,a1 represents the actual label of the a1th channel at the first arrival time, p A,a1 represents the predicted label of the a1th channel at the first arrival time, y A,b1 represents the actual label of the b1th channel at the end time, p A,b1 represents the predicted label for the b1th channel in the end time.
[0093] In the embodiment, the training of the microseismic event picking model (deep Q network and deep refinement inference module) adopts the following detailed strategy to ensure the efficiency and stability of the model. All network parameters are initialized using He (also known as Kaiming initialization), which is achieved by initializing the weights to a random value sampled from a standard normal distribution multiplied by a specific scaling factor. The specific formula is: Among them, W represents the weight matrix, N represents the normal distribution, and n refers to the number of neurons, which is suitable for the ReLU activation function and helps to keep the activation distribution consistent during the training process. The learning rate scheduling adopts the exponential decay method, with a high initial learning rate and adjusted at a decay rate of 0.95 every 10 epochs, so as to quickly find the optimal parameter configuration in the early stage of training and fine-tune the model by reducing the learning rate in the later stage. In addition, to prevent the model from overfitting during the training process, an early stopping strategy is adopted, and the training data is divided into a training set and a validation set. The model is evaluated using the validation set after each epoch. If the loss on the validation set no longer decreases in several consecutive epochs (for example, 10 epochs), the training is stopped. Through these strategies, the model can accurately identify and pick up microseismic events in a complex and changeable geological environment, thereby providing effective support for geological disaster warning and underground resource development.
[0094] After the model is built, this embodiment conducts experimental simulation on the STEAD dataset and illustrates the performance of the model through the intersection over union (IoU) indicator. At the same time, after segmentation based on the deep Q network (DQN), the output results generated by the refinement model for each segmented block are as follows: Figure 4 The main experimental parameters are as follows: batch size: 32; learning rate: 1e-3; number of iterations: maximum 30, and model training is stopped when the early stopping threshold is exceeded.
[0095] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present invention will not further describe various possible combinations.
Claims
1. A microseismic event picking method based on deep reinforcement learning, characterized in that: The specific steps include: Step 1: Obtain microseismic signals, construct a microseismic signal dataset, and preprocess each microseismic signal in the dataset; Step 2: Use the enhanced segmentation module to perform preliminary recognition and segmentation on the preprocessed signal to identify the potential start and end points of the microseismic event; Step 3: Based on the potential start and end points of the microseismic events in step 2, the deep refinement reasoning module is used to refine and confirm the exact time of the microseismic events.
2. According to the method for picking microseismic events based on deep reinforcement learning in claim 1, it is characterized in that: The preprocessing of each microseismic signal in step 1 specifically includes: noise filtering, signal enhancement and feature extraction.
3. According to the method for picking microseismic events based on deep reinforcement learning in claim 1, it is characterized in that: The step 1 is specifically as follows: Step 1.1: Randomly select k microseismic signals in the data set and add synthetic noise of various lengths to each selected microseismic signal; Step 1.2: Standardize the microseismic signals in the data set by removing the mean of the signal and dividing it by its standard deviation to limit the dynamic range of the signal to a certain threshold; Step 1.3: Extract the envelope of the microseismic signal to enhance the instantaneous features in the microseismic signal to highlight the instantaneous amplitude changes of the signal; Step 1.4: Perform low-pass filtering on the microseismic signal to weaken high-frequency noise and retain the typical frequency characteristics of the microseismic event.
4. The method for picking microseismic events based on deep reinforcement learning according to claim 3 is characterized in that: In step 1.3, the envelope of the microseismic signal is extracted by using Hilbert transform; and in step 1.4, the microseismic signal is subjected to low-pass filtering by using Butterworth low-pass filter.
5. The method for picking microseismic events based on deep reinforcement learning according to claim 1, characterized in that: In step 2, the enhanced segmentation module uses a deep Q network to perform preliminary recognition and segmentation on the preprocessed signal.
6. A microseismic event picking method based on deep reinforcement learning according to claim 5, characterized in that: The deep Q network uses the ε-greedy strategy for action selection during training. The ε-greedy strategy is expressed as: Among them, s represents the state, represents the set of all actions in state s, ε represents the parameter of the ε-greedy strategy, ξ represents a random number, and Q(s,a) represents the expected return that the agent can obtain by taking action a in state s and following the optimal strategy; The deep Q network uses a prioritized experience replay mechanism to sort and sample experience tuples according to the priority of experience during training; The deep Q network uses importance sampling weights during training to correct for the bias caused by non-uniform probability sampling. These weights are adjusted according to the inverse of the probability of being drawn empirically and then normalized. The deep Q network uses a three-layer fully connected neural network to calculate the Q value: Q(s,a)=RELU(W3·RELU(W2·RELU(W1·s+b1)+b2)+b3) Among them, RELU is the activation function, W1, W2, W3 are weight matrices, and b1, b2, b3 are bias vectors.
7. The method for picking microseismic events based on deep reinforcement learning according to claim 1, characterized in that: The deep refinement reasoning module uses a deep neural network, specifically: Step 3.1: Use a convolutional neural network to extract features of the input time series data, where the convolutional neural network includes L convolutional layers, and each convolutional layer in the first L-1 convolutional layers is connected to the corresponding maximum pooling layer; Step 3.2: Convolutional neural network output X = (x1, x2, ..., x t ,…,x T ') is input to the encoder, and the encoder outputs the feature sequence H = (h1,h2,…,h t ,…,h T' ), T' represents the total length of the encoder output sequence, and introduces the attention mechanism. At each decoding step, the attention mechanism calculates a weighted context vector c t As a weighted representation of the decoder input sequence: Among them, S t represents the hidden state of the decoder, T represents the transpose, d represents the scaling factor, and the context vector is used to update the hidden state of the decoder; Step 3.4: Set up independent first and second branches, the first branch is used to predict the first arrival time of the microseismic event, and the second branch is used to predict the end time of the microseismic event; both branches are composed of two fully connected layers.
8. The method for picking microseismic events based on deep reinforcement learning according to claim 7, characterized in that: The loss function of the depth refinement inference module is: Among them C A Represents Z A The number of channels, Z A is the first arrival time extracted from the first branch, C B Represents Z B The number of channels, Z B is the end time of the second branch extraction, y A,a1 represents the actual label of the a1th channel at the first arrival time, p A,a1 represents the predicted label of the a1th channel at the first arrival time, y A,b1 represents the actual label of the b1th channel at the end time, p A,b1 represents the predicted label for the b1th channel in the end time.
9. The method for picking microseismic events based on deep reinforcement learning according to claim 1, characterized in that: When training the microseismic event picking model composed of the enhanced segmentation module and the deep refinement reasoning module, all network parameters are initialized using He, and the learning rate scheduling adopts the exponential decay method.