Laboratory earthquake prediction method across multiple experimental devices based on transfer learning
The transfer learning method using deep convolutional neural networks and fine-tuning across different laboratory setups enhances earthquake prediction accuracy by leveraging pre-trained models to adapt to new data distributions, significantly reducing the data and computing resources required.
Patent Information
- Application Number
- CN202410433112.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-11
- Publication Date
- 2024-12-10
- Estimated Expiration
- 2044-04-11
AI Technical Summary
Current transfer learning methods in laboratory earthquake prediction are limited to migrating from numerical simulation to laboratory experiments, failing to effectively associate and fuse data from different types and conditions, limiting the depth and accuracy of earthquake prediction.
A transfer learning method using deep convolutional neural networks for pre-training and fine-tuning across multiple experimental devices, specifically adapting a model trained on one laboratory dataset to another by retaining the decoder structure and fine-tuning the regression layer with a smaller dataset, utilizing techniques like temporal convolutional networks and batch normalization.
The technical solution effectively addresses the challenge of cross-device laboratory earthquake prediction by enhancing model generalization, reducing data and computing requirements, and improving prediction accuracy.
Smart Images
Figure CN118330715B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of earthquake prediction, and in particular relates to a laboratory earthquake prediction method across multiple experimental devices based on transfer learning. Background Art
[0002] As a powerful machine learning technique, transfer learning has been widely used in various fields, especially image recognition and natural language processing. In the field of image recognition, transfer learning enables models to learn and transfer knowledge from a large number of images to recognize new, unseen objects. Similarly, in the field of natural language processing, this technology significantly improves the ability of language understanding and generation by learning language patterns from large text corpora. In the specific scientific field of laboratory earthquake prediction, the application of transfer learning is still in its infancy. Although there have been studies attempting to migrate from numerical simulations to laboratory experiments, the scope of this migration is relatively narrow. In contrast, applying transfer learning to migrate from one laboratory experiment to another is not only more technically challenging, but also more scientifically meaningful. This in-depth migration can deepen the present invention's understanding of the mechanism of earthquake occurrence and provide a more accurate and comprehensive method for predicting and mitigating earthquake disasters.
[0003] At present, the only attempt to apply transfer learning in the field of laboratory earthquakes is to migrate from numerical simulation to laboratory experiments (Wang, Johnson et al. 2021). The researchers first used the numerical simulation method Finite Discrete Element Method to simulate the double direct shear test and similar acoustic emission signals, and then pre-trained a convolutional neural network with the numerical simulation data set. This convolutional neural network is a classic variational autoencoder framework, which contains an encoder, a decoder, and the latent space in the middle. The data set generated by the numerical simulation is used to pre-train this model. Then the encoder and decoder parts remain unchanged, but the latent space is fine-tuned by laboratory data, so as to obtain a model that migrates from numerical simulation data to laboratory data. This is also the only attempt to transfer learning in the field of laboratory earthquake prediction so far.
[0004] Current technology has achieved a preliminary migration from numerical simulation to laboratory experiments, but the difficulty and limitations of this migration are still significantly smaller than the differences between different types of experiments. In order to further improve the efficiency and depth of experimental research, it is urgent to develop a new type of transfer learning method that can effectively associate, fuse and migrate laboratory test data of different types and conditions, and fully explore and utilize the inherent connections and laws between these heterogeneous data. Summary of the invention
[0005] The present invention aims to explore the in-depth application of transfer learning in the field of laboratory earthquake prediction, especially on how to effectively migrate from one experimental environment to another, and the impact of such migration on improving the accuracy of the prediction model.
[0006] The present invention proposes a transfer learning method for laboratory experiments, which is used to achieve efficient migration and fusion between laboratory test data of different types and under different conditions, so that the earthquake prediction knowledge obtained in one laboratory can be transferred to another laboratory earthquake experiment for prediction. This is a first, and no existing research has achieved this.
[0007] The specific technical solution is:
[0008] A laboratory earthquake prediction method based on transfer learning across multiple experimental devices first trains a large model of deep convolutional neural network on a laboratory test dataset to capture the key features of earthquake signals;
[0009] Subsequently, using fine-tuning techniques, the model was further adapted to another different category of laboratory test dataset, achieving effective knowledge transfer and model generalization.
[0010] The whole process can be divided into pre-training and fine-tuning. The following takes the transfer learning from biaxial test to double direct shear test as an example to illustrate the process of cross-device transfer training.
[0011] (1) Pre-training:
[0012] First, pre-train the model on the dataset of the first experiment;
[0013] The architecture of the model is a pure decoder (Decoder-only) architecture, which is divided into three parts: batch normalization layer (BatchNorm1d), decoder (Decoder) and regression layer (Regression Head).
[0014] Each layer in the decoder is a Temporal Convolutional Network (TCN) module (Bai, Kolter and Koltun 2018). The core computational component in each TCN module is the Dilated Causal Conv1D.
[0015] These temporal convolutional network modules at different layers set different dilation rates (Dilation factor), and the dilation factor for the i-th layer is set to 2^(i-1), where i=1,2,…,N, indicating the number of layers.
[0016] (2) Fine-tuning:
[0017] When migrating to the second experiment, the main structure of the pre-trained model, namely the decoder part, is retained.
[0018] The first half of the new type of laboratory seismic data was used to fine-tune the model's regression layer, while keeping the decoder parameters unchanged. During the fine-tuning process, the model learned to adapt to the new data distribution and the weights of the regression layer were optimized. A model suitable for analyzing acoustic emission data for the second test was obtained.
[0019] The technical effects of the present invention are as follows:
[0020] The cross-device laboratory earthquake prediction method of the present invention utilizes deep convolutional neural networks and fine-tuning technology to achieve efficient migration and fusion between laboratory test data of different types and under different conditions, and has the following beneficial effects:
[0021] 1. Significantly improve the generalization performance of earthquake prediction. The large model trained on the source laboratory data captures the key feature representation of the seismic signal. The knowledge is transferred to the target laboratory using fine-tuning technology, allowing the model to adapt to different data distributions.
[0022] 2. Significantly reduce the data requirements of the target laboratory. Since the model trained in the source laboratory has already learned the general characteristics of the seismic signal, only a small amount of data samples are needed for fine-tuning in the target laboratory to produce ideal prediction results. Experiments show that the amount of labeled data in the target laboratory required for fine-tuning is only about 70% of that required for training from scratch, effectively reducing the cost of data collection and annotation.
[0023] 3. Improve the efficiency of model training and tuning. Thanks to the excellent initial parameters obtained from training in the source laboratory, the parameters required for training are reduced by 97% compared to training from scratch. In addition, the fine-tuning process of the model in the target laboratory requires fewer computing resources, and an ordinary graphics card can meet the needs, while training from scratch usually requires multiple graphics cards to operate in parallel. At the same time, there are fewer hyperparameters to try during fine-tuning, so it is easier to tune, and the model is more robust in choosing hyperparameters.
[0024] 4. Provide a new technical route for laboratory earthquake prediction research. This invention is the first to introduce transfer learning into the cross-device problem of laboratory earthquake prediction, filling the gap in this field. The proposed method can be extended and applied to more types of experimental conditions and mechanism models, providing a new technical route and development direction for laboratory earthquake prediction research. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 Schematic diagram of a seven-layer decoder structure in an embodiment;
[0026] Figure 2 A data graph of prediction results of an embodiment in which the pre-trained model is directly applied to a target laboratory without fine-tuning;
[0027] Figure 3 This is a data diagram of the prediction results after adopting the fine-tuning strategy in the embodiment. DETAILED DESCRIPTION
[0028] The specific technical solution of the present invention is described in conjunction with the accompanying drawings.
[0029] The present invention is a laboratory earthquake prediction method across multiple experimental devices based on transfer learning. The method first trains a large model of a deep convolutional neural network on a laboratory test dataset to capture the key features of the earthquake signal. Subsequently, using fine-tuning technology, the model is further adjusted to adapt to another different category of laboratory test datasets to achieve effective knowledge transfer and model generalization. The whole process can be divided into pre-training and fine-tuning. The following takes the transfer learning from biaxial test to double direct shear test as an example to illustrate the process of cross-device transfer training.
[0030] (1) Pre-training:
[0031] First, the model is pre-trained on the dataset of the dual-axis test. The architecture of the model and the process of data preprocessing are as follows: The architecture of this model is a pure decoder (Decoder-only) architecture, which is divided into three parts: batch normalization layer (BatchNorm1d), decoder (Decoder) and regression layer (Regression Head).
[0032] The basis for implementing transfer learning is the multi-layer architecture of the decoder. Each layer in the decoder is a temporal convolutional network (TCN) module (Bai, Kolter and Koltun 2018). The core computing component in each temporal convolutional network module is the dilated causal convolution (Dilated Causal Conv1D). Causality means that the convolution operation can only see the information of the current time step and before, and cannot use future information, which ensures the temporal structure of the model. Dilation means that during the convolution operation, a certain number of values are skipped between two adjacent input values, which expands the perception field of the convolution kernel and enables the module to capture long-distance dependencies in sequence data more efficiently.
[0033] These temporal convolutional network modules at different layers are set with different dilation factors. The dilation factor for the i-th layer is set to 2^(i-1), where i=1,2,…,N, indicating the number of layers. Figure 1 The seven-layer decoder is shown, and the corresponding maximum expansion rate is 2^6,64. Therefore, as the number of decoder layers increases, the expansion rate increases exponentially, allowing the decoder's perception of long sequences to grow rapidly while keeping the total number of model parameters controllable. This setting also allows the model to be flexibly adjusted when migrating to different tasks. When migrating from long time series tasks to short time series tasks, the high-level temporal convolutional network modules can be discarded accordingly, because they have too large an expansion rate and are not suitable for short time series inputs, while retaining the low-level temporal convolutional network modules, and the time dependencies they learned from the pre-training dataset can be migrated to the new dataset for application.
[0034] (2) Fine-tuning:
[0035] When the model has learned enough knowledge in the pre-trained data set, this knowledge can be applied to new data sets and new tasks by means of fine-tuning. When migrating to the double direct shear test, the present invention retains the main structure of the pre-trained model, that is, the decoder part. Since the decoder adopts a multi-layer architecture, the higher the number of layers, the larger its perceptual field of view. Considering that the earthquake period of the double direct shear test is shorter than that of the biaxial test, during the migration process, the present invention only retains the first five layers of the temporal convolutional network of the decoder. The perceptual field of the last five layers is too large and is not suitable for extracting features of acoustic emission data in new types of laboratory earthquakes, so they are removed.
[0036] After that, the present invention uses the first half of the new type of laboratory seismic data to fine-tune the regression layer of the model while keeping the parameters of the decoder unchanged. During the fine-tuning process, the model learns to adapt to the new data distribution and optimizes the weights of the regression layer. In this way, the present invention obtains an acoustic emission data analysis model suitable for double direct shear tests.
[0037] It is worth mentioning that the universal feature representation learned by the pre-training-fine-tuning model can be applied to various tasks, such as time to failure, etc. Even if the pre-training dataset does not explicitly contain the annotation information of these tasks, the present invention can still achieve these goals through fine-tuning technology.
[0038] In summary, by adopting the pre-training-fine-tuning transfer learning strategy, the present invention can effectively utilize the model that has been trained in one field and quickly adapt it to another related field. This not only greatly reduces the amount of data and computing resources required to train the model from scratch in the new field, but also improves the generalization ability and robustness of the model.
[0039] The fine-tuning strategy adopted by the present invention is:
[0040] Since the decoder makes up the majority of the model and is frozen during fine-tuning, the fine-tuning process is much faster than fully training the model on a new dataset. For example, when migrating from biaxial testing to dual direct shear testing, the pre-trained model has a total of 19,537,357 parameters. By freezing the decoder parameters, the number of adjustable parameters (i.e., the number of parameters in the regression layer) is reduced to 647,629 (about 3.3%). Therefore, compared to training a large model from scratch, the amount of computation and computing power required during fine-tuning will be many orders of magnitude smaller.
[0041] Loss function: The goal of training is to optimize the model parameters to minimize the loss function. This minimization means that the difference between the model prediction and the actual data in the training set is minimized. Choosing a suitable loss function is crucial in this process. In practice, the present invention fixes the loss function to mean square error (MSE).
[0042] Backpropagation algorithm: Use the backpropagation algorithm to propagate this loss back through the neural network and calculate the gradient of the loss with respect to the model weights, which provides information on how to adjust these weights.
[0043] Learning Rate and Optimizer: Adjustments to model parameters are guided by the learning rate, which is a hyperparameter that determines the magnitude of these adjustments. A higher learning rate may allow the model to converge quickly, but there is also a risk of exceeding the minimum, while a lower learning rate ensures more accurate convergence at the expense of slower training. Therefore, optimizers are often used to efficiently and dynamically adjust the learning rate to minimize losses. The present invention trains the model using the AdamW optimizer, which is a variant of the Adam optimizer. AdamW improves model performance and computational efficiency by decoupling weight decay from the optimization step, thereby achieving more accurate gradient estimates.
[0044] Early stopping mechanism: The present invention also uses an early stopping mechanism to prevent overfitting, that is, if the loss on the validation set does not decrease within 10 consecutive cycles, the training is terminated. This method ensures that the model will not overfit.
[0045] Batch size: The batch size determines the number of samples that are forward propagated through the network at a time. A larger batch size allows the model to process more data at the same time, thereby speeding up training, but also requires more computing memory. In the case of this invention, the size is fixed to 64.
[0046] Pre-training-fine-tuning effect of the present invention:
[0047] In the cross-device laboratory earthquake prediction task, there are significant differences in the earthquake period between different types of laboratory earthquake data, which is caused by different experimental conditions (such as normal stress, displacement rate, etc.). This difference in earthquake period poses a challenge to the cross-device prediction ability of the model. It is often difficult to achieve ideal prediction results by directly applying the model trained in the source laboratory to the target laboratory.
[0048] However, by adopting the pre-training-fine-tuning transfer learning strategy, the present invention can effectively overcome the difficulties caused by the difference in earthquake periods. Figure 2 and Figure 3 As shown, Figure 2 This is the prediction result of applying the pre-trained model directly to the target laboratory without fine-tuning. It can be seen that the prediction performance is very poor. Figure 3 The prediction results after fine-tuning the strategy are shown, and the prediction ability of the model in the target laboratory is significantly improved.
[0049] This result shows that the pre-training-fine-tuning strategy can help the model adapt to the distribution of earthquake data under different experimental conditions and learn the specific patterns of earthquakes in the target laboratory, thereby achieving efficient and accurate cross-device earthquake prediction. This strategy makes full use of the universal earthquake feature representation learned by the pre-training model and optimizes the prediction performance of the model in the target laboratory through fine-tuning, providing an effective technical route for solving the problem of cross-device laboratory earthquake prediction.
Claims
1. Laboratory earthquake prediction method across multiple experimental devices based on transfer learning, It is characterized in that It is used to pre-train the model on the biaxial test dataset and migrate it to the double direct shear test; it includes the following steps: First, a large model of deep convolutional neural network is trained on the biaxial test dataset to capture the key features of seismic signals; Subsequently, using fine-tuning techniques, the model was further adapted to the laboratory test dataset of double direct shear tests to achieve effective knowledge transfer and model generalization; Including pre-training and fine-tuning, the specific steps are: (1) Pre-training: First, pre-train the model on the dataset of the biaxial test; The architecture of the model is a pure decoder architecture, which consists of three parts: batch normalization layer, decoder and regression layer; Each layer in the decoder is a temporal convolutional network module, and the core computing component in each temporal convolutional network module is the dilated causal convolution; Different expansion rates are set for the temporal convolutional network modules at different layers. The expansion factor for the i-th layer is set to 2^(i-1), where i = 1, 2, …, N, indicating the number of layers. (2) Fine-tuning: When migrating to the double direct shear test, the first five layers of the temporal convolutional network of the pre-trained model decoder are retained; The first half of the new type of laboratory seismic data is used to fine-tune the regression layer of the model while keeping the parameters of the decoder unchanged; during the fine-tuning process, the model learns to adapt to the new data distribution and optimizes the weights of the regression layer, resulting in an acoustic emission data analysis model suitable for double direct shear tests.