Geological disaster prediction method based on transformer and deep learning

Abstract

The invention relates to the technical field of geological engineering, in particular to a geological disaster prediction method based on transformer and deep learning, which comprises the following steps: collecting and processing geological information, and deleting abnormal values in a geological information data set; and signal processing: determining an embedding dimension and delay time, processing historical geological disaster data by using a power system reconstruction method, reconstructing a weight matrix, and converting a one-dimensional time sequence into a track in a high-dimensional phase space, thereby obtaining periodic changes of geological disasters. According to the method, weight parameters and bias parameters in a model can be effectively optimized by applying an Adam optimizer, so that the purpose of minimizing loss is achieved, meanwhile, an effective model can be obtained by training the model, and the capability of the effective model can be evaluated by utilizing the models, so that the effective model can be compared with other models, and the accuracy of the model is improved. By adopting the method, the defects that a theoretical model adopts various assumptions, the universality is poor and advanced model parameters are difficult to calibrate can be overcome.

Description

Technical Field

[0001] The present invention relates to the field of geological engineering technology, and in particular to a geological disaster prediction method based on transformer and deep learning. Background Art

[0002] Geotechnical engineering geological disaster prediction is a technical method for predicting and warning geological disasters through comprehensive investigation, analysis and evaluation of the geological environment. It aims to quickly and accurately identify and analyze geological disasters, reduce the burden of manual investigation, and provide important support for geotechnical engineering planning, design and construction, and reduce the risks and losses of geological disasters. However, the existing geotechnical engineering geological disaster prediction methods have technical problems such as the complex internal evolution law of the geological disaster system and nonlinear correlation, which makes it difficult to predict the periodic law of geological disasters; there are technical problems such as the prediction of geological disasters involves multiple characteristic variables and a large number of data samples, and the prediction effect and model generalization ability are poor; there are technical problems such as uncertainty in the process of geological disaster prediction, the combined influence of multiple factors, and the diverse changes in the geological environment, and the uncertainty of the geological disaster classification results. Summary of the invention

[0003] In view of the above defects or deficiencies in the prior art, the present application aims to provide a geological disaster prediction method based on transformer and deep learning, the method comprising the following steps:

[0004] Step 1: Collect and process geological information, and delete outliers in the geological information data set; signal processing, specifically determining the embedding dimension and delay time, using the dynamic system reconstruction method to process historical geological disaster data, reconstructing the weight matrix, and converting the one-dimensional time series into a trajectory in the high-dimensional phase space, thereby obtaining the periodic changes of geological disasters;

[0005] Step 2: Feature processing and extraction

[0006] Use a one-dimensional convolutional neural network (CNN) to convolve the power sequence and extract power features based on the convolution kernel in the convolution.

[0007] In this step, the formula of the one-dimensional convolutional neural network is: V = g(W*U+b), where: g represents the activation function. The introduction of the activation function can enhance the nonlinear expression ability of the model and make the model more expressive; * represents convolution; W represents the weight matrix in the convolution kernel; b represents the bias term of the weight matrix in the convolution kernel.

[0008] Step 3: Geological disaster prediction based on LSTM and transformer

[0009] Combine LSTM and transformer to build the transformerBCL model;

[0010] First, there is the input layer, which inputs data, followed by two one-dimensional convolutional layers, each consisting of 16 convolutional kernels of size 4, with a step size of 1 and a linear activation function. After the two convolutional layers extract features from the data, it passes through two bidirectional LSTM layers, which are also the core of the Bi-LSTM model. Finally, there are two fully connected layers. The first fully connected layer integrates all the features of the upper layer, and finally outputs the decomposition results through the fully connected layer of N=1.

[0011] Step 4: Model training, the training steps include:

[0012] a. Parameter initialization: Parameters such as weights and learning rates need to be set before they can be correctly applied to the model;

[0013] b. Establish the loss function and set the stopping condition: The loss function is an important learning criterion that can be used to measure the performance of a network and is often related to the optimization amount. The mean square error loss function is used to calculate the mean square error between them. The calculation formula is as follows:

[0014]

[0015] Where N is the length of the input sequence, is the predicted value of the model, y i is the true value;

[0016] c. By applying the Adam optimizer, the weight parameters and bias parameters in the model can be effectively optimized;

[0017] d. By carefully adjusting the parameters and functions, an effective network training process can be achieved. That is, according to the pre-set number of training times, the parameters of the model are continuously adjusted until a certain threshold is reached. In this way, even if the expected number of training times is not reached, the model can stop running when the threshold is met. After 3 epochs, if the loss function has not been significantly improved, the deep learning model will be terminated.

[0018] Furthermore, the basic architecture of the LSTM consists of five parts: input x t 、output h t , Update gate i t 、Forget gate t , output gate o t ,in:

[0019] The expression of the forget gate f_t is as follows:

[0020] ft =σ(W f *[h t-1 ,x t ]+b f )

[0021] The expression of update gate i_t is as follows:

[0022] i t =σ(W i *[h t-1 ,x t ]+b i )

[0023] c′ t =tanh(W c -[h t-1 ,x t ]+b c )

[0024] c t =f t *c t-1 +i t *c′ t

[0025] The expression of the output gate o_t is as follows:

[0026] o t =σ(W o *[h t-1 ,x t ]+b o )

[0027] h t =o t *tanh(c t ).

[0028] Compared with ordinary RNN, LSTM can better solve the problems of gradient vanishing and gradient exploding when processing long data sequences, and can also better mine the more valuable hidden information in the previous and next correlations in the data.

[0029] In the present invention, the workflow of extracting useful information from features is shown in the figure above. The sequence element value is Value, the sequence element address is Key, and the query element is called Query. First, the similarity between Query and the element value Value in each Key is calculated to obtain the weight coefficient; then the weight coefficient is normalized using the softmax function; finally, the weight and Value are weighted and summed to obtain the Attention value.

[0030] The beneficial effects of the present invention are as follows:

[0031] The present application provides a geological disaster prediction method based on transformer and deep learning. By applying the Adam optimizer, the weight parameters and bias parameters in the model can be effectively optimized to achieve the purpose of minimizing the loss. At the same time, an effective model can be obtained by training the model, and these models can be used to evaluate their capabilities for comparison with other models. This method can overcome the defects of theoretical models adopting multiple assumptions, poor universality and difficulty in calibrating advanced model parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on the provided drawings without creative work:

[0033] Figure 1 It is a structural block diagram of the LSTM unit of the present invention;

[0034] Figure 2 It is a structural block diagram of the transformer of the present invention;

[0035] Figure 3 A summary of the present invention is shown in FIG.

[0036] Figure 4 It is the transformerBCL model of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] like Figure 1 , Figure 3 as well as Figure 4 As shown, this application aims to provide a geological disaster prediction method based on transformer and deep learning, comprising the following steps:

[0039] Step 1: Collect and process geological information, and delete outliers in the geological information data set; signal processing, specifically determining the embedding dimension and delay time, using the dynamic system reconstruction method to process historical geological disaster data, reconstructing the weight matrix, and converting the one-dimensional time series into a trajectory in the high-dimensional phase space, thereby obtaining the periodic changes of geological disasters;

[0040] Step 2: Feature processing and extraction

[0041] Use a one-dimensional convolutional neural network (CNN) to convolve the power sequence and extract power features based on the convolution kernel in the convolution.

[0042] In this step, the formula of the one-dimensional convolutional neural network is: V = g (W * U + b), where: g represents the activation function. The introduction of the activation function can enhance the nonlinear expression ability of the model and make the model more expressive; * represents convolution; W represents the weight matrix in the convolution kernel; b represents the bias term of the weight matrix in the convolution kernel;

[0043] Step 3: Geological disaster prediction based on LSTM and transformer

[0044] Combine LSTM and transformer to build the transformerBCL model;

[0045] First is the input layer, which inputs data, followed by two one-dimensional convolutional layers, each consisting of 16 convolutional kernels of size 4, with a step size of 1 and a linear activation function. After two convolutional layers, the data is feature extracted and then passes through two bidirectional LSTM layers, which are also the core of the Bi-LSTM model. Finally, there are two fully connected layers. The first fully connected layer integrates all the features of the upper layer, and finally outputs the decomposition result through a fully connected layer with N=1.

[0046] Step 4: Model training, the training steps include:

[0047] a. Parameter initialization: Parameters such as weights and learning rates need to be set before they can be correctly applied to the model;

[0048] b. Establish the loss function and set the stopping condition: The loss function is an important learning criterion that can be used to measure the performance of a network and is often related to the optimization amount. The mean square error loss function is used to calculate the mean square error between them. The calculation formula is as follows:

[0049]

[0050] Where N is the length of the input sequence, is the predicted value of the model, yi is the true value;

[0051] c. By applying the Adam optimizer, the weight parameters and bias parameters in the model can be effectively optimized;

[0052] d. By carefully adjusting the parameters and functions, an effective network training process can be achieved. That is, according to the pre-set number of training times, the parameters of the model are continuously adjusted until a certain threshold is reached. In this way, even if the expected number of training times is not reached, the model can stop running when the threshold is met. After 3 epochs, if the loss function has not been significantly improved, the deep learning model will be terminated.

[0053] In this invention application, the basic architecture of LSTM consists of five parts: input x t 、output h t , Update gate i t 、Forget gate t , output gate o t ,in:

[0054] The expression of the forget gate f_t is as follows:

[0055] f t =σ(W f *[h t-1 ,x t ]+b f )

[0056] The expression of update gate i_t is as follows:

[0057] i t =σ(W i *[h t-1 ,x t ]+b i )

[0058] c′ t =tanh(W c *[h t-1 ,x t ]+b c )

[0059] c t =f t *c t-1 +i t *c′ t

[0060] The expression of the output gate o_t is as follows:

[0061] o t =σ(W o *[h t-1 ,xt ]+b o )

[0062] h t =o t *tanh(c t ).

[0063] Compared with ordinary RNN, LSTM can better solve the problems of gradient vanishing and gradient exploding when processing long data sequences, and can also better mine the more valuable hidden information in the previous and next correlations in the data.

[0064] In the present invention, the workflow for extracting useful information from features is as follows: Figure 2 As shown in the figure, the sequence element value is Value, the sequence element address is Key, and the query element is called Query. First, the similarity between Query and the element value Value in each Key is calculated to obtain the weight coefficient; then the weight coefficient is normalized using the softmax function; finally, the weight and Value are weighted and summed to obtain the Attention value.

[0065] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A geological disaster prediction method based on transformer and deep learning, characterized in that: The steps include: Step 1: Collect and process geological information, and delete outliers in the geological information data set; signal processing, specifically determining the embedding dimension and delay time, using the dynamic system reconstruction method to process historical geological disaster data, reconstructing the weight matrix, and converting the one-dimensional time series into a trajectory in the high-dimensional phase space, thereby obtaining the periodic changes of geological disasters; Step 2: Feature processing and extraction Use a one-dimensional convolutional neural network (CNN) to convolve the power sequence and extract power features based on the convolution kernel in the convolution. In this step, the formula of the one-dimensional convolutional neural network is: V = g (W * U + b), where: g represents the activation function. The introduction of the activation function can enhance the nonlinear expression ability of the model and make the model more expressive; * represents convolution; W represents the weight matrix in the convolution kernel; b represents the bias term of the weight matrix in the convolution kernel; Step 3: Geological disaster prediction based on LSTM and transformer Combine LSTM and transformer to build the transformerBCL model; First, there is the input layer, which inputs data, followed by two one-dimensional convolutional layers, each consisting of 16 convolutional kernels of size 4, with a step size of 1 and a linear activation function. After the two convolutional layers extract features from the data, it passes through two bidirectional LSTM layers, which are also the core of the Bi-LSTM model. Finally, there are two fully connected layers. The first fully connected layer integrates all the features of the upper layer, and finally outputs the decomposition results through the fully connected layer of N=1. Step 4: Model training, the training steps include: a. Parameter initialization: Parameters such as weights and learning rates need to be set before they can be correctly applied to the model; b. Establish the loss function and set the stopping condition: The loss function is an important learning criterion that can be used to measure the performance of a network and is often related to the optimization amount. The mean square error loss function is used to calculate the mean square error between them. The calculation formula is as follows: Where N is the length of the input sequence, is the predicted value of the model, y i is the true value; c. By applying the Adam optimizer, the weight parameters and bias parameters in the model can be effectively optimized; d. By carefully adjusting the parameters and functions, an effective network training process can be achieved. That is, according to the pre-set number of training times, the parameters of the model are continuously adjusted until a certain threshold is reached. In this way, even if the expected number of training times is not reached, the model can stop running when the threshold is met. After 3 epochs, if the loss function has not been significantly improved, the deep learning model will be terminated.

2. The geological disaster prediction method based on transformer and deep learning according to claim 1 is characterized in that: The basic architecture of the LSTM consists of five parts: input x t 、output h t , Update gate i t 、Forget gate t , output gate o t ,in: Forget Gate t The expression of is as follows: f t =σ(W f *[h t-1 ,x t ]+b f ) Update Gate t The expression of is as follows: i t =σ(W i *[h t-1 ,x t ]+b i ) c' t =tanh(W c *[h t-1 ,x t ]+b c ) c t =f t *c t-1 +i t *c′ t Output gate o t The expression of is as follows: the t =σ(W o *[h t-1 ,x t ]+b o ) h t =o t *tanh(c t )。

Images