A geological disaster prediction method based on transformer and deep learning

Abstract

The present invention relates to the field of geological engineering technology, and specifically to a method for geological disaster prediction based on transformers and deep learning. The method includes collecting and processing geological information, removing outliers from geological information datasets, and performing signal processing, specifically determining the embedding dimension and delay time. The method uses a dynamic system reconstruction method to process historical geological disaster data, reconstructing a weight matrix, and converting one-dimensional time series into trajectories in a high-dimensional phase space, thereby obtaining the periodic changes of geological disasters. By applying the Adam optimizer, the method effectively optimizes the weight and bias parameters in the model to minimize losses. Simultaneously, by training the model, an effective model can be obtained, and these models can be used to evaluate their capabilities for comparison with other models. This method overcomes the shortcomings of theoretical models, such as the multiple assumptions they employ, the poor universality they offer, and the difficulty of calibrating advanced model parameters.

Description

Technical Field

[0001] The present invention relates to the field of geological engineering technology, and in particular to a geological disaster prediction method based on transformer and deep learning. Background Art

[0002] Geotechnical engineering geological hazard prediction is a technical method for predicting and warning of geological hazards through comprehensive investigation, analysis, and evaluation of the geological environment. It aims to quickly and accurately identify and analyze geological hazards, reduce the burden of manual investigations, provide important support for geotechnical engineering planning, design, and construction, and reduce the risks and losses of geological hazards. However, existing geotechnical engineering geological hazard prediction methods face technical challenges: the internal evolution of geological hazard systems is complex and nonlinear, making it difficult to predict the cyclical patterns of geological hazard occurrence; geological hazard prediction involves multiple characteristic variables and a large number of data samples, resulting in poor prediction results and model generalization capabilities; and the geological hazard prediction process faces uncertainty. Due to the combined influence of multiple factors and the diverse changes in the geological environment, the geological hazard classification results are uncertain. Summary of the Invention

[0003] In view of the above-mentioned defects or deficiencies in the prior art, this application aims to provide a geological disaster prediction method based on transformer and deep learning, which includes the following steps:

[0004] Step 1: Collect and process geological information and delete outliers in the geological information dataset; signal processing, specifically determining the embedding dimension and delay time, using the dynamic system reconstruction method to process historical geological disaster data, reconstructing the weight matrix, and converting the one-dimensional time series into trajectories in the high-dimensional phase space, thereby obtaining the periodic changes of geological disasters;

[0005] Step 2: Feature processing and extraction

[0006] Use a one-dimensional convolutional neural network (CNN) to convolve the power sequence and extract power features based on the convolution kernel.

[0007] In this step, the formula of the one-dimensional convolutional neural network is: V = g(W*U+b), where: g represents the activation function. The introduction of the activation function can enhance the nonlinear expression ability of the model and make the model more expressive; * represents convolution; W represents the weight matrix in the convolution kernel; b represents the bias term of the weight matrix in the convolution kernel.

[0008] Step 3: Geological disaster prediction based on LSTM and transformer

[0009] Combine LSTM and transformer to build the transformerBCL model;

[0010] First, the input layer takes in data, followed by two one-dimensional convolutional layers, each consisting of 16 convolution kernels of size 4, with a stride of 1 and a linear activation function. After the two convolutional layers extract features from the data, the data passes through two bidirectional LSTM layers, which are also the core of the Bi-LSTM model. Finally, there are two fully connected layers. The first fully connected layer integrates all the features of the previous layer, and finally the decomposition results are output through the fully connected layer with N=1.

[0011] Step 4: Model training. The training steps include:

[0012] a. Parameter initialization: Parameters such as weights and learning rates need to be set before they can be correctly applied to the model;

[0013] b. Establish the loss function and set the stopping condition: The loss function is an important learning criterion that can be used to measure the performance of a network and is often related to the optimization amount. The mean square error loss function is used to calculate the mean square error between them. The calculation formula is as follows:

[0014]

[0015] Where N is the length of the input sequence, is the predicted value of the model, y i is the true value;

[0016] c. By applying the Adam optimizer, the weight parameters and bias parameters in the model can be effectively optimized;

[0017] d. By carefully adjusting the parameters and functions, an effective network training process can be achieved. That is, according to the pre-set number of training times, the model parameters are continuously adjusted until a certain threshold is reached. In this way, even if the expected number of training times is not reached, the model can stop running when the threshold is met. After 3 epochs, if the loss function still does not improve significantly, the deep learning model will be terminated.

[0018] Furthermore, the basic architecture of the LSTM consists of five parts: input x t 、output h t , update gate i t 、Forget Gate t , output gate o t ,in:

[0019] The expression of the forget gate f_t is as follows:

[0020] ft =σ(W f *[h t-1 ,x t ]+b f )

[0021] The expression of update gate i_t is as follows:

[0022] i t =σ(W i *[h t-1 ,x t ]+b i )

[0023] c′ t =tanh(W c -[h t-1 ,x t ]+b c )

[0024] c t =f t *c t-1 +i t *c′ t

[0025] The expression of the output gate o_t is as follows:

[0026] o t =σ(W o *[h t-1 ,x t ]+b o )

[0027] h t =o t *tanh(c t ).

[0028] Compared with ordinary RNN, LSTM can better solve the problems of gradient vanishing and gradient exploding when processing long data sequences, and can also better mine the more valuable information hidden in the previous and next correlations in the data.

[0029] In this invention, the workflow for extracting useful information from features is shown in the figure above. The sequence element value is Value, the sequence element address is Key, and the query element is called Query. First, the similarity between Query and the element value in each Key is calculated to obtain the weight coefficient; then the weight coefficient is normalized using the softmax function; finally, the weight and Value are weighted and summed to obtain the Attention value.

[0030] The beneficial effects of the present invention are as follows:

[0031] This application provides a geological disaster prediction method based on transformer and deep learning. By applying the Adam optimizer, it can effectively optimize the weight parameters and bias parameters in the model, thereby achieving the purpose of minimizing losses. At the same time, by training the model, an effective model can be obtained, and these models can be used to evaluate their capabilities for comparison with other models. This method can overcome the defects of theoretical models that adopt multiple assumptions, have poor universality, and are difficult to calibrate advanced model parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only embodiments of the present invention. Those skilled in the art can also derive other drawings based on the provided drawings without inventive work.

[0033] Figure 1 This is a block diagram of the LSTM unit structure of the present invention;

[0034] Figure 2 This is a structural block diagram of the transformer of the present invention;

[0035] Figure 3 A summary of the present invention is shown in FIG.

[0036] Figure 4 This is the transformerBCL model of the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] like Figure 1 、 Figure 3 as well as Figure 4 As shown, this application aims to provide a geological disaster prediction method based on transformer and deep learning, including the following steps:

[0039] Step 1: Collect and process geological information and delete outliers in the geological information dataset; signal processing, specifically determining the embedding dimension and delay time, using the dynamic system reconstruction method to process historical geological disaster data, reconstructing the weight matrix, and converting the one-dimensional time series into trajectories in the high-dimensional phase space, thereby obtaining the periodic changes of geological disasters;

[0040] Step 2: Feature processing and extraction

[0041] Use a one-dimensional convolutional neural network (CNN) to convolve the power sequence and extract power features based on the convolution kernel.

[0042] In this step, the formula of the one-dimensional convolutional neural network is: V = g(W*U+b), where: g represents the activation function. The introduction of the activation function can enhance the nonlinear expression ability of the model and make the model more expressive; * represents convolution; W represents the weight matrix in the convolution kernel; b represents the bias term of the weight matrix in the convolution kernel;

[0043] Step 3: Geological disaster prediction based on LSTM and transformer

[0044] Combine LSTM and transformer to build the transformerBCL model;

[0045] First is the input layer, which inputs data, followed by two one-dimensional convolutional layers, each consisting of 16 convolution kernels of size 4, with a step size of 1 and a linear activation function. After the two convolutional layers extract features from the data, it passes through two bidirectional LSTM layers, which are also the core of the Bi-LSTM model. Finally, there are two fully connected layers. The first fully connected layer integrates all the features of the upper layer, and finally outputs the decomposition results through the fully connected layer with N=1.

[0046] Step 4: Model training. The training steps include:

[0047] a. Parameter initialization: Parameters such as weights and learning rates need to be set before they can be correctly applied to the model;

[0048] b. Establish the loss function and set the stopping condition: The loss function is an important learning criterion that can be used to measure the performance of a network and is often related to the optimization amount. The mean square error loss function is used to calculate the mean square error between them. The calculation formula is as follows:

[0049]

[0050] Where N is the length of the input sequence, is the predicted value of the model, yi is the true value;

[0051] c. By applying the Adam optimizer, the weight parameters and bias parameters in the model can be effectively optimized;

[0052] d. By carefully adjusting the parameters and functions, an effective network training process can be achieved. That is, according to the pre-set number of training times, the model parameters are continuously adjusted until a certain threshold is reached. In this way, even if the expected number of training times is not reached, the model can stop running when the threshold is met. After 3 epochs, if the loss function still does not improve significantly, the deep learning model will be terminated.

[0053] In this application, the basic architecture of LSTM consists of five parts: input x t 、output h t , update gate i t 、Forget Gate t , output gate o t ,in:

[0054] The expression of the forget gate f_t is as follows:

[0055] f t =σ(W f *[h t-1 ,x t ]+b f )

[0056] The expression of update gate i_t is as follows:

[0057] i t =σ(W i *[h t-1 ,x t ]+b i )

[0058] c′ t =tanh(W c *[h t-1 ,x t ]+b c )

[0059] c t =f t *c t-1 +i t *c′ t

[0060] The expression of the output gate o_t is as follows:

[0061] o t =σ(W o *[h t-1 ,xt ]+b o )

[0062] h t =o t *tanh(c t ).

[0063] Compared with ordinary RNN, LSTM can better solve the problems of gradient vanishing and gradient exploding when processing long data sequences, and can also better mine the more valuable information hidden in the previous and next correlations in the data.

[0064] In the present invention, the workflow for extracting useful information from features is as follows: Figure 2 As shown in the figure, the sequence element value is Value, the sequence element address is Key, and the query element is called Query. First, the similarity between Query and the element value in each Key is calculated to obtain the weight coefficient; then the weight coefficient is normalized using the softmax function; finally, the weight and Value are weighted and summed to obtain the Attention value.

[0065] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A geological disaster prediction method based on transformer and deep learning, characterized in that: The steps include: Step 1: Collect and process geological information and delete outliers in the geological information dataset; signal processing, specifically determining the embedding dimension and delay time, using the dynamic system reconstruction method to process historical geological disaster data, reconstructing the weight matrix, and converting the one-dimensional time series into trajectories in the high-dimensional phase space, thereby obtaining the periodic changes of geological disasters; Step 2: Feature processing and extraction Use a one-dimensional convolutional neural network (CNN) to convolve the power sequence and extract power features based on the convolution kernel. In this step, the formula of the one-dimensional convolutional neural network is: V = g(W*U+b), where: g represents the activation function. The introduction of the activation function can enhance the nonlinear expression ability of the model and make the model more expressive; * represents convolution; W represents the weight matrix in the convolution kernel; b represents the bias term of the weight matrix in the convolution kernel; Step 3: Geological disaster prediction based on LSTM and transformer Combine LSTM and transformer to build the transformerBCL model; First, the input layer takes in data, followed by two one-dimensional convolutional layers, each consisting of 16 convolution kernels of size 4, with a stride of 1 and a linear activation function. After the two convolutional layers extract features from the data, the data passes through two bidirectional LSTM layers, which are also the core of the Bi-LSTM model. Finally, there are two fully connected layers. The first fully connected layer integrates all the features of the previous layer, and finally the decomposition results are output through the fully connected layer with N=1. Step 4: Model training. The training steps include: a. Parameter initialization: Parameters such as weights and learning rates need to be set before they can be correctly applied to the model; b. Establish the loss function and set the stopping condition: The loss function is an important learning criterion that can be used to measure the performance of a network and is often related to the optimization amount. The mean square error loss function is used to calculate the mean square error between them. The calculation formula is as follows: Where N is the length of the input sequence, is the predicted value of the model, y i is the true value; c. By applying the Adam optimizer, the weight parameters and bias parameters in the model can be effectively optimized; d. By carefully adjusting the parameters and functions, an effective network training process can be achieved. That is, according to the pre-set number of training times, the model parameters are continuously adjusted until a certain threshold is reached. In this way, even if the expected number of training times is not reached, the model can stop running when the threshold is met. After 3 epochs, if the loss function still does not improve significantly, the deep learning model will be terminated.

2. The geological disaster prediction method based on transformer and deep learning according to claim 1, characterized in that: The basic architecture of the LSTM consists of five parts: input x t 、output h t , update gate i t 、Forget Gate t , output gate o t ,in: Forget Gate f t The expression of is as follows: f t =σ(W f *[h t-1 ,x t ]+b f ) Update Gate i t The expression of is as follows: i t =σ(W i *[h t-1 ,x t ]+b i ) c' t =tanh(W c *[h t-1 ,x t ]+b c ) c t =f t *c t-1 +i t *c′ t Output gate o t The expression of is as follows: the t =σ(W o *[h t-1 ,x t ]+b o ) h t =o t *fishy(c) t )

Images