A Multi-Task Prediction Method for Multi-Source Data in Underground Space Based on BERT Model

Abstract

This invention relates to the field of artificial intelligence in underground space technology, and in particular to a multi-task prediction method for multi-source data in underground space based on the BERT model. It utilizes underground space domain data to build a corpus, pre-trains the BERT model to obtain GEO-BERT, and combines it with a deep learning model. This method can capture both the complex correlation between feature values ​​and target values, and the correlation between target values ​​themselves. This has significant advantages and importance for simultaneous multi-indicator prediction based on multi-source data in the underground space domain. Furthermore, this prediction model can also be used to verify existing indicators and identify outliers. The advantages of this invention are: significantly improved prediction accuracy, enhanced model prediction performance on underground space data, reduced computational costs, capture of correlations between various indicators, improved prediction precision, and identification of potential outliers based on outlier analysis by comparing the model's prediction results with measured values.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence underground space technology, and in particular to a multi-task prediction method for multi-source data in underground space based on the BERT model. Background Technology

[0002] BERT (bi-directional encoder representations from Transformers) is a deep bi-directional representation model that can automatically learn complex feature representations from multiple data sources. Furthermore, pre-trained BERT models can be fine-tuned by adding additional output layers, and the Transformer's own multi-task head network structure can build accurate multi-task models for a wide range of tasks.

[0003] In the field of underground space engineering, there is an urgent need for multi-source data, multi-task, and multi-index prediction. Simultaneously, there is also a pressing need to review existing indicators and screen out outliers. For example, soil mechanical properties such as compression modulus and shear strength are key parameters in geotechnical engineering design. When foundation soil bears the weight of the superstructure, it undergoes consolidation settlement, which is crucial for building safety and requires calculation using compression modulus. During foundation pit excavation, displacement occurs under active earth pressure, necessitating the design of a suitable retaining structure to resist this displacement. This requires calculating the shear strength of the soil layer using cohesion and internal friction angle, and then calculating the soil layer's anti-sliding force for foundation pit retaining structure design. Determining these mechanical properties through geotechnical testing is time-consuming and cannot cover soil samples at every depth, and may contain errors. Traditional empirical formulas often infer soil mechanical properties based on one or two parameters, but such simplified models cannot fully reflect the complexity of soil consolidation and shear mechanisms, and the relationships between different mechanical property indices are also difficult to determine. For example, determining the safety and stability of a foundation pit during excavation often requires comprehensive consideration of multiple monitoring indicators. Under the influence of soil gravity, the retaining structure will deform, while the ground outside the pit will subside, causing deformation of the surrounding environment. Furthermore, the columns inside the pit will bulge. Different deformation amounts are also interrelated. Traditional machine learning often cannot accurately capture the interrelationships between multiple tasks, necessitating a method to predict multiple deformations simultaneously, capture the relationships between different deformation amounts, and provide auxiliary predictions for the safety assessment of the underground retaining system during foundation pit excavation.

[0004] CN115659972A discloses a BERT-TextCNN-based method for classifying food delivery reviews, using the BERT-TextCNN network for multi-feature text classification to handle multi-attribute evaluations of Chinese food delivery reviews. This method targets a completely different domain than the one described in this application, and only performs text classification without regression, making it a single-task, single-index prediction model. Furthermore, this method does not pre-train BERT with data specific to its domain.

[0005] CN115408116A discloses a model-based multi-task target prediction method and device, specifically disclosing that, considering the complexity of the business and the characteristics of the main business directions, and to avoid information loss due to excessive information length, a transformer with a self-attention mechanism is selected, using its encoder (BERT) and decoder (GPT) as two expert networks, and then using a CNN as a third expert network. This method targets a completely different domain than the one described in this application, and it does not pre-train BERT based on data from that specific domain.

[0006] Traditional neural network models typically only perform one type of prediction: regression or classification. Furthermore, when performing regression predictions, they usually predict only one or two target values. This makes it impossible to learn the relationships between multiple target values ​​that need to be predicted simultaneously in the underground space domain. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a multi-task prediction method for multi-source data in underground space based on the BERT model. This method pre-trains the BERT model using underground space data and combines the pre-trained BERT model with a convolutional neural network (CNN) and fully connected layers to simultaneously predict multiple tasks, including classification and regression. While learning the correlation between feature values ​​and target values, it also captures the interrelationships between target values, enabling accurate prediction of multiple target values. Furthermore, this prediction model can also be used to verify existing indicators by identifying outliers through box plot analysis of the ratio of predicted to measured values.

[0008] The objective of this invention is achieved through the following technical solutions:

[0009] A multi-task prediction method for underground space multi-source data based on the BERT model, characterized by the following steps:

[0010] S1. Obtain the raw data of underground space geotechnical engineering projects, preprocess the raw data to make all the preprocessed data the same scale and generate underground space multi-source datasets.

[0011] S2. Convert the original data into a string, perform tokenization and map it into a vector, and form a multi-source data corpus of underground space with a large number of sample data. Use the multi-source data corpus of underground space to pre-train the BERT model to obtain the GEO-BERT model.

[0012] S3. Use CNN to extract features from the vector representation of underground space multi-source data obtained by GEO-BERT model, and select the largest feature from the output of the convolutional layer through max pooling operation for dimensionality reduction and parameter sharing.

[0013] S4. Perform multi-index regression and classification prediction of mechanical properties through a fully connected neural network layer. Randomly select a portion of the data as the training set and the remaining data as the validation set. Use appropriate evaluation metrics to evaluate the regression and classification prediction results.

[0014] S5. Input the measured data of geotechnical engineering into the multi-source data prediction model and output the corresponding multi-index prediction results.

[0015] S6. Use the prediction model to verify the existing data, and perform box plot analysis on the ratio of predicted values ​​to measured values ​​to identify abnormal indicators.

[0016] In S1, data preprocessing is performed for different scenarios, including handling missing values ​​using bilinear interpolation and standardizing the data using StandardScaler; filtering using Kalman filtering and then training a fully connected neural network model to predict in-situ test results using three-dimensional coordinates, and using this model to predict in-situ test results based on the three-dimensional coordinates of the sampling location in the space geotechnical test; and directly using Kalman filtering.

[0017] In S2, the tokenization process refers to using the BERT model's tokenizer to convert a text sequence into its acceptable representation. The tokenizer splits the text into tokens, adds special tokens, and finally converts it into the vector representation required by the BERT model input.

[0018] In S2, the BERT model is pre-trained using MaskedLM for masked language model training, which involves randomly masking parts of the words in the input sentence and using the model to predict the masked words.

[0019] In S2, the BERT model is pre-trained using an early stopping algorithm.

[0020] In S3, a 1DCNN is used. In the classification task, the learning rate is 0.001 and the kernel length is 2. In the regression task, the learning rate is 0.001 and the kernel length is 4.

[0021] In S4, for classification tasks, the confusion matrix, F1 score, precision, and recall are used to evaluate prediction performance; for regression tasks, scatter plots and R... 2 Mean absolute error and root mean square error are used as evaluation criteria for prediction accuracy.

[0022] In S6, the box plot uses 1.5 times the interquartile range to determine the upper and lower limits of normal data, and data outside this range is identified as anomalies.

[0023] The advantages of this invention are:

[0024] 1) This invention performs data processing and spatial prediction based on the characteristics of multi-source data in different scenarios of underground space, and builds a multi-source dataset for underground space.

[0025] 2) It has the characteristics of complex relationships. After preprocessing, the BERT model is used in combination with a custom neural network structure to extract features. It fully learns the complex relationship and feature representation between multiple indicators and target values, rather than being limited to one or two indicators, which greatly improves the prediction accuracy.

[0026] 3) By processing and vectorizing the original soil index data, an underground space data corpus was constructed, and BERT was pre-trained to obtain a customized model GEO-BERT, which further improved the model's prediction performance on underground space data.

[0027] 4) By combining the GEO-BERT model with CNN and fully connected layers, a multi-task, multi-source data prediction model for underground space is built. This model can simultaneously perform classification and regression training and prediction of underground space data. It can not only capture the correlation between feature values, but also share parameters among tasks during training, learning the dependencies between different tasks. This model saves computational costs while capturing the correlation between various indicators, thus improving prediction accuracy.

[0028] 5) The GEO-MULTI multi-source data prediction model for underground space obtained through supervised training was used to verify the original data. The statistical box plot method was used to identify potential outliers by comparing the model's prediction results with the measured values.

[0029] 6) The results show that the overall accuracy for classification prediction is 1.00, achieving a very good prediction effect. Regression prediction R for different target values... 2A relatively ideal prediction accuracy is achieved when the value is between 0.91 and 0.92. Attached Figure Description

[0030] Figure 1 This is a flowchart of the present invention;

[0031] Figure 2 This is a diagram showing the overall structure of the network model of the present invention;

[0032] Figure 3 This is a scatter plot of the predicted and measured values ​​of the compression modulus obtained by this invention, along with the normal range.

[0033] Figure 4 The scatter plot shows the predicted and measured values ​​of triaxial effective stress cohesion obtained by this invention, as well as the normal range.

[0034] Figure 5 This is a scatter plot of the predicted and measured values ​​of the triaxial effective internal friction angle obtained by the present invention, as well as the normal range of the value.

[0035] Figure 6 This is a confusion matrix analysis diagram of the soil layer names predicted by this invention;

[0036] Figure 7 This is the ROC plot of the soil layer names predicted by this invention;

[0037] Figure 8 A bar chart showing the accuracy evaluation score of the classification prediction of soil layer names obtained by this invention;

[0038] Figure 9 This is a box plot showing the ratio of the predicted to the measured compression modulus obtained by this invention.

[0039] Figure 10 This is a box plot showing the ratio of the predicted cohesion value to the measured value obtained by this invention.

[0040] Figure 11 This is a box plot showing the ratio of the predicted internal friction angle to the measured value obtained by this invention. Detailed Implementation

[0041] The features and other related features of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments, so as to facilitate understanding by those skilled in the art:

[0042] Example: Figures 1 to 11 As shown, the multi-task prediction method for multi-source data of underground space based on the BERT model in this embodiment includes the following steps:

[0043] 1) Collect the original geotechnical test data of geotechnical engineering investigation projects, including physical index data such as water content, liquid limit, plastic limit, and void ratio, in-situ static test PS value, mechanical index data such as compression modulus, triaxial effective stress cohesion and triaxial effective stress internal friction angle, and soil name data.

[0044] 2) Kalman filtering is applied to the static borehole data. A prediction model is trained based on the spatial 3D coordinates of the static boreholes and the corresponding filtered static PS values, etc., of the in-situ test results. This model is used to predict the in-situ test parameters, such as the static PS values, corresponding to the spatial 3D coordinates of the geotechnical test sampling locations. The final acquired data is preprocessed. StandardScaler is used to standardize the data, ensuring a mean of 0 and a standard deviation of 1, thus bringing all data to the same scale and generating a multi-source dataset for underground space. The physical and mechanical parameters of each soil sample are converted into strings, tokenized, and mapped to vectors. BERT's tokenizer is used to convert the text sequence into a token representation acceptable to the model. The tokenizer performs tokenization in the vocabulary, splitting the text into tokens and adding special tokens (such as [CLS] and [SEP]), ultimately converting it into the vector representation required for model input.

[0045] In this embodiment, if preprocessing is required for soil layer parameters, bilinear interpolation is used to handle missing values ​​according to different soil layers; if multi-index prediction scenario for foundation pit monitoring is required, Kalman filtering is applied to the measured data such as inclination, column settlement, and surrounding environment settlement.

[0046] 3) A corpus of soil physical and mechanical indices, composed of a large amount of sample data, was used to pre-train the BERT model. The pre-trained BERT model, GEO-BERT, can generate high-dimensional embedded representations of soil physical properties for any sample. The pre-training method specifically employs MaskedLM for masked language model training, which involves randomly masking parts of the words in the input sentence and using the model to predict the masked words. In the training corpus, MaskedLM randomly covers up to 30% of the words in each sentence. To avoid overfitting, early stopping was used during BERT pre-training, with 100 epochs. The model was trained until the accuracy on the validation dataset showed no improvement within a certain number of epochs, at which point a checkpoint was used to backtrack to the best model found during training. The pre-trained BERT model can use soil physical indices as input values ​​to generate their corresponding encoded representations. This approach represents multi-source features as vectors, which can be further used for downstream tasks.

[0047] 4) Use CNN (Convolutional Neural Network) to extract features from the vector representation of soil physical properties obtained from the GEO-BERT model.

[0048]

[0049] Where x in This represents the encoded representation of the soil physical property indicators generated after pre-training, with a size of n×128, where n represents the number of tokens in the input sequence. Represents the convolution operation, w conv Represents the parameter to be learned, o conv This represents the output after passing through a 1D convolutional network. Max pooling is used to select the largest feature from the convolutional layer outputs for dimensionality reduction and parameter sharing. The convolutional model used is a 1D CNN. In the classification task, the learning rate is 0.001, the kernel length is 2, the number of kernels is 128, and the number of neurons in the fully connected layers is 128. In the regression task, the learning rate is 0.001, the kernel length is 4, the number of kernels is 512, and the number of neurons in the fully connected layers is 128, 56, and 76 respectively.

[0050] Finally, a fully connected neural network layer is used for regression prediction of mechanical properties and classification prediction of soil names. 80% of the data is randomly selected as the training set, and the remaining 20% ​​is used as the validation set.

[0051] 5) Input the physical properties obtained from geotechnical tests into the multi-source data prediction model, and output the corresponding mechanical properties and soil moisture prediction results. A scatter plot of the predicted and measured values ​​of the compression modulus is shown below. Figure 3 As shown in the figure. The scatter plot of the predicted and measured values ​​of the triaxial effective stress cohesion is shown in the figure. Figure 4 As shown in the figure. The scatter plot of the predicted and measured values ​​of the triaxial effective stress internal friction angle is shown in the figure. Figure 5 As shown. The confusion matrix of the predicted local names is as follows. Figure 6 As shown in the bar chart, the classification prediction accuracy evaluation scores for each soil layer are as follows. Figure 8 As shown. Appropriate evaluation metrics are used to evaluate the regression and classification prediction results. For classification tasks, the confusion matrix, F1 score, precision, and recall are used to evaluate prediction performance. For regression tasks, scatter plots and R-squared are used. 2 Mean absolute error (MAE) and root mean square error (RMSE) are used as evaluation criteria for prediction accuracy. The smaller the values ​​of MAE and RMSE, the better the prediction performance of the model.

[0052] The calculation results of the evaluation indices for each mechanical property parameter are shown in Table 1 below.

[0053]

[0054]

[0055] 6) Compare the predicted values ​​obtained from the multi-source data and multi-indicator model with the measured values ​​to verify the original indicators. Use statistical methods to perform box plot analysis of the predicted / measured value ratio, and identify outliers based on outliers. The box plot analysis of the predicted / measured value ratios for each indicator is shown below. Figures 9 to 11 As shown in the figure, the interquartile range (IQR) is used in box plots to determine the upper and lower limits of normal data; data outside this range are considered outliers.

[0056] Although the above embodiments have described the concept and embodiments of the present invention in detail with reference to the accompanying drawings, those skilled in the art will recognize that various improvements and modifications can still be made to the present invention without departing from the scope of the claims, and therefore will not be elaborated here.

Claims

1. A multi-task prediction method for multi-source data in underground space based on the BERT model, characterized in that: The method includes the following steps: S1. Obtain the original data of the underground space geotechnical engineering project, preprocess the obtained original data to make all the preprocessed data at the same scale and generate an underground space multi-source dataset, perform Kalman filtering on the static test data, train the prediction model based on the spatial three-dimensional coordinates of the static test hole and the corresponding filtered static test PS value in situ test results, and use the model to predict the static test PS value in situ test index corresponding to the spatial three-dimensional coordinates of the geotechnical test sampling location. S2. Convert the original data into a string, perform tokenization and map it into a vector, and form a multi-source data corpus of underground space with a large number of sample data. Use the multi-source data corpus of underground space to pre-train the BERT model to obtain the GEO-BERT model. S3. Use CNN to extract features from the vector representation of underground space multi-source data obtained by GEO-BERT model, and select the largest feature from the output of the convolutional layer through max pooling operation for dimensionality reduction and parameter sharing. S4. Perform multi-index regression and classification prediction of mechanical properties through a fully connected neural network layer; randomly select a portion of the data as the training set and the remaining data as the validation set, and use appropriate evaluation metrics to evaluate the regression and classification prediction results; S5. Input the measured data of geotechnical engineering into the multi-source data prediction model and output the corresponding multi-index prediction results. S6. Use the multi-source data prediction model to verify existing data, and perform box plot analysis on the ratio of predicted values ​​to measured values ​​to identify abnormal indicators.

2. The method for multi-task prediction of underground space multi-source data based on the BERT model according to claim 1, characterized in that: In S1, data preprocessing is performed for different scenarios, including handling missing values ​​using bilinear interpolation and standardizing the data using StandardScaler. This includes directly using Kalman filtering for filtering.

3. The method for multi-task prediction of underground space multi-source data based on the BERT model according to claim 1, characterized in that: In S2, the tokenization process refers to using the BERT model's tokenizer to convert a text sequence into its acceptable representation. The tokenizer splits the text into tokens, adds special tokens, and finally converts it into the vector representation required by the BERT model input.

4. The method for multi-task prediction of underground space multi-source data based on the BERT model according to claim 1, characterized in that: In S2, the BERT model is pre-trained using MaskedLM for masked language model training, which involves randomly masking parts of the words in the input sentence and using the model to predict the masked words.

5. The method for multi-task prediction of underground space multi-source data based on the BERT model according to claim 1, characterized in that: In S2, the BERT model is pre-trained using an early stopping algorithm.

6. The method for multi-task prediction of underground space multi-source data based on the BERT model according to claim 1, characterized in that: In S3, the CNN used is 1DCNN. In the classification task, the learning rate is 0.001 and the kernel length is 2. In the regression task, the learning rate is 0.001 and the kernel length is 4.

7. The method for multi-task prediction of underground space multi-source data based on the BERT model according to claim 1, characterized in that: In S4, for classification tasks, the confusion matrix, F1 score, precision, and recall are used to evaluate prediction performance; for regression tasks, scatter plots and R... 2 Mean absolute error and root mean square error are used as evaluation criteria for prediction accuracy.

8. The method for multi-task prediction of underground space multi-source data based on the BERT model according to claim 1, characterized in that: In S6, the box plot uses 1.5 times the interquartile range to determine the upper and lower limits of normal data, and data outside this range is identified as anomalies.

Images