Deep learning prediction method for rock freeze-thaw damage based on text embedding

Abstract

The invention belongs to the technical field of rock freeze-thaw damage prediction, and relates to a rock freeze-thaw damage deep learning prediction method based on text embedding, and the method comprises the steps: introducing a text embedding technology into a deep learning model, fusing rock freeze-thaw damage experimental data with unstructured text description information, and importing a neural network for training. And the accuracy, generalization ability and information utilization efficiency of the rock freeze-thaw damage prediction model are obviously improved. The prediction method comprises the steps of collecting data, standardizing numerical data, embedding unstructured text data, generating a data set, constructing a model, adjusting and optimizing hyper-parameters, and storing and deploying an optimal model after a preset evaluation index is met. According to the prediction method, the problem that an existing rock freeze-thaw damage model cannot utilize unstructured text information and cannot accept uneven semantic description is solved, the data utilization efficiency is improved, the prediction precision is improved, and the operation convenience and the technology popularization convenience are further improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of rock freeze-thaw damage prediction, and specifically relates to a rock freeze-thaw damage deep learning prediction method based on text embedding. Background Art

[0002] Studies have found that rocks subjected to repeated freeze-thaw cycles in cold regions will suffer internal damage, resulting in a decrease in strength and durability, seriously affecting the long-term stability of geotechnical engineering projects such as slopes and underground caverns. Typically, indoor physical tests and empirical models are used to study the freeze-thaw damage mechanism of rocks. For example, the strength attenuation of rocks is measured through repeated freeze-thaw tests, and empirical formulas or damage mechanics models are established to predict the residual strength. However, traditional methods have shortcomings such as long experimental cycles, high costs, and limitations on sample acquisition, and the results are easily affected by the discreteness of the samples. Therefore, it is difficult to efficiently and accurately assess the degree of freeze-thaw damage by relying solely on physical experiments and empirical formulas, and new technical means are needed. In recent years, artificial intelligence technologies such as machine learning and deep learning have been introduced into this field to improve prediction efficiency and accuracy.

[0003] At present, a large number of studies have attempted to use machine learning algorithms to predict the mechanical properties and damage indicators of rocks after freezing and thawing. For example, the dynamic parameter data of rocks after freezing and thawing are collected, and the support vector machine (SVM) regression model is used to predict the dynamic compressive strength of rocks after freezing and thawing, and the swarm intelligence algorithm is combined to optimize the model hyperparameters to improve the prediction accuracy. The existing technology has established 216 sets of frozen-thaw rock sample data sets, selected five features such as the number of freeze-thaw cycles, confining pressure, impact load, wave impedance and porosity as input, and used the optimized support vector regression model to achieve high-precision prediction of the dynamic compressive strength (DCS) of rocks after freezing and thawing, with a determination coefficient R 2 As high as 0.98. For example, existing technologies have introduced ensemble learning algorithms such as random forest (RF) to predict the dynamic compressive strength of rocks under freeze-thaw cycles in alpine regions, and integrated the sparrow search algorithm (SSA) to optimize model parameters. The results show that the SSA-optimized RF model is better than other models in generalization performance. Some studies have also used classic machine learning methods such as support vector machines, decision trees, and k-nearest neighbors (KNN) to model and predict the deterioration of rock strength under the influence of freeze-thaw cycles, and have achieved certain results. Overall, data-driven models based on machine learning can learn the mapping relationship between freeze-thaw damage and rock performance attenuation from experimental data without complex theoretical assumptions, thereby effectively predicting the degree of freeze-thaw damage to rocks.

[0004] With the growth of data and computing power, more complex deep learning models have been introduced in the field of rock freeze-thaw damage prediction. For example, existing technologies have used various deep learning algorithms to analyze the changes in mechanical parameters of natural stone after freeze-thaw and thermal shock cycles. Comparing the performance of models such as artificial neural networks (ANNs) and deep neural networks (DNNs), they found that increasing model complexity does not necessarily lead to higher accuracy. Instead, the AdaBoost ensemble algorithm achieved the best prediction results. This suggests that deep learning models must be carefully designed to prevent overfitting on small and medium-sized rock datasets. Researchers have begun exploring the use of new deep learning architectures such as convolutional neural networks (CNNs) and Transformers for rock material property prediction. For example, one study used element capture spectrum (ECS) logging data to predict rock strength using random forest and Transformer models, demonstrating that Transformers can effectively mine the complex relationship between high-dimensional geological data and lithologic strength. In the freeze-thaw field, existing technologies have developed an interpretable XGBoost ensemble model for the frost resistance of aeolian sand fiber-reinforced concrete. Combined with feature importance analysis, they successfully revealed the influence of material composition on freeze-thaw damage indicators. The above research shows that deep learning shows great potential in the prediction of rock freeze-thaw damage: on the one hand, multi-layer networks and attention mechanisms are expected to capture the nonlinear effects of freeze-thaw damage; on the other hand, the model can integrate feature interpretation tools to provide quantitative evaluation of key influencing factors.

[0005] Compared to traditional physical models that rely on assumptions about freeze-thaw damage mechanisms and empirical formulas, machine learning and deep learning methods establish an "input-output" mapping directly from data, avoiding certain mechanistic assumptions that are difficult to accurately describe. For example, existing technologies have established empirical prediction models for mudstone strength degradation under freeze-thaw and chemical coupling, but these models still require extensive experimental calibration of parameters. Data-driven AI models, on the other hand, can rapidly predict rock strength under various conditions using existing experimental data. Traditional methods offer a certain degree of authority and physical interpretability under small sample sizes. For example, models based on damage mechanics can reflect the mechanism by which pore water ice expansion causes rock degradation. Therefore, in engineering applications, physical and data-driven models are often used to corroborate each other. However, the applicability of traditional empirical models is often limited to specific lithologies and operating conditions, and predictive accuracy is difficult to guarantee when conditions vary significantly. In contrast, AI models, provided they are trained with sufficient and diverse data, have the potential to generalize across a wider range of conditions. Furthermore, AI models can integrate information from multiple sources, a feat difficult for traditional models to achieve.

[0006] Both traditional machine learning and deep learning models mostly utilize only structured numerical data as input, ignoring the textual descriptions of rock samples. This textual information, including information about the rock sample's origin, geological age, mineralogical composition, color, and structural characteristics, is crucial for rock material properties. However, existing studies often only collect numerical metrics such as porosity, wave velocity, and strength. Even when some studies distinguish rock types, they typically convert these into discrete category codes, which fails to fully capture the rich semantic information contained in the textual descriptions. Missing rock textual information cannot be replaced or automatically inferred by the model: even the most complex deep neural networks cannot account for the impact of key features if they are not included in the input. This issue has been identified as a bottleneck limiting the generalization and accuracy of predictive models. For example, when comparing different models for predicting freeze-thaw degradation, researchers have suggested that improved model performance is limited by the quality of the input features, not simply the depth of the model. When rock origin or composition varies significantly and is not reflected in the numerical features, the model often exhibits prediction bias. This indicates that the lack of input of text descriptions of rock samples will result in the model being unable to capture implicit factors related to freeze-thaw sensitivity, thereby affecting the applicability and reliability of the prediction results under different lithologies and regional conditions.

[0007] In the prior art, researchers in geotechnical and materials engineering have also applied artificial intelligence to performance prediction. Chinese patent CN117972852A proposes a dynamic rockburst prediction method based on multi-source heterogeneous data and machine learning. This method improves the speed and accuracy of rockburst intensity prediction by integrating multiple data sources, such as numerical simulation data, microseismic monitoring data, and acoustic emission signals. This approach demonstrates that combining different types of structured data (such as sensor time series and simulation results) can improve model performance. However, in these prior art solutions, the input data is still primarily numerical and sensor signals, and does not include textual descriptions of the rock mass or rock samples. Saudi Aramco has proposed a method for automatically annotating rock core photographs using computer vision to assist in machine learning prediction of rock properties. This method focuses on the use of unstructured data, such as images, but similarly does not address the integration of textual descriptions of rock samples.

[0008] In summary, while existing technologies have improved the intelligent prediction capabilities of rock engineering parameters to a certain extent, they lack the ability to leverage rock text semantic information. This deficiency prevents models from fully reflecting rock genesis and material differences, becoming a key obstacle to further improving freeze-thaw damage prediction. Therefore, there is an urgent need to develop a new freeze-thaw damage prediction method that integrates rock text descriptions with numerical data to overcome the limitations of existing technologies. Summary of the Invention

[0009] The purpose of this invention is to provide a deep learning rock freeze-thaw damage prediction method that integrates text embedding to solve the problem that traditional rock freeze-thaw damage prediction models cannot effectively utilize unstructured text information, resulting in low model prediction accuracy and poor generalization performance.

[0010] The purpose of the present invention is achieved through the following technical solutions: A deep learning prediction method for rock freeze-thaw damage based on text embedding includes the following steps: Step A: Collect rock freeze-thaw damage experimental data, including numerical data and unstructured text data consisting of physical parameters, environmental parameters, and damage indicators; Step B: Standardize the numerical data to obtain a processed feature data matrix; embed the unstructured text data, clean and preprocess the rock sample text description fields, and use BERT language processing technology to convert the text description of each rock sample into a fixed-dimensional numerical vector, directly generating a sentence-level embedding vector; Step C: Generate a dataset. The vector generated by text embedding is concatenated with the numerical feature vector normalized in step B to form a unified input feature vector. The data is randomly seeded and divided into a 70% training set, a 10% validation set, and a 20% test set for model training, parameter adjustment, and final effect evaluation. The divided dataset is encapsulated with feature data and target labels. Step D: Deep learning model construction. A Transformer-based end-to-end model is constructed to predict rock damage under freeze-thaw cycles. The ReLU activation function is used as the nonlinear activation function and the Adam optimizer is used for optimization. Step E: Hyperparameter adjustment and optimization: Design the key hyperparameters of the model, use grid search to traverse the key parameters and select the optimal model; In step F, the trained model is input with new initial parameters and environmental parameters to generate freeze-thaw damage variables that need to be predicted.

[0011] Furthermore, in step A, physical parameters include porosity, dry density, saturated density, elastic modulus, compressive strength, and longitudinal wave velocity; environmental parameters include freezing temperature, melting temperature, freeze-thaw cycles, and loading rate; damage indicators include uniaxial compressive strength after freeze-thaw, peak strain after freeze-thaw, and elastic modulus after freeze-thaw; and unstructured text data is a description of the rock origin, rock color, mineral composition, and degree of weathering.

[0012] Furthermore, in step B, the normalized formula is:

[0013] in, Xs is the normalized eigenvalue, X is the original eigenvalue, μ is the mean of the feature, σ is the standard deviation of the feature.

[0014] Furthermore, in step B, the rock sample text description fields are cleaned and preprocessed, including removing punctuation marks, extra spaces, and special characters to make the text neat and standardized for easy model processing.

[0015] Furthermore, in step D, the model size is selected based on the sample size. The number of samples should be greater than 300. When the number of samples is less than 1000, a lightweight Transformer model (encoder layer < 2) is used.

[0016] Furthermore, the Transformer model consists of an input embedding layer, a positional encoding layer, a Transformer encoder module, a feedforward neural network layer, and an output layer; The input embedding layer maps the input features to a high-dimensional vector space, making it easier for the model to learn the relationship between features. The formula is:

[0017] in, X embed is the concatenated feature matrix, X is the original input feature matrix, W embed is the embedding weight matrix, b embed is bias; Transformer has no sequence perception capability, so position encoding is added to retain the position information between features. The formula is:

[0018] in, PE ( pos, 2 i )and PE ( pos, 2 i+ 1) is the position encoding vector, pos Indicates the feature location, i represents the position encoding dimension index, d model represents the embedding vector dimension; The Transformer encoder consists of a multi-head self-attention layer and a positional feed-forward network, both with residual connections and layer normalization; Multi-head attention mechanism formula:

[0019] in:

[0020] in:

[0021] in, MultiHead is the attention mechanism, Q, K, V are the input Query, Key and Value matrices respectively, head i For the i The output of an attention head, W i Q ，W i K ，W i V ，W O are training parameters, Attention is the attention mechanism function, QK T is the product of Q and the transpose of K, d k is the dimension of the Key; Position feedforward network FFN formula:

[0022] in, W 1, W 2 and b 1, b 2 is a trainable parameter; The features encoded by Transformer are output through the fully connected layer to output the final predicted damage index; The difference between the model output value and the true observation value is measured using the mean squared error (MSE), which is:

[0023] in, L data is the total loss, N is the number of samples, For the i The true value of the sample, For the i The predicted value of the sample.

[0024] Furthermore, the ReLU activation function is used as the nonlinear activation function, and its formula is:

[0025] in, ReLU is the activation function, when the input x When the input is less than 0, the output is 0. x When it is greater than or equal to 0, the output is x .

[0026] Furthermore, we use the Adam optimizer to optimize and dynamically adjust the learning rate using first-order momentum and second-order momentum. The specific steps are as follows: First, compute the exponentially weighted moving average of the gradients:

[0027] in, m t is the first moment estimate of the gradient, g t is the current gradient, β 1 is the momentum parameter; Then, we compute the exponentially weighted moving average of the squared gradients:

[0028] in, v t is the second moment estimate of the gradient, β 2 is the momentum parameter; The first-order and second-order moment estimates are then bias-corrected:

[0029] Finally, update the parameters:

[0030] in, θ t is the current parameter, α is the learning rate, ε is a small constant that prevents division by zero.

[0031] Furthermore, in step E, the key hyperparameters of model design include learning rate, batch size, L2 regularization weight, Dropout ratio, number of Transformer layers, and number of Transformer heads; the adjustment order is: learning rate > number of Transformer layers > number of Transformer heads > Dropout ratio > L2 regularization weight > batch size.

[0032] Furthermore, each set of hyperparameter combinations uses 5-fold cross validation (KFold, k=5), and MAE, MSE, RMSE, MAPE, R 2 , export all prediction results, and select the model with the best comprehensive evaluation as the optimal model by comparing the performance of each model; Mean absolute error (MAE): represents the average of the absolute values of the differences between the predicted value and the true value. The calculation formula is:

[0033] Mean square error (MSE): represents the average of the squares of the differences between the predicted value and the true value. The calculation formula is:

[0034] Root mean square error (RMSE): is the square root of MSE, providing an error assessment in the same unit as the original data. The calculation formula is:

[0035] Mean Absolute Percentage Error (MAPE): represents the average of the ratio of the absolute value of the difference between the predicted value and the true value to the true value, expressed as a percentage. The data points where the true value is zero are ignored in the MAPE calculation. The calculation formula is:

[0036] Coefficient of determination (R-squared, R 2 ): Measures the amount of variation explained by the model, calculated as:

[0037] in, is the mean of the true values.

[0038] Compared with the prior art, the present invention has the following beneficial effects: 1. Improve generalization ability: Traditional theoretical models or existing machine learning models can often only show good prediction effects for sample data of specific rock types or under specific conditions, but have poor prediction performance for data of other types of rocks or different experimental conditions; the fundamental reason is that the proposal of such methods relies on limited experimental samples and can only effectively process numerical experimental data, and fails to fully consider the large amount of unstructured information contained in the rock description, such as the rock's origin, color, mineral composition, degree of weathering and other key information. Therefore, when there is a difference between the actual application scenario and the scenario when the model was developed, the generalization performance of the model will be significantly reduced. The present invention incorporates the unstructured text information of the rock sample into the model in an embedded manner, so that the model can recognize and utilize potential influencing factors such as the rock's origin, color, mineral composition and degree of weathering, so that when faced with new rock samples that have not been trained or encountered, it can still maintain a high prediction accuracy and good generalization performance; 2. Improve data utilization efficiency and enhance prediction accuracy: Existing rock damage prediction models are mostly limited to numerical data and ignore the textual information contained in rock descriptions. This invention uses BERT natural language processing technology to convert the textual descriptions of rock samples into high-dimensional numerical vectors that can be used for model learning. This method effectively mines and utilizes the implicit feature information in the rock sample text, significantly improving data utilization efficiency and further enhancing the model's prediction performance. 3. Savings on experimental costs and resources: Traditional models typically rely on large amounts of experimental data to improve prediction accuracy, increasing data acquisition costs and the frequency of testing. This invention, by utilizing unstructured textual information about rocks, reduces reliance on large amounts of experimental data, reducing the number of actual tests and data acquisition costs. Furthermore, the improved model prediction accuracy further reduces the need for actual experimental verification, a benefit that is more aligned with the principles of green, low-carbon, and sustainable development. 4. Improved ease of operation and technology promotion: Traditional methods require a lot of manual feature extraction, data preprocessing, and manual adjustment of model parameters, which requires a high level of professional expertise from the user and is difficult to promote in actual projects. The automated text processing and prediction framework proposed in this invention can quickly obtain prediction results without manual intervention for complex feature engineering and parameter adjustment, lowering the threshold for technology use and facilitating its promotion and application in actual engineering projects. 5. Solved the problem of uneven description of semantic information, making the fusion of multi-source data more convenient: Although existing methods have also attempted to quantify information such as the origin, color and mineral composition of rocks as input features, the prominent problem they face is inconsistent semantic granularity. Specifically, there is inconsistency in the regional scale of rock origin descriptions, some descriptions are specific to provinces or counties, and some are more general; there are also similar but slightly different expressions in color descriptions, such as "dark green" and "dark green", "gray" and "grayish white", and in addition, there are often some missing samples in mineral composition data. These problems are particularly evident in traditional numerical coding or feature engineering processing, which often requires complex manual intervention and processing, resulting in coding confusion and degraded model performance. The present invention uses a text information embedding method to uniformly embed these semantic information of different granularities and unevenness into a high-dimensional space, solving the above-mentioned problem of inconsistent granularity, simplifying the data processing process and improving the robustness and applicability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 Flowchart of the steps of the deep learning prediction method for rock freeze-thaw damage based on text embedding of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described below in conjunction with embodiment: The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0042] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are used only to distinguish the description and should not be understood as indicating or implying relative importance.

[0043] Traditional rock freeze-thaw damage prediction models are unable to effectively utilize unstructured textual information such as rock origin, mineral composition, color, and degree of weathering, resulting in low model prediction accuracy and poor generalization performance. Traditional methods can usually only process numerical data and cannot directly use the rich textual information of rock samples, resulting in a large amount of information waste. Based on this, the present invention proposes a deep learning prediction method for rock freeze-thaw damage based on text embedding. By introducing text embedding technology into the deep learning model, it effectively utilizes the unstructured textual description information of rock samples, significantly improving the accuracy, generalization ability, and information utilization efficiency of the rock freeze-thaw damage prediction model.

[0044] The present invention provides a deep learning prediction method for rock freeze-thaw damage based on text embedding, comprising the following steps: Step A, data preparation: Collect rock freeze-thaw damage experimental data. The data sources are published academic papers, research reports, or various experimental data that have been obtained, including numerical data (physical parameters, environmental parameters and damage indicators) and unstructured text data. Among them, physical parameters include porosity, dry density, saturated density, elastic modulus, compressive strength, longitudinal wave velocity, etc. Environmental parameters include freezing temperature, melting temperature, freeze-thaw times, loading rate, etc. Damage indicators include uniaxial compressive strength after freeze-thaw, peak strain after freeze-thaw, elastic modulus after freeze-thaw, etc. Unstructured text data is a description of the rock origin, rock color, mineral composition, weathering degree, etc., using a Chinese description, and different levels of fuzziness can be accepted. The following are a few examples: (1) Mudstone; (2) Basalt in a certain area in western China: albite: 46.9%, microcline: 39.62%, quartz: 7.57%, mica: 5.37%, calcite: 0.54%; (3) Moderately weathered red sandstone in Xi'an, Shaanxi Province, quartz: 36.8%, feldspar: 44.6%, calcite: 6.2%, dolomite: 1.7%, hematite: 0.9%, clay minerals: 9.8%; (4) Sandstone, quartz, kaolinite, illite, and siderite from Huaibei City, Anhui Province.

[0045] Because this method uses a Transformer-based text embedding method to extract semantic features from rock descriptive information, the training data size requirement is somewhat larger than that of traditional numerical feature models. Specifically, the training data sample size should be greater than 300, with an optimal sample size of more than 1,000 (one data point includes all numerical data and unstructured text data for a rock sample). Using Pandas, all raw data is consolidated into a unified CSV data table, with each row representing a rock sample and each column representing a feature or indicator.

[0046] Step B, data processing. First, perform normalization on the numerical data. In order to eliminate the dimensional differences between features, the numerical data is normalized (Z-score normalization). The normalization formula is:

[0047] in, X s is the normalized eigenvalue, X is the original eigenvalue, μ is the mean of the feature, σ is the standard deviation of the feature.

[0048] The StandardScaler class in the scikit-learn library is used to process the data, and the fit_transform method is called to perform standardization operations to obtain the processed feature data matrix.

[0049] Unstructured text data embedding and processing. The rock sample text description fields were cleaned and preprocessed, including removing punctuation, extra spaces, and special characters to make the text neat and standardized for model processing. This was achieved using the re toolkit to remove non-essential punctuation such as commas, colons, periods, and parentheses, while retaining percent signs, decimal points, numbers, letters, and Chinese characters. Comma positions were replaced with single spaces.

[0050] The pre-trained Chinese Sentence-BERT model, distiluse-base-multilingual-cased-v2, from the sentence-transformers library was used to transform the text description of each rock sample into a fixed-dimensional numerical vector. Sentence-BERT directly generates sentence-level embeddings. Distiluse-base-multilingual-cased-v2 is trained based on the DistilBERT framework (a distilled version of BERT). After this step, the description text of each rock sample becomes a high-dimensional numerical vector (512 dimensions) that represents the semantic information of the rock text.

[0051] Step C: Generate a dataset. The vector generated by text embedding is concatenated and fused with the standardized numerical feature vector to form a unified input feature vector, which is then fed into the subsequent deep learning model for training. The train_test_split method in scikit-learn is then used to partition the data using a random seed (e.g., random_state=10). All data is randomly divided into three parts: a training set (70%), a validation set (10%), and a test set (20%) for model training, parameter adjustment, and final effect evaluation. Finally, the partitioned dataset is encapsulated into a dataset format supported by PyTorch. A custom dataset loading class is created by inheriting the torch.utils.data.Dataset class, encapsulating the feature data and target labels separately for batch reading during subsequent training.

[0052] Step D: Build a deep learning model. The Transformer is a deep neural network model based on the self-attention mechanism. It efficiently captures global dependencies in input data and demonstrates strong generalization capabilities in sequence modeling and regression prediction tasks. By building an end-to-end Transformer-based model, we can predict rock damage under freeze-thaw cycles. The model size is selected based on the sample size. For samples with less than 1,000, a lightweight Transformer model (with fewer than 2 encoder layers) is used.

[0053] The Transformer model consists of an input embedding layer, a positional encoding layer, a Transformer encoder module, a feed-forward neural network layer, and an output layer.

[0054] The input embedding layer maps the input features to a high-dimensional vector space, making it easier for the model to learn the relationship between features. The formula is:

[0055] in, X embed is the concatenated feature matrix, X is the original input feature matrix, W embed is the embedding weight matrix, b embed For bias.

[0056] Transformer has no sequence perception ability, so position encoding is added to preserve the position information between features. The formula is:

[0057] in, PE ( pos, 2 i )and PE ( pos, 2 i+ 1) is the position encoding vector, pos Indicates the feature location, i represents the position encoding dimension index, d model represents the embedding vector dimension.

[0058] The Transformer encoder contains a Multi-Head Self-Attention layer and a Position-wise Feed-Forward network, both with Residual Connection and Layer Normalization.

[0059] Multi-head attention mechanism formula:

[0060] in:

[0061] in:

[0062] in, MultiHead is the attention mechanism, Q, K, V are the input Query, Key and Value matrices respectively, head i For the i The output of an attention head, W i Q ，W i K ，W i V ，WO are training parameters, Attention is the attention mechanism function, QK T is the product of Q and the transpose of K, d k is the dimension of the Key; Position feedforward network FFN formula:

[0063] in, W 1, W 2 and b 1, b 2 is a trainable parameter.

[0064] The features encoded by Transformer are output through the fully connected layer to obtain the final predicted damage indicators.

[0065] The difference between the model output value and the true observation value is measured using the mean squared error (MSE), which is:

[0066] in, L data is the total loss, N is the number of samples, For the i The true value of the sample, For the i The predicted value of the sample.

[0067] The ReLU activation function is used as the nonlinear activation function, and its formula is:

[0068] in, ReLU is the activation function, when the input x When the input is less than 0, the output is 0. x When it is greater than or equal to 0, the output is x .

[0069] The Adam optimizer (Adaptive Moment Estimation) is used for optimization. The Adam optimizer accelerates convergence by adjusting the learning rate of each parameter at each iteration. Its core idea is to dynamically adjust the learning rate using first-order momentum (the average of the gradient) and second-order momentum (the average of the squared gradient). The specific steps are as follows: First, compute the exponentially weighted moving average of the gradients:

[0070] in, m t is the first moment estimate of the gradient,g t is the current gradient, β 1 is the momentum parameter.

[0071] Then, we compute the exponentially weighted moving average of the squared gradients:

[0072] in, v t is the second moment estimate of the gradient, β 2 is the momentum parameter.

[0073] The first-order and second-order moment estimates are then bias-corrected:

[0074] Finally, update the parameters:

[0075] in, θ t is the current parameter, α is the learning rate, ε is a small constant that prevents division by zero.

[0076] Step E, hyperparameter adjustment and optimization. The key hyperparameters of model design include learning rate, batch size, L2 regularization weight, Dropout ratio, number of Transformer layers, and number of Transformer heads. The adjustment order is: learning rate > number of Transformer layers > number of Transformer heads > Dropout ratio > L2 regularization weight > batch size. Use Grid Search to traverse the key parameters. Each set of hyperparameter combinations uses 5-fold cross validation (KFold, k=5) and is implemented using the KFold method in sklearn. MAE, MSE, RMSE, MAPE, R are recorded for each round of training. 2 Finally, use pandas to export all prediction results to CSV. By comparing the performance of each model, the model with the best overall evaluation is selected as the optimal model.

[0077] Mean Absolute Error (MAE): represents the average of the absolute values of the differences between the predicted values and the true values. The calculation formula is:

[0078] Mean Squared Error (MSE): represents the average of the squares of the differences between the predicted value and the true value. The calculation formula is:

[0079] Root mean square error (RMSE): is the square root of MSE, providing an error estimate in the same units as the original data. The calculation formula is:

[0080] Mean Absolute Percentage Error (MAPE): The average of the ratios of the absolute value of the difference between the predicted value and the true value, expressed as a percentage. In particular, to avoid infinite or uncertain values caused by the true value being zero, data points with a true value of zero are ignored in the MAPE calculation. The calculation formula is:

[0081] Coefficient of determination (R-squared, R 2 ): Measures the amount of variation explained by the model, calculated as:

[0082] in, is the mean of the true values.

[0083] Step F, model application: New initial parameters and environmental parameters are input into the trained model to generate the freeze-thaw damage variables to be predicted.

[0084] like Figure 1 As shown in the figure, the present invention uses a deep learning prediction method for rock freeze-thaw damage based on text embedding. This method introduces a text embedding mechanism to convert unstructured rock text information into numerical vector features for input into the deep learning model. The text embedding and numerical data fusion mechanism utilizes the text features of rock samples to improve model performance.

[0085] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A deep learning prediction method for rock freeze-thaw damage based on text embedding, characterized by: The following steps are involved: Step A: Collect rock freeze-thaw damage experimental data, including numerical data and unstructured text data consisting of physical parameters, environmental parameters, and damage indicators; Step B: Standardize the numerical data to obtain a processed feature data matrix; embed the unstructured text data, clean and preprocess the rock sample text description fields, and use BERT language processing technology to convert the text description of each rock sample into a fixed-dimensional numerical vector, directly generating a sentence-level embedding vector; Step C: Generate a dataset. The vector generated by text embedding is concatenated with the numerical feature vector normalized in step B to form a unified input feature vector. The data is randomly seeded and divided into a 70% training set, a 10% validation set, and a 20% test set for model training, parameter adjustment, and final effect evaluation. The divided dataset is encapsulated with feature data and target labels. Step D: Deep learning model construction. A Transformer-based end-to-end model is constructed to predict rock damage under freeze-thaw cycles. The ReLU activation function is used as the nonlinear activation function and the Adam optimizer is used for optimization. Step E: Hyperparameter adjustment and optimization: Design the key hyperparameters of the model, use grid search to traverse the key parameters and select the optimal model; In step F, the trained model is input with new initial parameters and environmental parameters to generate freeze-thaw damage variables that need to be predicted.

2. The method for predicting rock freeze-thaw damage based on deep learning and text embedding according to claim 1 is characterized by: Step A: Physical parameters include porosity, dry density, saturated density, elastic modulus, compressive strength, and longitudinal wave velocity; environmental parameters include freezing temperature, melting temperature, freeze-thaw cycles, and loading rate; damage indicators include uniaxial compressive strength after freeze-thaw, peak strain after freeze-thaw, and elastic modulus after freeze-thaw; unstructured text data is a description of the rock origin, rock color, mineral composition, and degree of weathering.

3. The rock freeze-thaw damage deep learning prediction method based on text embedding according to claim 1 is characterized in that: Step B, the normalized formula is: , in, X s is the normalized eigenvalue, X is the original eigenvalue, μ is the mean of the feature, σ is the standard deviation of the feature.

4. The method for predicting rock freeze-thaw damage based on deep learning and text embedding according to claim 1 is characterized by: In step B, the rock sample text description fields are cleaned and preprocessed, including removing punctuation marks, extra spaces, and special characters to make the text neat and standardized for easy model processing.

5. The method for predicting rock freeze-thaw damage based on deep learning using text embedding according to claim 1, characterized in that: In step D, select the model size based on the sample size. The sample size should be greater than 300. When the sample size is less than 1000, use a lightweight Transformer model with <2 encoder layers.

6. The method for predicting rock freeze-thaw damage based on deep learning using text embedding according to claim 5, characterized in that: The Transformer model consists of an input embedding layer, a positional encoding layer, a Transformer encoder module, a feedforward neural network layer, and an output layer; The input embedding layer maps the input features to a high-dimensional vector space, making it easier for the model to learn the relationship between features. The formula is: , in, X embed is the concatenated feature matrix, X is the original input feature matrix, W embed is the embedding weight matrix, b embed is bias; Transformer has no sequence perception capability, so position encoding is added to retain the position information between features. The formula is: , in, PE ( pos, 2 i )and PE ( pos, 2 i+ 1) is the position encoding vector, POS Indicates the feature location, i represents the position encoding dimension index, d model represents the embedding vector dimension; The Transformer encoder consists of a multi-head self-attention layer and a positional feed-forward network, both with residual connections and layer normalization; Multi-head attention mechanism formula: , in: , in: , in, MultiHead is the attention mechanism, Q, K, V are the input Query, Key and Value matrices respectively, head i For the i The output of an attention head, W i Q ，W i K ，W i V ，W O are training parameters, Attention is the attention mechanism function, QK T is the product of Q and the transpose of K, d k is the dimension of the Key; Position feedforward network FFN formula: , in, W 1, W 2 and b 1, b 2 is a trainable parameter; The features encoded by Transformer are output through the fully connected layer to output the final predicted damage index; The difference between the model output value and the true observation value is measured by the mean square error MSE, which is in the form of: , in, L data is the total loss, N is the number of samples, For the i The true value of the sample, For the i The predicted value of the sample.

7. The method for predicting rock freeze-thaw damage based on deep learning using text embedding according to claim 6, characterized in that: The ReLU activation function is used as the nonlinear activation function, and its formula is: , in, ReLU is the activation function, when the input x When the input is less than 0, the output is 0. x When it is greater than or equal to 0, the output is x .

8. The method for predicting rock freeze-thaw damage based on deep learning and text embedding according to claim 6 is characterized by: Use the Adam optimizer for optimization and use the first-order momentum and second-order momentum to dynamically adjust the learning rate. The specific steps are as follows: First, compute the exponentially weighted moving average of the gradients: , in, m t is the first moment estimate of the gradient, g t is the current gradient, β 1 is the momentum parameter; Then, we compute the exponentially weighted moving average of the squared gradients: , in, v t is the second moment estimate of the gradient, β 2 is the momentum parameter; The first-order and second-order moment estimates are then bias-corrected: , Finally, update the parameters: , in, θ t is the current parameter, α is the learning rate, ε is a small constant that prevents division by zero.

9. The method for predicting rock freeze-thaw damage based on deep learning and text embedding according to claim 1, characterized in that: In step E, the key hyperparameters for model design include learning rate, batch size, L2 regularization weight, Dropout ratio, number of Transformer layers, and number of Transformer heads. The adjustment order is: learning rate > number of Transformer layers > number of Transformer heads > Dropout ratio > L2 regularization weight > batch size.

10. The method for predicting rock freeze-thaw damage based on deep learning using text embedding according to claim 9, characterized in that: Each set of hyperparameter combinations uses 5-fold cross validation KFold, k=5, and MAE, MSE, RMSE, MAPE, R 2 , export all prediction results, and select the model with the best comprehensive evaluation as the optimal model by comparing the performance of each model; Mean absolute error (MAE): represents the average of the absolute values of the differences between the predicted value and the true value. The calculation formula is: , Mean square error (MSE): represents the average of the squares of the differences between the predicted value and the true value. The calculation formula is: , Root mean square error (RMSE): is the square root of MSE, providing an error assessment in the same unit as the original data. The calculation formula is: , Mean Absolute Percentage Error (MAPE): represents the average of the ratio of the absolute value of the difference between the predicted value and the true value to the true value, expressed as a percentage. The data points where the true value is zero are ignored in the MAPE calculation. The calculation formula is: , Coefficient of determination R 2 : Measures the amount of variation explained by the model, calculated as: , in, is the mean of the true values.

Images