Cement-based grouting material proportioning optimization method and system based on deep reinforcement learning

Abstract

The cement-based grouting material proportioning optimization method and system based on deep reinforcement learning are provided, and relate to the technical field of cement-based material optimization, and include: obtaining original grouting material proportioning data; constructing a deep reinforcement learning network, inputting the original grouting material proportioning data as initial input features into the deep reinforcement learning network, and obtaining initial predicted output grouting parameters; introducing a Q-Learning reinforcement update module in the deep reinforcement learning network, giving an optimal grouting parameter target output, designing a dynamic environment adaptive update strategy, and realizing reverse optimization of the input features through iterative adjustment of a feedback mechanism; in the fine-tuning action of the input features, selecting an input feature update mode and an update direction, inputting the updated features into a prediction module, gradually approaching the target output in the iteration of the prediction module, optimizing the initial input features, and obtaining optimal grouting material proportioning input.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of cement-based material optimization, in particular to a cement-based grouting material proportioning optimization method and system based on deep reinforcement learning. BACKGROUND

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute the prior art.

[0003] As an effective underground engineering measure, grouting is widely used in the disposal of gushing water in tunnel construction. The main function of grouting is to block the flow of groundwater and improve the stability of the tunnel face and its surrounding strata. The proportioning of grouting material is a key and direct factor in determining its material properties and greatly affects the grouting effect. Therefore, in the treatment of gushing water, it is particularly important to reasonably design the proportioning of grouting materials. Traditional research and design of grouting materials highly depend on frequent formulation adjustment and parameter optimization under laboratory conditions. This method not only consumes time and effort, resulting in a complex and inefficient overall design process, but also may pose safety risks when handling toxic materials.

[0004] With the rapid development of information technology, machine learning technology has gradually become a new tool for the research and development of grouting materials. Support vector machines, random forests, neural networks and other algorithms have been used for performance prediction, structural analysis and proportioning optimization design of grouting materials. By using these algorithms, the internal relationship between material performance and formulation can be mined from existing data to achieve intelligent prediction of material performance and optimization of proportioning. However, although existing machine learning methods have promoted the intelligentization process of grouting material research to some extent, there are still obvious deficiencies in the explainability of the algorithms. Especially when exploring how different material proportions specifically affect grouting performance, the lack of intuitive explanation mechanisms and visualization tools becomes a bottleneck limiting further development. For the optimization design of material proportioning, performance-oriented reverse design methods face higher technical challenges and need to rely on more complex customized algorithms, such as autoencoders, generative adversarial networks, etc., to generate reasonable crystal structures of specific material systems from a microscopic perspective.

[0005] Currently, the methods for performance prediction and analysis of grouting materials mainly include single-factor experiments, response surface analysis, polynomial regression and machine learning algorithms. On this basis, combined with multi-objective optimization strategies and multi-index comprehensive evaluation technologies such as entropy weight method, the optimization or optimization design of grouting material proportioning can be achieved. However, performance prediction and proportioning optimization design usually require the construction of different algorithm models, resulting in a lengthy and complex overall process. SUMMARY

[0006] The cement-based grouting material proportioning optimization method and system based on deep reinforcement learning are proposed to solve the above problems, the neural network back propagation mechanism is combined with reinforcement learning to establish a unified material performance prediction and proportioning optimization design framework, the neural network learns the nonlinear mapping relationship between material proportioning and performance to realize accurate prediction of grouting material performance, the epsilon-greedy strategy of reinforcement learning is introduced to iteratively adjust the initial material proportioning, and by setting a reasonable reward function and action selection strategy, the optimization process can be dynamically updated according to the prediction error, and gradually approaches the designed grouting performance.

[0007] According to some embodiments, the present disclosure adopts the technical solutions as follows:

[0008] The cement-based grouting material proportioning optimization method based on deep reinforcement learning comprises:

[0009] Obtaining original grouting material proportioning data;

[0010] Constructing a deep reinforcement learning network, inputting the original grouting material proportioning data as initial input features into the deep reinforcement learning network, and outputting the initial predicted grouting parameters by a prediction module;

[0011] Introducing a Q-Learning reinforcement update module in the deep reinforcement learning network, giving an optimal grouting parameter target output, designing a dynamic environment adaptive update strategy in the Q-Learning reinforcement update module, and iteratively adjusting the input features through a feedback mechanism to realize reverse optimization of the input features, selecting an input feature update mode and an update direction in the fine-tuning action of the input features, inputting the updated features into the prediction module, and dynamically iteratively adjusting according to the difference between the prediction module prediction output and the target output, so that the prediction module prediction output gradually approaches the target output, and the initial input features are optimized, i.e. the optimal grouting material proportioning input is obtained.

[0012] According to some embodiments, the present disclosure adopts the technical solutions as follows:

[0013] The cement-based grouting material proportioning optimization system based on deep reinforcement learning comprises:

[0014] The data acquisition module is configured to acquire original grouting material proportioning data;

[0015] The reinforcement learning reverse optimization module is configured to construct a deep reinforcement learning network, input the original grouting material proportioning data as initial input features into the deep reinforcement learning network, and output the initial predicted grouting parameters by a prediction module;

[0016] In the deep reinforcement learning network, a Q-Learning reinforcement update module is introduced, and the optimal grouting parameter target output is given. In the Q-Learning reinforcement update module, a dynamic environment adaptive update strategy is designed, and the input features are optimized in the reverse direction through the feedback mechanism and iterative adjustment. In the fine-tuning action of the input features, the input feature update mode and the update direction are selected, and the updated features are input into the prediction module. According to the difference between the prediction output of the prediction module and the target output, the dynamic iterative adjustment is performed, so that the prediction output of the prediction module gradually approaches the target output, and the optimization of the initial input features is realized, that is, the optimal grouting material ratio input is obtained.

[0017] According to some embodiments, the present disclosure adopts the following technical solutions:

[0018] A computer program product comprising a computer program which, when executed by a processor, implements the cement-based grouting material ratio optimization method based on deep reinforcement learning.

[0019] According to some embodiments, the present disclosure adopts the following technical solutions:

[0020] A non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the cement-based grouting material ratio optimization method based on deep reinforcement learning.

[0021] According to some embodiments, the present disclosure adopts the following technical solutions:

[0022] An electronic device comprising a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to make the electronic device execute the cement-based grouting material ratio optimization method based on deep reinforcement learning.

[0023] Compared with the prior art, the present disclosure has the following beneficial effects:

[0024] The cement-based grouting material ratio optimization method based on deep reinforcement learning of the present disclosure extracts valuable features from a large amount of historical data using a machine learning method from a data-driven perspective, establishes a CNN neural network model, and realizes high-precision grouting performance prediction through effective hyperparameter optimization, reduces errors caused by human intervention and subjective judgment in the grouting material research and development and performance test processes, and improves the scientificity and reliability of material research and design.

[0025] The cement-based grouting material proportioning optimization method based on deep reinforcement learning of the present disclosure utilizes the neural network back propagation mechanism to calculate the gradient of the loss function on the input features, and then realizes local gradient optimization; in the exploration mode, the new area is explored by means of the reinforcement learning ε-greedy strategy. The two modes are used randomly, which can realize local gradient optimization, ensure that each optimization is towards a better proportioning direction, and explore different input feature update paths in the optimization process, so as to jump out of the local trap and enhance the convergence speed. The method greatly widens the search space, is more flexible than the traditional optimization method due to the relatively fixed and limited optimization strategy, improves the accuracy of grouting material optimization design, and meets the diversified engineering requirements with higher efficiency and better effect.

[0026] The cement-based grouting material proportioning optimization method based on deep reinforcement learning of the present disclosure realizes the performance prediction and material optimization design automation and intelligentization on the basis of the traditional grouting material design process. The joint optimization based on reinforcement learning and neural network can not only effectively shorten the optimization time, but also adapt to multi-objective optimization of different engineering requirements, improve the efficiency and applicability of grouting material proportioning design, provide a reliable material selection scheme for fractured rock mass grouting engineering, and has important theoretical and practical value. BRIEF DESCRIPTION OF DRAWINGS

[0027] The drawings accompanying the specification of the present disclosure serve to provide further understanding of the present disclosure, and the illustrative embodiments of the present disclosure and the description thereof are used to explain the present disclosure, and do not constitute an improper limitation on the present disclosure.

[0028] Figure 1 The training flowchart of the deep reinforcement learning network of the embodiment of the present disclosure is shown in the figure.

[0029] Figure 2 The iterative optimization input feature flowchart of the embodiment of the present disclosure is shown in the figure. DETAILED DESCRIPTION

[0030] The present disclosure will be further described below in combination with the drawings and embodiments.

[0031] It should be pointed out that the following detailed description is exemplary and is intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present disclosure belongs.

[0032] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0033] Embodiment 1

[0034] In an embodiment of the present disclosure, a cement-based grouting material proportioning optimization method based on deep reinforcement learning is provided, comprising the following steps:

[0035] Step one: obtain the original grouting material proportioning data;

[0036] Step two: build a deep reinforcement learning network, input the original grouting material proportioning data as initial input features into the deep reinforcement learning network, and obtain the initial predicted output grouting parameters by the output of the prediction module;

[0037] In the deep reinforcement learning network, a Q-Learning reinforcement update module is introduced, given the optimal grouting parameter target output, a dynamic environment adaptive update strategy is designed in the Q-Learning reinforcement update module, and the input features are optimized in reverse through feedback mechanism iteration adjustment, the input feature update mode and update direction are selected in the fine-tuning action of the input features, the updated features are input into the prediction module, and dynamic iteration adjustment is performed according to the difference between the prediction module prediction output and the target output, so that the prediction module prediction output gradually approaches the target output, and the optimization of the initial input features is realized, i.e. the optimal grouting material proportioning input is obtained.

[0038] As an embodiment, the cement-based grouting material proportioning optimization method based on deep reinforcement learning of the present disclosure utilizes the learning ability of neural network to establish a unified material performance prediction and proportioning optimization design framework. Specifically, the machine learning neural network realizes the accurate prediction of grouting material performance by training and learning the nonlinear mapping relationship between material proportioning and performance. On the basis of the trained model, the ε-greedy strategy of reinforcement learning is introduced to iteratively adjust the initial material proportioning, and by setting a reasonable reward function and action selection strategy, the optimization process can be dynamically updated according to the prediction error, gradually approaching the designed grouting performance. The specific implementation process is as follows:

[0039] Step 1: obtain the original data of grouting material proportioning and grouting performance parameters, and pre-process and divide the original data to obtain the required data set;

[0040] Specifically, 1) the original data of the grouting material ratio and grouting parameters are obtained by using indoor tests, and the original data includes grouting parameters for characterizing the grouting performance of the slurry strength, fluidity, and setting time under different material ratios and conditions;

[0041] 2) The original data is preprocessed, including abnormal data cleaning, data interpolation supplement, and feature selection;

[0042] Specifically, abnormal data can be found by establishing existing data variation indicators, such as range, standard deviation, and coefficient of variation, and cleaned by deleting abnormal values and mean correction.

[0043] Data interpolation supplement can choose existing linear interpolation, Lagrange interpolation, etc., to ensure that the interpolated data can maintain the overall trend and structure of the data.

[0044] Further, the cement-based material ratio data in the original data is selected for feature selection, which can include principal component analysis, linear discriminant analysis, etc., to retain the key input features most useful for model prediction.

[0045] Further, the preprocessed original data is divided into three parts: training set, validation set, and test set, wherein the training set is used for model training, which can account for 70-80% of the total data set, the validation set is used for model hyperparameter tuning and model selection, which can account for 10-15% of the total data set, and the test set is used for evaluating the final performance of the model, which can account for 10-15% of the total data set.

[0046] Step 2: Establish a deep reinforcement learning network for grouting material ratio optimization, and use the training set for model training;

[0047] 1) The deep reinforcement learning network includes a prediction module, which uses a machine learning model to predict grouting parameters, and uses the training set to train the machine learning model and adjust the hyperparameters with the help of the validation set.

[0048] Specifically, the establishment of the machine learning model includes selecting a suitable network framework, configuring the network framework, defining the loss function, and selecting the optimizer.

[0049] First, select a suitable network framework, which can choose convolutional neural network CNN, recurrent neural network RNN, and multilayer perceptron MLP, and as the preferred choice, select convolutional neural network CNN;

[0050] Further, each layer of the network framework is configured, including setting the number of input layer nodes, selecting the appropriate number of layers and the number of neurons in each layer in the hidden layer, and setting the output layer.

[0051] Among them, the number of input layer nodes is the same as the number of input features, assuming that the number of input features retained after feature engineering is n, then the input layer dimension is (n, 1); the hidden layer includes convolutional layer, pooling layer and fully connected layer, here 2 convolutional layers are set, 16 filters are used in convolutional layer 1, the convolution kernel size is 1x3, the activation layer uses ReLU function, 32 filters are used in convolutional layer 2, the convolution kernel size is 1x3, and the activation layer uses ReLU function; a maximum pooling layer is set after each convolutional layer, with a window size of 1x2; finally, a flattening layer and a fully connected layer are set, the fully connected layer contains 64 neuron nodes, and uses ReLU function activation; the output layer needs 3 neuron nodes to match the prediction target form.

[0052] Define the loss function and select the optimizer, the loss function can be selected as mean square error (MSE) or mean absolute error (MAE), and the calculation formula is:

[0053]

[0054]

[0055] Wherein, n is the number of training samples, y i is the true output corresponding to the training data x i , and y i is the predicted value corresponding to the training data x t ; the optimizer can be selected as Adam algorithm, stochastic gradient descent, etc.

[0056] As preferred, the loss function is selected as mean square error (MSE), and the optimizer is selected as Adam algorithm.

[0057] The training process of the model includes forward propagation, loss calculation, backward propagation, algorithm optimization, and setting appropriate training rounds (epoch) and batch size (batch size), and the specific process is:

[0058] The initial input features are input into the prediction module for forward propagation, and the predicted grouting parameters are output;

[0059] The loss function is set to calculate the loss value between the current output and the true target; the gradient of the loss function with respect to the model weight is calculated, and the weight is updated by the optimizer during backward propagation;

[0060] The validation dataset is input into the model for hyperparameter tuning, and the optional hyperparameter tuning methods include grid search, random search, Bayesian optimization, etc.

[0061] The test set is input into the model for testing and evaluation, which should include index evaluation or cross-validation. Appropriate evaluation indexes include accuracy, precision, recall, F1 score, etc.

[0062] 2) The deep reinforcement learning network further comprises a Q-Learning reinforcement update module, which is arranged behind the machine learning prediction model, establishes a dynamic environment adaptive reinforcement learning model, initializes model parameters, and sets a reinforcement learning dynamic environment.

[0063] The Q-Learning method in reinforcement learning is improved and optimized, a dynamic environment adaptive update strategy is designed, and the input features of the neural network of the prediction module are inversely optimized through feedback mechanism iteration adjustment. This method can be regarded as constructing an optimization process on the basis of a trained neural network, selecting an update mode and an update direction in the fine-tuning action of the input features, inputting the updated features into the prediction module, adjusting the update direction according to the difference between the prediction output of the prediction module and the target output, and gradually making the prediction output of the neural network close to the target output in iteration.

[0064] Specifically, the setting of the reinforcement learning dynamic environment includes a reward function, a Q function update strategy, and an action set setting.

[0065] Among them, 1) the reward function is used to evaluate the effect of each operation, based on the difference between the actual output result and the target output result, and the negative mean square error is used as the reward. The reward R t obtained in the tthround of circulation is

[0066] R t =-(y pred -y target ) 2

[0067] Among them, y pred is the neural network prediction output, and y target is the target output.

[0068] 2) The Q function update strategy is designed to be dynamic environment adaptive. The Q function is a function used by reinforcement learning to evaluate the expected effect of optimization operation. In the Q-Learning algorithm, the Q function is updated gradually with iteration, and the learning rate a controls the influence of new information on old information, and its update formula is:

[0069]

[0070] Among them, x t is the input feature in the tthround of circulation; Δx t is the update direction of the input x t in the tthround of circulation; Q(x t , Δx t ) is the Q value in the tthround of circulation; a is the learning rate, which is used to control the iteration update degree; and R tis the reward obtained in the t-th round of the cycle, also known as the immediate reward; is the maximum Q value that can be obtained by taking the fine-tuning action in the action set in the t+1-th round of the cycle, also known as the future reward; γ t is a discount factor, used to balance the immediate reward and the future reward, and the value range is [0, 1].

[0071] Specifically, during the training process, the learning rate α and the discount factor γ in the Q value are dynamically adjusted according to the changes of the state and the action space. For the learning rate α, if the reward does not significantly improve after the Q value is updated within a certain period, the learning rate can be increased. Conversely, if the reward gradually increases, the learning rate is reduced, and the adjustment formula is:

[0072]

[0073] where λ α is a hyperparameter that controls the learning rate adjustment speed, R t is the reward obtained in the t-th round of the cycle, μ and σ are the average value and the standard deviation of the reward within a certain fixed time window,

[0074]

[0075] where N is the window size, which can be set to 100. For the discount factor γ, by observing the prediction accuracy of the model at different time steps, when the model's prediction of long-term rewards is stable, gradually increase the value of γ to pay more attention to future rewards, and the adjustment formula is:

[0076]

[0077] where λ γ is a hyperparameter that controls the increasing rate of γ, ΔQ is the change of Q value, ΔQ = Q(x t+1 , Δx t+1 )-Q(x t , Δx t ), and Q max is the current maximum Q value.

[0078] 3) Action set setting, the action is a small adjustment to the input feature, and all input features are fine-tuned to form an action set, and the specific setting process is as follows: for each input feature, set three fixed adjustment amplitudes of increasing 5%, decreasing 5% and unchanged, then for n input features, the action set will include 3 n actions.

[0079] Further, the model parameter initialization includes initializing the Q function, the initial input feature, and the target output.

[0080] Specifically, the Q function can be initially set to 0 or a random value. The initial input x0, i.e., the initial grouting material ratio, can be selected based on historical data or prior knowledge. The target output y target , i.e., the specific values of the target slurry performance parameters such as compressive strength, viscosity, and setting time, are selected based on the grouting design requirements.

[0081] Step 3: Through the feedback mechanism, the input features are iteratively adjusted and optimized. In the fine-tuning action of the input features, the input feature update mode and update direction are selected. The updated features are input into the prediction module. Based on the difference between the prediction output of the prediction module and the target output, dynamic iterative adjustment is performed. In the iteration, the prediction output of the prediction module gradually approaches the target output, achieving optimization of the initial input features, i.e., obtaining the optimal grouting material ratio input.

[0082] In the fine-tuning action of the input features, the input feature update mode and update direction are selected, including: randomly selecting the backpropagation algorithm of the neural network model or the exploration strategy of reinforcement learning to iteratively update the initial input value. The backpropagation algorithm is used for local search optimization of the input features, and the exploration strategy of reinforcement learning is used for global search. Finally, the optimal grouting material ratio based on the target grouting parameters is obtained.

[0083] Specifically, the iterative update includes three steps: forward propagation, selection of input feature update mode, and judgment of iteration progress.

[0084] Step 1) Forward propagation: input the initial input features x0 into the trained machine learning model (prediction module) for prediction to obtain the current output prediction y′ pred ;

[0085] Step 2) Select the input feature update mode: randomly select the backpropagation update or use the exploration mechanism of the reinforcement learning ε-greedy strategy to guide the direction of the new input features. Backpropagation can provide a clear gradient direction to accelerate convergence, while reinforcement learning can help the model find possible good directions in the early stage. This combined mode can explore new input directions with a certain probability, rather than relying entirely on the gradient direction, helping to jump out of local optimization and making the prediction output of the input features calculated by the neural network approach the target output.

[0086] Specifically, first, as shown in Figure 2 , a suitable exploration rate is set, with a value range of [0, 1]. At the same time, a number r is randomly generated, with a value range of [0, 1], and compared with the exploration rate ε. If r < ε, enter the exploration mode; if r > ε, enter the utilization mode.

[0087] In the exploration mode, the update direction Δx t is selected based on the ε-greedy strategy, and the calculation formula is:

[0088]

[0089] where x t is the input feature in the t-th round of iteration; random action means randomly selecting a fine-tuning action from the action set as the update direction; argmax Δx Q(x t ,Δx) means selecting the fine-tuning action that makes the maximum value of Q as the update direction; ε is the exploration rate.

[0090] Then, the input feature is updated according to the selected update direction, and the formula is:

[0091] x t+1 = x t + Δx t

[0092] Then, the updated input feature x t+1 is input into the prediction module to obtain the new prediction output y pre d .

[0093] Further, the mode is to update the input feature by means of back propagation mechanism. The loss L between the current output prediction and the target output is calculated. The calculation formula is:

[0094]

[0095] Further, the gradient of the loss L with respect to the input feature x is calculated, and the calculation formula is:

[0096]

[0097] where x is the input feature, is the gradient of the loss L with respect to the prediction output y′ pred , is the gradient of the prediction output y′ pred with respect to the input feature x.

[0098] The input feature x grad in the gradient direction is obtained by using the adaptive learning rate Adam to update the input feature by back propagation. The update formula is:

[0099]

[0100] where η is the learning rate.

[0101] The updated input feature x grad is input into the neural network to obtain the new prediction output y pred .

[0102] Step 3) judging the iteration process, calculating the current predicted output y pred and the target output y target , the loss function L is:

[0103]

[0104] checking whether the loss function L and the iteration number t reach the threshold value, if yes, stopping the iteration, if not, calculating the reward value R t , and optimizing the Q function using the update strategy and the update function. The iteration repeats the steps of forward propagation, selecting the input feature update mode, judging the iteration process, and switching between exploration and utilization, updating the input feature, until the loss function converges to the preset requirement or the iteration number reaches the threshold value, and finally outputting the updated input feature, i.e. the optimized grouting material ratio.

[0105] Embodiment 2

[0106] In an embodiment of the present disclosure, a cement-based grouting material ratio optimization system based on deep reinforcement learning is provided, comprising:

[0107] a data acquisition module for acquiring original grouting material ratio data;

[0108] a reinforcement learning reverse optimization module for constructing a deep reinforcement learning network, inputting the original grouting material ratio data as initial input features into the deep reinforcement learning network, and obtaining the initial predicted output grouting parameters by the prediction module output;

[0109] an Q-Learning reinforcement update module is introduced into the deep reinforcement learning network, given the optimal grouting parameter target output, a dynamic environment adaptive update strategy is designed in the Q-Learning reinforcement update module, the input features are iteratively adjusted through the feedback mechanism to realize the reverse optimization of the input features, the input feature update mode and the update direction are selected in the fine-tuning action of the input features, the updated features are input into the prediction module, and the difference between the prediction module prediction output and the target output is iteratively adjusted dynamically, so that the prediction module prediction output gradually approaches the target output, realizing the optimization of the initial input features, i.e. obtaining the optimal grouting material ratio input.

[0110] Embodiment 3

[0111] In an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the cement-based grouting material ratio optimization method based on deep reinforcement learning.

[0112] Embodiment 4

[0113] In an embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the cement-based grouting material proportioning optimization method based on deep reinforcement learning.

[0114] Embodiment 5

[0115] In an embodiment of the present disclosure, an electronic device is provided, comprising a processor, a memory, and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device implements the cement-based grouting material proportioning optimization method based on deep reinforcement learning.

[0116] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices generate a device that implements the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in one or more flows or blocks.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable data processing device to generate a computer-implemented process, so that the instructions executed by the computer or other programmable data processing device provide a process for implementing the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 a step that performs the functions specified in one or more flows or blocks.

[0118] Although the specific embodiments of the present disclosure are described above with reference to the accompanying drawings, the present disclosure is not limited to the specific embodiments described above, and various modifications or changes can be made to the specific embodiments without departing from the scope of the present disclosure.

Claims

1. A method for optimizing the mix proportion of cement-based grouting materials based on deep reinforcement learning, characterized in that, include: Obtain the original grouting material mix ratio data; A deep reinforcement learning network is constructed, and the original grouting material ratio data is used as the initial input feature to the deep reinforcement learning network. The grouting parameters of the initial prediction output are obtained by using the prediction module output. A Q-Learning reinforcement update module is introduced into the deep reinforcement learning network. Given the optimal grouting parameters and target output, a dynamic environment adaptation update strategy is designed in the Q-Learning reinforcement update module. The input features are inversely optimized through iterative adjustment via a feedback mechanism. In the fine-tuning action of the input features, the input feature update mode and update direction are selected. The updated features are input into the prediction module. Dynamic iterative adjustment is performed based on the difference between the prediction output of the prediction module and the target output. In the iteration, the prediction output of the prediction module gradually approaches the target output, thereby optimizing the initial input features and obtaining the optimal grouting material ratio input. In the fine-tuning of input features, the input feature update mode and direction are selected, including: randomly selecting backpropagation update or using reinforcement learning. - A greedy strategy exploration mechanism guides the direction of new input features, iteratively updates the initial input value, uses the backpropagation algorithm for local search optimization of input features, and uses a reinforcement learning exploration strategy for global search, so that the predicted output obtained by the neural network calculation of the input features approximates the target output, and finally obtains the optimized grouting material ratio based on the target grouting parameters.

2. The method for optimizing the mix proportion of cement-based grouting materials based on deep reinforcement learning as described in claim 1, characterized in that, The output grouting parameters include grout strength, fluidity, and setting time. After obtaining the original grouting material ratio data, it is preprocessed, including abnormal data cleaning, data interpolation, and feature selection, retaining the most critical input features for the predicted output.

3. The method for optimizing the mix proportion of cement-based grouting materials based on deep reinforcement learning as described in claim 1, characterized in that, The prediction module of the deep reinforcement learning network is a machine learning network model. The network framework of the machine learning network model is configured. The configuration of each layer of the network framework includes setting the number of input layer nodes, selecting an appropriate number of hidden layers and the number of neurons in each layer, and setting the output layer. The number of input layer nodes is the same as the number of input features. If the number of input features is n, then the dimension of the input layer is (n,1). The machine learning network model outputs the predicted injection parameters by propagating the initial input features through the forward propagation of the model.

4. The method for optimizing the mix proportion of cement-based grouting materials based on deep reinforcement learning as described in claim 1, characterized in that, The Q-Learning reinforcement update module is a reinforcement learning dynamic environment. The setting of the reinforcement learning dynamic environment includes a reward function, Q The function update strategy and action set are configured, and the reward function evaluates the effect of each operation. Based on the difference between the output result and the target output result, the negative mean squared error is used as the reward. t Rewards obtained in the cycle The calculation formula is: in, For the neural network to predict output, Output the target.

5. The method for optimizing the mix proportion of cement-based grouting materials based on deep reinforcement learning as described in claim 1, characterized in that, The Q-Learning enhancement update module utilizes Q The function update strategy evaluates the expected effect of the optimization operation by introducing the learning rate and discount factor. Q The function is updated iteratively, and the learning rate controls the impact of new information on old information, dynamically adjusting according to changes in the state and action space. Q The learning rate and discount factor in the value; for the learning rate, if within a certain period, Q If the reward does not increase significantly after the value is updated, the learning rate is increased; conversely, if the reward gradually increases, the learning rate is decreased.

6. A cement-based grouting material mix optimization system based on deep reinforcement learning, characterized in that, include: The data acquisition module is used to acquire the original grouting material ratio data; The reinforcement learning reverse optimization module is used to construct a deep reinforcement learning network. The original grouting material ratio data is used as the initial input feature to the deep reinforcement learning network, and the grouting parameters are obtained by using the output of the prediction module. A Q-Learning reinforcement update module is introduced into the deep reinforcement learning network. Given the optimal grouting parameters and target output, a dynamic environment adaptation update strategy is designed in the Q-Learning reinforcement update module. The input features are inversely optimized through iterative adjustment via a feedback mechanism. In the fine-tuning action of the input features, the input feature update mode and update direction are selected. The updated features are input into the prediction module. Dynamic iterative adjustment is performed based on the difference between the prediction output of the prediction module and the target output. In the iteration, the prediction output of the prediction module gradually approaches the target output, thereby optimizing the initial input features and obtaining the optimal grouting material ratio input. In the fine-tuning of input features, the input feature update mode and direction are selected, including: randomly selecting backpropagation update or using reinforcement learning. - A greedy strategy exploration mechanism guides the direction of new input features, iteratively updates the initial input value, uses the backpropagation algorithm for local search optimization of input features, and uses a reinforcement learning exploration strategy for global search, so that the predicted output obtained by the neural network calculation of the input features approximates the target output, and finally obtains the optimized grouting material ratio based on the target grouting parameters.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for optimizing the proportion of cement-based grouting materials based on deep reinforcement learning as described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the cement-based grouting material proportion optimization method based on deep reinforcement learning as described in any one of claims 1-5.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the method for optimizing the proportion of cement-based grouting materials based on deep reinforcement learning as described in any one of claims 1-5.

Images