Deep reinforcement learning-driven drilling parameter intelligent real-time optimization method

Abstract

The invention discloses a deep reinforcement learning-driven intelligent real-time optimization method for drilling parameters, which belongs to the technical field of drilling optimization, and is technically characterized by comprising the following steps of: 1, constructing a drilling speed prediction model; step 2, modeling in a decision-making process: selecting a group of parameter combinations # imgabs0 # to obtain the optimal drilling speed, only selecting one parameter # imgabs2 # at each moment # imgabs1 # needing decision-making, and representing the MDP as a quintuple # imgabs3 #; each drilling parameter is decided one by one, and the previous decision parameter and drilling data are input; and step 3, obtaining a state # imgabs5 # from the environment at each time step # imgabs4 #, inputting the state # imgabs5 # into an Agent, obtaining an action # imgabs6 # and the logarithmic probability # imgabs7 # of the current strategy for updating the subsequent strategy of the algorithm, and achieving the optimization of the bit pressure, the torque and the rotating speed parameters.

Description

Technical Field

[0001] The present invention relates to the technical field of drilling optimization, and in particular to a method for intelligent real-time optimization of drilling parameters driven by deep reinforcement learning. Background Art

[0002] Drilling parameter optimization is a core component in improving the efficiency, safety, and economics of oil and gas drilling development, and its impact persists throughout the entire drilling lifecycle. Technically, this field achieves dynamic adaptation of the drilling system to the geological environment by dynamically adjusting key parameters such as drilling pressure, rotational speed, displacement, and drilling fluid properties. Engineering-wise, its core value lies in the coordinated optimization of the trinity of "speed improvement, cost reduction, and risk control." Under complex geological conditions, scientifically adjusting key parameters such as drilling pressure, rotational speed, and displacement can significantly increase the rate of penetration (ROP), shorten drilling cycles by 20%-30%, and directly reduce development costs. Furthermore, precisely matching parameter combinations to formation characteristics can reduce drill tool vibration, wear, and downhole accidents (such as kicks and stuck pipe), ensuring operational safety. Furthermore, optimizing hydraulic parameters enhances cuttings carryover efficiency, maintains wellbore cleanliness, and mitigates the risk of wellbore instability. With the integration of intelligent technologies, real-time data-driven dynamic parameter adjustments have further broken through the limitations of traditional empirical models, enabling drilling systems to become "adaptive." This not only extends drill bit life and reduces non-productive time, but also provides technical support for the low-cost development of unconventional resources such as deepwater and shale. Drilling parameter optimization is a key means of balancing the contradictions of "speeding up, reducing costs, and controlling risks," and is an indispensable technological cornerstone for driving the oil and gas industry's transformation toward intelligent, efficient, green, and low-carbon development.

[0003] Existing technologies for drilling parameter optimization primarily rely on empirical models, physical equations, and offline optimization algorithms, which present significant limitations. Traditional methods (such as mechanical specific energy models and drilling rate equations) rely on expert experience to pre-determine parameter relationships, making them difficult to dynamically adapt to complex and changing formation environments. While offline optimization methods based on genetic algorithms (GA) or particle swarm optimization (PSO) can address multi-objective problems, they suffer from low computational efficiency and reliance on high-quality historical data, making them unable to respond to unexpected downhole conditions in real time. Furthermore, traditional models lack the ability to analyze nonlinear coupling factors (such as the dynamic interaction between drilling pressure, rotational speed, and torque) and are susceptible to noise from downhole sensors, resulting in conservative optimization results or deviations from actual requirements.

[0004] Compared to evolutionary computing, reinforcement learning offers advantages in sample efficiency, high-dimensional space modeling, and online learning. In terms of sample efficiency, reinforcement learning can achieve efficient learning with a small number of high-value samples. By combining it with an experience replay mechanism, reinforcement learning can significantly improve sample utilization and reduce trial-and-error costs. In terms of high-dimensional space modeling, reinforcement learning methods combined with deep neural networks can handle continuous actions and state spaces, making them suitable for complex physical systems. In terms of online learning, reinforcement learning supports online learning, where strategies can be adjusted in real time based on environmental feedback. It can instantly optimize equipment parameters based on sensor data, improving production efficiency. Evolutionary algorithms, on the other hand, typically require offline iterations, making them difficult to adapt to scenarios with high real-time requirements. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the purpose of the embodiments of the present invention is to provide a deep reinforcement learning-driven intelligent real-time optimization method for drilling parameters to solve the problems in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A deep reinforcement learning-driven intelligent real-time optimization method for drilling parameters includes the following steps:

[0008] Step 1: Build a drilling rate prediction model. Collect historical drilling operation data, clean it, and use correlation analysis to extract highly correlated parameters as input data for the prediction model to build an XGBoost model.

[0009] Step 2: Modeling the decision process, which involves selecting a set of parameter combinations , so that it can achieve the best drilling speed and make decisions at every moment Select only one parameter , the MDP is represented as a five-tuple ,in represents the set of valid actions, Indicates the parameters Perform the setting operation. The value range of is the action space. According to the physical definitions of drilling pressure, drilling speed and torque, its value range is non-negative.

[0010] Each drilling parameter is decided one by one, and its input is the previous decision parameter and drilling data, and its output is the distribution , at each time step After the parameter selection is completed, the parameters are input into the prediction model to generate the predicted drilling speed. And calculate the reward function and update the agent parameters through the RL algorithm;

[0011] Step 3: Overall framework structure, at each time step , get the state from the environment Get action from input agent The logarithmic probability of the current policy Used for subsequent strategy updates of the algorithm, and then from The distribution represented Sampling a specific parameter ,Will Input into the prediction model to obtain the predicted drilling speed , after obtaining the drilling speed, the environment will give rewards and next status And update the end mark , the decision process of the current time step is completed, and the tuple is stored in the experience replay buffer pool.

[0012] As a further solution of the present invention, the XGBoost model in step 1 is evaluated, and if the performance is not excellent, the parameters are adjusted until the performance is excellent.

[0013] As a further solution of the present invention, the correlation analysis data analysis method in step 1 uses the Pearson correlation coefficient to perform data correlation analysis. The coefficient ranges from -1 to 1. A value close to 1 or -1 indicates that there is a strong linear relationship between the two variables.

[0014] As a further solution of the present invention, the XGBoost model in step 1 uses the root mean square error as an indicator for evaluating the accuracy of the prediction model. The closer the RMSE is to 0, the closer the prediction result is to the actual observation value.

[0015] As a further solution of the present invention, the step 2 also includes adding a numerical upper limit , is the maximum drilling pressure, speed and torque that the machine can achieve in actual conditions, that is, the action space is , is the distribution output from the Agent Sampling.

[0016] As a further solution of the present invention, the step 2 Represents a set of valid states, State of the moment is the concatenation of the drilling data and the distribution of the last output, that is, ,in represents drilling data, is the distribution parameter of the last decision.

[0017] As a further solution of the present invention, the step 2 represents the reward function, which is set to the difference between the predicted drilling speed and the original drilling speed, i.e. , To optimize the actual drilling speed before, in the middle step of the sequence decision, that is, When using the expected drilling rate instead of .

[0018] As a further solution of the present invention, the step 2 represents the discount factor, .

[0019] In summary, the embodiments of the present invention have the following beneficial effects compared with the prior art:

[0020] This algorithm uses machine learning algorithms to construct drilling rate prediction models and optimization models, realizing real-time analysis of drilling operation parameters and selection of optimization parameters. The drilling rate is increased by 5% to 10% compared with the pre-optimization level, which has practical significance for increasing drilling operation efficiency.

[0021] In order to more clearly illustrate the structural features and effects of the present invention, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flowchart of the prediction model training of an embodiment of the invention.

[0023] Figure 2 This is a comparison chart of predicted values ​​and true values ​​in an embodiment of the invention.

[0024] Figure 3 This is a model decision flow chart in an embodiment of the invention.

[0025] Figure 4 Figure 2 is a diagram of the agent network structure in an embodiment of the invention.

[0026] Figure 5 It is a diagram of the overall interaction framework in an embodiment of the invention.

[0027] Figure 6 This is a comparison chart of drilling speed before and after the embodiment of the invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0029] The specific implementation of the present invention is described in detail below with reference to specific embodiments.

[0030] In one embodiment, a deep reinforcement learning driven intelligent real-time optimization method for drilling parameters is described. Figures 1 to 6 , including the following steps:

[0031] Step 1: Build a drilling rate prediction model. Collect historical drilling operation data, clean it, and use correlation analysis to extract highly correlated parameters as input data for the prediction model to build an XGBoost model.

[0032] Step 2: Modeling the decision process, which involves selecting a set of parameter combinations , so that it can achieve the best drilling speed and make decisions at every moment Select only one parameter , the MDP is represented as a five-tuple ,in represents the set of valid actions, Indicates the parameters Perform the setting operation. The value range of is the action space. According to the physical definitions of drilling pressure, drilling speed and torque, its value range is non-negative.

[0033] Each drilling parameter is decided one by one, and its input is the previous decision parameter and drilling data, and its output is the distribution , at each time step After the parameter selection is completed, the parameters are input into the prediction model to generate the predicted drilling speed. And calculate the reward function and update the agent parameters through the RL algorithm;

[0034] Step 3: Overall framework structure, at each time step , get the state from the environment Get action from input agent The logarithmic probability of the current policy Used for subsequent strategy updates of the algorithm, and then from The distribution represented Sampling a specific parameter ,Will Input into the prediction model to obtain the predicted drilling speed , after obtaining the drilling speed, the environment will give rewards and next status And update the end mark , the decision process of the current time step is completed, and the tuple is stored in the experience replay buffer pool.

[0035] For further information, see Figures 1 to 6, the XGBoost model in step 1 is evaluated, and if the performance is not good, the parameters are adjusted until the performance is good.

[0036] For further information, see Figures 1 to 6 The XGBoost model in step 1 uses the root mean square error as an indicator to evaluate the accuracy of the prediction model. The closer the RMSE is to 0, the closer the prediction result is to the actual observation value.

[0037] For further information, see Figures 1 to 6 , the step 2 also includes adding a numerical upper limit , is the maximum drilling pressure, speed and torque that the machine can achieve in actual conditions, that is, the action space is , is the distribution output from the Agent Sampling.

[0038] For further information, see Figures 1 to 6 , in the step 2 Represents a set of valid states, State of the moment is the concatenation of the drilling data and the distribution of the last output, that is, ,in represents drilling data, is the distribution parameter of the last decision.

[0039] For further information, see Figures 1 to 6 , in the step 2 represents the reward function, which is set to the difference between the predicted drilling speed and the original drilling speed, i.e. , To optimize the actual drilling speed before, in the middle step of the sequence decision, that is, When using the expected drilling rate instead of .

[0040] In the step 2 represents the discount factor, .

[0041] In this embodiment, one: drilling rate prediction model construction

[0042] The process of building a drilling speed prediction model is as follows: Figure 1 As shown in the figure, historical drilling operation data is collected and cleaned. After that, correlation analysis and other data analysis methods are used to extract highly correlated parameters as input data for the prediction model to build the XGBoost model. The data is then used for training. After training, the model performance is calculated and evaluated. If the performance is not good, the parameters are adjusted until the performance is good.

[0043] Data correlation analysis was performed using the Pearson correlation coefficient. The Pearson correlation coefficient is a statistic that measures the strength and direction of the linear relationship between two variables. Its value ranges from -1 to 1, with values ​​closer to 1 or -1 indicating a stronger linear relationship between the two variables. Based on the calculation results, the 15 parameters with the highest correlation with ROP (i.e., hanging weight, bit depth, bit vertical depth, well depth (m), vertical depth (m), inlet flow rate (log), inlet flow rate, delay time, bit pressure drop, hook speed, wellhead regulated pressure, additional back pressure, bit pressure, torque, and rotational speed) were used as predictor variables to train a regression model to predict ROP.

[0044] The root mean square error (RMSE) is used as an indicator to evaluate the accuracy of the prediction model. RMSE is a commonly used regression model evaluation indicator used to measure the difference between the predicted value and the true value. It is generally believed that the closer the RMSE is to 0, the closer the prediction result is to the actual observed value. The RMSE of this prediction model is 0.88, which can be considered effective. The prediction value of the prediction model on the test set is compared with the true value. Figure 2 shown.

[0045] 2: Markov decision process modeling and agent design

[0046] The core of reinforcement learning lies in transforming the problem into a Markov decision process. The following details how to transform the drilling parameter optimization problem into a Markov decision process, the setting of the reward function, and the specific design of the agent.

[0047] First, the drilling parameter optimization process is transformed into a sequential decision process. The drilling parameter optimization problem in this invention is defined as: based on a set of drilling data at a certain moment, a set of parameter combinations is selected. , so that it can achieve the best drilling speed. Generally speaking, the search space for selecting a set of parameters at a time is too large to be handled, so at each moment when a decision is required Select only one parameter , which greatly reduces the search space and converts it into a sequential decision process. The parameters to be optimized in this invention are only WOB, RPM and torque, so .

[0048] After converting it into a sequential decision process, the problem can be further converted into a Markov decision process (MDP), which is usually represented as a five-tuple ,in represents the set of valid actions, Indicates the parameters Perform the setting operation. The value range of is the action space. According to the physical definition of drilling pressure, drilling speed and torque, its value range is non-negative. This method adds an additional upper limit to it. , is the maximum drilling pressure, speed and torque that the machine can achieve in actual conditions, that is, the action space is . is the distribution output from the Agent Sampling is done to explore more fully, which will be described in detail in the construction of the reinforcement learning framework below. Represents a set of valid states, and it is hoped that its decision will be affected by the drilling data and the last decision. State of the moment is the concatenation of the drilling data and the distribution of the last output, that is, ,in represents drilling data, is the distribution parameter of the last decision. Represents the reward function. For the design of the reward function, the agent should be encouraged to explore the direction of increasing the drilling speed as much as possible. Therefore, the reward function is set to the difference between the predicted drilling speed and the original drilling speed, that is, , is the actual drilling speed before optimization. In the middle step of the sequence decision, When the parameter is incomplete, the expected drilling speed should be used instead. , where the expected drilling rate is approximated using the current incomplete parameters. is the transition probability function, which is unknown to our problem. represents the discount factor, where .

[0049] A complete decision-making process of the agent is as follows Figure 3 As shown. The agent makes decisions on each drilling parameter one by one. Its input is the previous decision parameter and drilling data, and its output is the distribution At each time step After the parameter selection is completed, the parameters are input into the prediction model to generate the predicted drilling speed. The reward function is calculated and the agent parameters are updated through the RL algorithm, so that the agent can learn how to adjust the drilling parameters.

[0050] The internal structure of the Agent is as follows Figure 4 As shown, the Agent is constructed using the Actor-Critic structure, and its network structure is the input layer-fully connected layer-output layer architecture, where the Actor has two output layers, which output the distributed and parameter.

[0051] Three: Overall frame structure

[0052] The reinforcement learning algorithm framework of the present invention is as follows Figure 5 As shown. At each time step , get the state from the environment Get action from input agent The logarithmic probability of the current policy Used for subsequent strategy updates of the algorithm, and then from The distribution represented Sampling a specific parameter ,Will Input into the prediction model to obtain the predicted drilling speed , after obtaining the drilling speed, the environment will give rewards and next status And update the end mark , the decision process of the current time step is completed, and the tuple Stored in the experience replay buffer pool.

[0053] By building a ROP prediction model as environmental feedback, the reinforcement learning model uses ROP as a reward signal to learn how to adjust drilling parameters, thereby improving ROP. The specific intelligent algorithm used in model construction is as follows.

[0054] The prediction model uses the XGBoost model. XGBoost (eXtreme Gradient Boosting) is an efficient, flexible, and powerful machine learning algorithm widely used in data science and machine learning. It is an optimized implementation of the Gradient Boosting algorithm, particularly adept at processing structured data, and has achieved excellent results in many machine learning competitions, such as Kaggle. Its efficiency and flexibility provide strong support for the construction of the drilling rate prediction model used in this method.

[0055] The reinforcement learning model uses Proximal Policy Optimization (PPO), an online policy gradient method whose core concept is to avoid large fluctuations during training by constraining the difference between the new policy and the old policy. It improves on traditional policy gradient methods by introducing a clipping mechanism or a KL divergence penalty term to ensure that the magnitude of policy updates remains within a reasonable range.

[0056] The parameters of torque, speed and drilling pressure were optimized for an oil field. The experimental results are as follows: Figure 6 As shown in the figure, the drilling speed generally increased by 5% to 10%, and the average drilling speed increased by 6.874%.

[0057] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A deep reinforcement learning driven intelligent real-time optimization method for drilling parameters, characterized in that: The following steps are involved: Step 1: Build a drilling rate prediction model. Collect historical drilling operation data, clean it, and use correlation analysis to extract highly correlated parameters as input data for the prediction model to build an XGBoost model. Step 2: Modeling the decision process, which involves selecting a set of parameter combinations , so that it can achieve the best drilling speed and make decisions at every moment Select only one parameter , the MDP is represented as a five-tuple ,in represents the set of valid actions, Indicates the parameters Perform the setting operation. The value range of is the action space. According to the physical definitions of drilling pressure, drilling speed and torque, its value range is non-negative. Each drilling parameter is decided one by one, and its input is the previous decision parameter and drilling data, and its output is the distribution , at each time step After the parameter selection is completed, the parameters are input into the prediction model to generate the predicted drilling speed. And calculate the reward function and update the agent parameters through the RL algorithm; Step 3: Overall framework structure, at each time step , get the state from the environment Get action from input agent The logarithmic probability of the current policy Used for subsequent strategy updates of the algorithm, and then from The distribution represented Sampling a specific parameter ,Will Input into the prediction model to obtain the predicted drilling speed , after obtaining the drilling speed, the environment will give rewards and next status And update the end mark , the decision process of the current time step is completed, and the tuple is stored in the experience replay buffer pool.

2. The deep reinforcement learning driven intelligent real-time optimization method for drilling parameters according to claim 1, characterized in that: The XGBoost model in step 1 is evaluated, and if the performance is not good, the parameters are adjusted until the performance is good.

3. The deep reinforcement learning driven intelligent real-time optimization method for drilling parameters according to claim 2, characterized in that: The correlation analysis data analysis method in the step 1 uses the Pearson correlation coefficient to perform data correlation analysis. The coefficient ranges from -1 to 1. A value close to 1 or -1 indicates that there is a strong linear relationship between the two variables.

4. The deep reinforcement learning driven intelligent real-time optimization method for drilling parameters according to claim 3, characterized in that: The XGBoost model in step 1 uses the root mean square error as an indicator to evaluate the accuracy of the prediction model. The closer the RMSE is to 0, the closer the prediction result is to the actual observed value.

5. The deep reinforcement learning driven intelligent real-time optimization method for drilling parameters according to claim 1, characterized in that: The step 2 also includes adding a numerical upper limit , is the maximum drilling pressure, speed and torque that the machine can achieve in actual conditions, that is, the action space is , is the distribution output from the Agent Sampling.

6. The deep reinforcement learning driven intelligent real-time optimization method for drilling parameters according to claim 5, characterized in that: In the step 2 Represents a set of valid states, State of the moment is the concatenation of the drilling data and the distribution of the last output, that is, ,in represents drilling data, is the distribution parameter of the last decision.

7. The deep reinforcement learning driven intelligent real-time optimization method for drilling parameters according to claim 6, characterized in that: In the step 2 represents the reward function, which is set to the difference between the predicted drilling speed and the original drilling speed, i.e. , To optimize the actual drilling speed before, in the middle step of the sequence decision, that is, When using the expected drilling rate instead of .

8. The deep reinforcement learning driven intelligent real-time optimization method for drilling parameters according to claim 7, characterized in that: In the step 2 represents the discount factor, .

Images