A Deep Reinforcement Learning-Driven Intelligent Real-Time Optimization Method for Drilling Parameters

Abstract

This invention discloses a deep reinforcement learning-driven intelligent real-time optimization method for drilling parameters, belonging to the field of drilling optimization technology. Its key technical points include the following steps: Step 1: Construction of a drilling rate prediction model; Step 2: Modeling of the decision-making process, where the decision-making process involves selecting a set of parameters to achieve the optimal drilling rate. At each decision-making moment, only one parameter is selected, and the MDP is represented as a quintuple; each drilling parameter is decided sequentially, with the input being the previous decision parameter and drilling data; Step 3: Overall framework structure, at each time step, the logarithmic probability of the action and the current policy is obtained from the state input from the environment in the Agent and used for subsequent policy updates in the algorithm. This method has the advantage of optimizing drilling pressure, torque, and rotational speed parameters.

Description

Technical Field

[0001] This invention relates to the field of drilling optimization technology, specifically to a deep reinforcement learning-driven intelligent real-time optimization method for drilling parameters. Background Technology

[0002] Drilling parameter optimization is a core element in improving the efficiency, safety, and economy of oil and gas drilling development, and its impact extends throughout the entire drilling operation lifecycle. Technically, this field achieves dynamic adaptation of the drilling system to the geological environment by dynamically adjusting key parameters such as drilling pressure, rotational speed, displacement, and drilling fluid properties. Engineeringly, its core value lies in the synergistic optimization of "speed increase, cost reduction, and risk control": Under complex geological conditions, scientifically adjusting key parameters such as drilling pressure, rotational speed, and displacement can significantly improve the rate of penetration (ROP), shorten the drilling cycle by 20%-30%, and directly reduce development costs. Simultaneously, precisely matched parameter combinations can reduce drill string vibration, wear, and downhole accidents (such as well kicks and stuck pipe), ensuring operational safety. Furthermore, optimizing hydraulic parameters can enhance cuttings carrying efficiency, maintain wellbore cleanliness, and avoid the risk of wellbore instability. With the integration of intelligent technologies, real-time data-driven dynamic parameter adjustment has further broken through the limitations of traditional experience-based models, enabling drilling systems to possess "adaptive" capabilities. This not only extends drill bit life and reduces non-productive time but also provides technical support for the low-cost development of unconventional resources such as deepwater and shale. It can be said that drilling parameter optimization is a key means of balancing the contradictions of "speeding up, reducing costs, and controlling risks," and an indispensable technological foundation for promoting the oil and gas industry's transformation towards intelligent, efficient, green, and low-carbon practices.

[0003] Existing technologies for drilling parameter optimization mainly rely on empirical models, physical equations, and offline optimization algorithms, which have significant limitations. Traditional methods (such as mechanical energy specificity models and drilling rate equations) depend on expert experience to pre-determine parameter relationships, making it difficult to dynamically adapt to complex and changing formation environments. While offline optimization based on genetic algorithms (GA) or particle swarm optimization (PSO) can handle multi-objective problems, it suffers from low computational efficiency and relies on high-quality historical data, making it unable to respond in real time to sudden downhole conditions. Furthermore, traditional models lack the analytical ability to handle nonlinear coupling factors (such as the dynamic interaction between drilling pressure, rotational speed, and torque) and are susceptible to noise interference from downhole sensors, leading to conservative optimization results or deviations from actual requirements.

[0004] Compared to evolutionary computation, reinforcement learning offers advantages in three areas: sample efficiency, high-dimensional space modeling, and online learning. Regarding sample efficiency, reinforcement learning can achieve efficient learning with a small number of high-value samples. By incorporating experience replay mechanisms, it can significantly improve sample utilization and reduce trial-and-error costs. In terms of high-dimensional space modeling, reinforcement learning methods combined with deep neural networks can handle continuous actions and state spaces, making them suitable for complex physical systems. Regarding online learning, reinforcement learning supports online learning, and policies can be adjusted in real-time based on environmental feedback. It can optimize equipment parameters instantly based on sensor data, improving production efficiency, while evolutionary algorithms typically require offline iteration and are difficult to adapt to scenarios with high real-time requirements. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide a deep reinforcement learning-driven intelligent real-time optimization method for drilling parameters, thereby solving the problems mentioned in the background.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A deep reinforcement learning-driven intelligent real-time optimization method for drilling parameters includes the following steps:

[0008] Step 1: Construction of drilling rate prediction model. Collect historical drilling operation data, clean it, and use correlation analysis data analysis to extract parameters with high correlation as input data for prediction model to construct XGBoost model;

[0009] Step 2: Modeling the decision-making process, which involves selecting a set of parameter combinations. To achieve the optimal drilling rate, at every moment requiring a decision. Select only one parameter An MDP is represented as a quintuple. ,in Represents the set of valid actions. Indicates the first Parameters Perform the settings operation. The range of values ​​for is the action space. According to the physical definitions of drilling pressure, drilling speed, and torque, its range of values ​​is non-negative.

[0010] Each drilling parameter is decided sequentially, with the input being the previous decision parameter and drilling data, and the output being a distribution. At each time step After selecting the parameters, input them into the prediction model to generate the predicted drilling speed. It also calculates the reward function and updates the Agent's parameters using the RL algorithm;

[0011] Step 3: Overall framework structure, at each time step Obtain the state from the environment Get actions by entering Agent Logarithmic probability of the current policy Used for subsequent policy updates in the algorithm, and then from The distribution represented Sample a specific parameter from ,Will Input into the prediction model to obtain the predicted drilling rate The environment provides a reward after the drilling speed is obtained. and the next step status And update the end marker. Once the decision-making process for the current time step is complete, the tuple is stored in the experience replay buffer.

[0012] As a further aspect of the present invention, the XGBoost model in step one is evaluated, and if the performance is not good, the parameters are adjusted until the performance is good.

[0013] As a further aspect of the present invention, in step one, the correlation analysis data analysis method uses the Pearson correlation coefficient to perform data correlation analysis. The coefficient ranges from -1 to 1, and a value close to 1 or -1 indicates that there is a strong linear relationship between the two variables.

[0014] As a further aspect of the present invention, the XGBoost model in step one uses root mean square error (RMSE) as an indicator to evaluate the accuracy of the prediction model. The closer the RMSE is to 0, the closer the prediction result is to the actual observed value.

[0015] As a further aspect of the present invention, step two also includes adding a numerical upper limit. , The maximum drilling pressure, rotational speed, and torque that the machine can achieve in actual operation, i.e., the action space is... , The distribution is output from the Agent. Sampling.

[0016] As a further aspect of the present invention, in step two... The set representing the valid states, in state of time This involves splicing drilling data with the distribution of the previous output, i.e. ,in Represents drilling data, These are the distribution parameters from the previous decision.

[0017] As a further aspect of the present invention, in step two... This represents the reward function, which is set as the difference between the predicted drilling rate and the original drilling rate. , To optimize the actual drilling rate before optimization, in the intermediate step of the sequence decision, i.e. When using the expected drilling speed instead .

[0018] As a further aspect of the present invention, in step two... Represents the discount factor. .

[0019] In summary, the embodiments of the present invention have the following beneficial effects compared with the prior art:

[0020] This algorithm uses machine learning to build a drilling speed prediction model and an optimization model, enabling real-time analysis of drilling operation parameters and selection of optimization parameters. The drilling speed is improved by 5% to 10% compared with that before optimization, which is of practical significance for increasing drilling operation efficiency.

[0021] To more clearly illustrate the structural features and effects of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the training process of the prediction model in an embodiment of the invention.

[0023] Figure 2 This is a comparison chart of predicted and actual values ​​in an embodiment of the invention.

[0024] Figure 3 This is a flowchart of the model decision-making process in an embodiment of the invention.

[0025] Figure 4 This is a diagram of the agent network structure in an embodiment of the invention.

[0026] Figure 5 This is a diagram of the overall interaction framework in an embodiment of the invention.

[0027] Figure 6 This is a comparison diagram of drilling speeds before and after the invention embodiment. Detailed Implementation

[0028] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0029] The specific implementation of the present invention will be described in detail below with reference to specific embodiments.

[0030] In one embodiment, a deep reinforcement learning-driven intelligent real-time optimization method for drilling parameters is described in [reference needed]. Figures 1-6 This includes the following steps:

[0031] Step 1: Construction of drilling rate prediction model. Collect historical drilling operation data, clean it, and use correlation analysis data analysis to extract parameters with high correlation as input data for prediction model to construct XGBoost model;

[0032] Step 2: Modeling the decision-making process, which involves selecting a set of parameter combinations. To achieve the optimal drilling rate, at every moment requiring a decision. Select only one parameter An MDP is represented as a quintuple. ,in Represents the set of valid actions. Indicates the first Parameters Perform the settings operation. The range of values ​​for is the action space. According to the physical definitions of drilling pressure, drilling speed, and torque, its range of values ​​is non-negative.

[0033] Each drilling parameter is decided sequentially, with the input being the previous decision parameter and drilling data, and the output being a distribution. At each time step After selecting the parameters, input them into the prediction model to generate the predicted drilling speed. It also calculates the reward function and updates the Agent's parameters using the RL algorithm;

[0034] Step 3: Overall framework structure, at each time step Obtain the state from the environment Get actions by entering Agent Logarithmic probability of the current policy Used for subsequent policy updates in the algorithm, and then from The distribution represented Sample a specific parameter from ,Will Input into the prediction model to obtain the predicted drilling rate The environment provides a reward after the drilling speed is obtained. and the next step status And update the end marker. Once the decision-making process for the current time step is complete, the tuple is stored in the experience replay buffer.

[0035] Further, see Figures 1-6In step one, the XGBoost model is evaluated, and if the performance is not good, the parameters are adjusted until the performance is good.

[0036] Further, see Figures 1-6 In step one, the XGBoost model uses root mean square error (RMSE) as an indicator to evaluate the accuracy of the prediction model. The closer the RMSE is to 0, the closer the prediction result is to the actual observed value.

[0037] Further, see Figures 1-6 Step two also includes adding a numerical upper limit. , The maximum drilling pressure, rotational speed, and torque that the machine can achieve in actual operation, i.e., the action space is... , The distribution is output from the Agent. Sampling.

[0038] Further, see Figures 1-6 In step two The set representing the valid states, in state of time This involves splicing drilling data with the distribution of the previous output, i.e. ,in Represents drilling data, These are the distribution parameters from the previous decision.

[0039] Further, see Figures 1-6 In step two This represents the reward function, which is set as the difference between the predicted drilling rate and the original drilling rate. , To optimize the actual drilling rate before optimization, in the intermediate step of the sequence decision, i.e. When using the expected drilling speed instead .

[0040] In step two Represents the discount factor. .

[0041] In this embodiment, the first step is to construct a drilling speed prediction model.

[0042] The process of building a drilling speed prediction model is as follows: Figure 1 As shown, historical drilling data is first collected and cleaned. Then, data analysis techniques such as correlation analysis are used to extract highly correlated parameters as input data for the prediction model to construct an XGBoost model. The model is then trained using the data. After training, the model's performance is calculated and evaluated. If the performance is not good, the parameters are adjusted until the performance is satisfactory.

[0043] Pearson correlation coefficient was used for data correlation analysis. The Pearson correlation coefficient is a statistic that measures the strength and direction of the linear relationship between two variables, ranging from -1 to 1. Values ​​closer to 1 or -1 indicate a stronger linear relationship between the two variables. Based on the calculation results, 15 parameters with the strongest correlation to drilling speed were used as predictor variables: suspended weight, bit depth, bit vertical depth, well depth (m), vertical depth (m), inlet flow rate log, inlet flow rate, lateness, bit pressure drop, hook speed, wellhead regulating pressure, additional back pressure, drilling pressure, torque, and rotational speed. These were used to train the regression model to predict drilling speed.

[0044] The root mean square error (RMSE) is used as a metric to evaluate the accuracy of the prediction model. RMSE is a commonly used metric for evaluating regression models, measuring the difference between predicted and actual values. Generally, the closer the RMSE is to 0, the closer the prediction is to the actual observed value. The RMSE of this prediction model is 0.88, which indicates that the prediction model is effective. The comparison between the predicted and actual values ​​of the prediction model on the test set is shown below. Figure 2 As shown.

[0045] Part 2: Markov Decision Process Modeling and Agent Design

[0046] The core of reinforcement learning lies in transforming a problem into a Markov decision process. The following section will detail how to transform the drilling parameter optimization problem into a Markov decision process, the setting of the reward function, and the specific design of the agent.

[0047] First, the drilling parameter optimization process is transformed into a sequential decision-making process. The drilling parameter optimization problem in this invention is defined as: based on a set of drilling data at a certain moment, selecting a set of parameter combinations. This allows for achieving the optimal drilling rate. Typically, the search space for selecting a single set of parameters at a time becomes unmanageable, therefore, at every moment a decision needs to be made... Select only one parameter This significantly reduces the search space and transforms it into a sequential decision-making process. The only parameters to be optimized in this invention are drilling pressure, rotational speed, and torque. .

[0048] After transforming it into a sequential decision process, the problem can be further transformed into a Markov Decision Process (MDP), which is usually represented as a quintuple. ,in Represents the set of valid actions. Indicates the first Parameters Perform the settings operation. The range of values ​​for this parameter, i.e., the action space, is defined by the physical definitions of drilling pressure, drilling speed, and torque. Its range is non-negative. This method adds an additional upper limit to its value. , The maximum drilling pressure, rotational speed, and torque that the machine can achieve in actual operation, i.e., the action space is... . The distribution is output from the Agent. The sampling is for more thorough exploration, which will be discussed in detail in the section on building the reinforcement learning framework below. This represents the set of valid states, where decisions are expected to be influenced by both drilling data and the previous decision. state of time This involves splicing drilling data with the distribution of the previous output, i.e. ,in Represents drilling data, These are the distribution parameters from the previous decision. Let represent the reward function. The design of the reward function should encourage the agent to explore in directions that increase drilling speed. Therefore, the reward function is set as the difference between the predicted drilling speed and the original drilling speed, i.e., ... , The actual drilling rate before optimization. In the intermediate step of the sequence decision-making process, i.e. When the parameters are incomplete, the expected drilling rate should be used instead. Here, the expected drilling rate is approximated by the current incomplete parameters. Let be the transition probability function, which is unknown to our problem. Represents the discount factor, here .

[0049] A complete decision-making process for an agent is as follows: Figure 3 As shown. The agent makes decisions on each drilling parameter one by one. Its input is the previous decision parameter and the drilling data, and its output is a distribution. At each time step After selecting the parameters, input them into the prediction model to generate the predicted drilling speed. The reward function is calculated, and the Agent's parameters are updated using the RL algorithm, thereby enabling the Agent to learn how to adjust drilling parameters.

[0050] The internal structure of the agent is as follows: Figure 4 As shown, the Agent is constructed using an Actor-Critic structure, with a network architecture consisting of an input layer, a fully connected layer, and an output layer. The Actor has two output layers, each outputting a distributed... and parameter.

[0051] III: Overall Framework Structure

[0052] The reinforcement learning algorithm framework of this invention is as follows: Figure 5 As shown. At each time step Obtain the state from the environment Get actions by entering Agent Logarithmic probability of the current policy Used for subsequent policy updates in the algorithm, and then from The distribution represented Sample a specific parameter from ,Will Input into the prediction model to obtain the predicted drilling rate The environment provides a reward after the drilling speed is obtained. and the next step status And update the end marker. The decision-making process at the current time step is complete, and the tuple will be... Stored in the experience replay buffer pool.

[0053] By constructing a mechanical drilling rate prediction model as environmental feedback, a reinforcement learning model is guided to learn how to adjust drilling parameters using the mechanical drilling rate as a reward signal, thereby improving the mechanical drilling rate. The specific intelligent algorithm selected for model construction is as follows.

[0054] The prediction model uses the XGBoost model. XGBoost (eXtreme Gradient Boosting) is an efficient, flexible, and powerful machine learning algorithm widely used in data science and machine learning. It is an optimized implementation of the gradient boosting algorithm, particularly adept at handling structured data, and has achieved excellent results in many machine learning competitions (such as Kaggle). Its efficiency and flexibility provide strong support for the construction of the drilling rate prediction model in this method.

[0055] The reinforcement learning model employs Proximal Policy Optimization (PPO), an online policy gradient method. Its core idea is to avoid large fluctuations during training by constraining the difference between the new and old policies. It improves upon traditional policy gradient methods by introducing a pruning mechanism or a KL divergence penalty term to ensure that the magnitude of policy updates remains within a reasonable range.

[0056] The parameters of torque, rotational speed, and drilling pressure were optimized for an oilfield. The experimental results are as follows: Figure 6 As shown, the drilling speed generally increased by 5% to 10%, with an average increase of 6.874%.

[0057] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A deep reinforcement learning-driven intelligent real-time optimization method for drilling parameters, characterized in that, Includes the following steps: Step 1: Construction of drilling rate prediction model. Collect historical drilling operation data, clean it, and use correlation analysis data analysis to extract parameters with high correlation as input data for prediction model to construct XGBoost model; Step 2: Modeling the decision-making process, which involves selecting a set of parameter combinations. To achieve the optimal drilling rate, at every moment requiring a decision. Select only one parameter An MDP is represented as a quintuple. ,in Represents the set of valid actions. Indicates the first Parameters Perform the settings operation. The range of values ​​for is the action space. According to the physical definitions of drilling pressure, drilling speed, and torque, its range of values ​​is non-negative. Each drilling parameter is decided sequentially, with the input being the previous decision parameter and drilling data, and the output being a distribution. At each time step After selecting the parameters, input them into the prediction model to generate the predicted drilling speed. It also calculates the reward function and updates the Agent's parameters using the RL algorithm; Step 3: Overall framework structure, at each time step Obtain the state from the environment Get actions by entering Agent Logarithmic probability of the current policy Used for subsequent policy updates in the algorithm, and then from The distribution represented Sample a specific parameter from ,Will Input into the prediction model to obtain the predicted drilling rate The environment provides a reward after the drilling speed is obtained. and the next step status And update the end marker. Once the decision-making process for the current time step is complete, the tuple is stored in the experience replay buffer pool. In step one, the XGBoost model is evaluated. If the performance is not good, the parameters are adjusted until the performance is good. In step one, the correlation analysis data analysis method uses Pearson correlation coefficient to perform data correlation analysis. The coefficient ranges from -1 to 1. A value close to 1 or -1 indicates that there is a strong linear relationship between the two variables. Step two also includes adding a numerical upper limit. , The maximum drilling pressure, rotational speed, and torque that the machine can achieve in actual operation, i.e., the action space is... , The distribution is output from the Agent. Sampling; In step two The set representing the valid states, in state of time This involves splicing drilling data with the distribution of the previous output, i.e. ,in Represents drilling data, The distribution parameters of the previous decision; In step two This represents the reward function, which is set as the difference between the predicted drilling rate and the original drilling rate. , To optimize the actual drilling rate before optimization, in the intermediate step of the sequence decision, i.e. When using the expected drilling speed instead .

2. The deep reinforcement learning-driven intelligent real-time optimization method for drilling parameters according to claim 1, characterized in that, In step one, the XGBoost model uses root mean square error (RMSE) as an indicator to evaluate the accuracy of the prediction model. The closer the RMSE is to 0, the closer the prediction result is to the actual observed value.

3. The deep reinforcement learning-driven intelligent real-time optimization method for drilling parameters according to claim 1, characterized in that, In step two Represents the discount factor. .

Images