Drilling parameter self-adaptive regulation and control method based on reinforcement learning

Abstract

The invention discloses a drilling parameter self-adaptive regulation and control method based on reinforcement learning, and the core idea of the method is that a drilling parameter optimization regulation and control problem is converted into a reinforcement learning problem which is represented by a Markov decision process. The method comprises the steps that S1, drilling parameters are optimized according to historical drilling data of adjacent wells to serve as the initial state of an intelligent agent; s2, implementing and evaluating a pre-drilling drilling parameter optimization strategy, and controlling a drilling parameter exploration direction; and S3, the action is controlled by utilizing a reward function, a random drilling parameter optimization strategy is generated, and the influence of different drilling parameters on the mechanical drilling speed is actively explored. And S4, the steps S2 and S3 are repeated, and the optimal drilling parameter combination is explored under the drilling parameter regulation and control safety constraint condition. According to the drilling parameter self-adaptive regulation and control method based on reinforcement learning, drilling engineering and an artificial intelligence algorithm are effectively fused, so that the drilling machine has the capability of autonomously regulating and controlling the drilling parameters, and the drilling parameter self-adaptive regulation and control method based on reinforcement learning is of great significance to intelligent drilling.

Description

Technical Field

[0001] The present invention relates to the technical field of oil and gas drilling, and in particular to a drilling parameter adaptive control method based on reinforcement learning. Background Art

[0002] In traditional drilling sites, drillers usually control drilling parameters based on experience or drilling design, but in the context of intelligent drilling, drillers will be replaced, and drilling parameter optimization and control methods driven by drilling data or theory are the future trend. Many research results have been achieved in this field. However, the current drilling parameter optimization research is based on the driller's operation of drilling parameters, and both data acquisition and implementation of optimization results still rely on the driller. In the context of intelligent drilling, how to achieve autonomous control of drilling parameters by the drilling rig still lacks theoretical support.

[0003] In recent years, artificial intelligence algorithms represented by deep learning and reinforcement learning have achieved amazing development. With the help of reinforcement learning's reasoning ability, reinforcement learning has shown great potential in dealing with decision-making problems. Reinforcement learning has achieved great success in the fields of autonomous driving, robot control, and computer Go. Therefore, if reinforcement learning can be applied to drilling technology, it will be a feasible solution to achieve adaptive control of drilling parameters by continuously interacting with the environment through reinforcement learning to obtain the optimal strategy. Summary of the invention

[0004] In order to realize the application of reinforcement learning in intelligent drilling technology, the present invention provides a drilling parameter adaptive control method based on reinforcement learning.

[0005] The core idea of ​​the drilling parameter adaptive control method based on reinforcement learning provided by the present invention is to transform the drilling parameter optimization control problem into a reinforcement learning problem, which is represented by a Markov decision process. The Markov decision process consists of a state space, an action space of a control strategy, a state transition probability, a reward function, and a discount factor, and is represented as:

[0006] M=<S,A,P,R,γ>

[0007] Where, s = {s 1 ,s 2 ,...,s n}∈S represents the set of all states perceived by the agent in interaction with the environment, and all state sets constitute a state space. The state space includes all state sets related to past and recent formation conditions, tool states, drilling parameters, mechanical penetration rate, and stick-slip index.

[0008] The agent is a decision-making intelligent drilling rig. The environment includes the construction site of all relevant processes including the formation, rock breaking tools, and wellbore. The state is composed of mechanical drilling speed, stick-slip index, drilling pressure, and rotation speed, which is expressed as:

[0009] s={ROP,TSE,WOB,RPM}

[0010] Where s is the state of the drilling parameter control process; ROP is the mechanical penetration rate, m / h; TSE is the stick-slip index; WOB is the bit pressure, kN; RPM is the rotation speed, r / min.

[0011] a={a 1 ,a 2 ,...,a n}∈A represents the set state of all actions that the agent can perform. The actions refer to the drilling parameter control strategies including drilling pressure and speed. The set of drilling parameter control strategies constitutes the action space of the control strategy.

[0012] P represents the state transition probability, and the state transition represents the state change caused by the action of the intelligent agent during the drilling process, such as the change of the mechanical drilling speed from "4m / h" to "7m / h".

[0013] R represents a reward function; the reward function is a means of evaluating the drilling parameter control strategy, which is specifically calculated based on the feedback value after the drilling parameter control strategy is executed. The reward function includes positive rewards, negative rewards, and hard boundary rewards. The positive rewards include an increase in mechanical drilling speed and exploring different drilling parameters, the negative rewards include a continuous decrease in mechanical drilling speed and a decrease in mechanical drilling speed, and the hard boundary rewards include drilling parameters exceeding the drilling parameter safe operation window and exceeding the stick-slip strength constraint.

[0014] γ∈[0,1] represents the discount factor.

[0015] Preferably, the action space refers to a set of drilling pressure and speed control strategies, and the drilling pressure and speed have three settings: maintain, increase and decrease. After random combination, there are 9 drilling parameter control strategies, which correspond to 9 dimensions of the action space, respectively, as shown below:

[0016]

[0017] Where, WOB a 、WOB h 、WOB d Respectively represent increasing drilling pressure, maintaining drilling pressure and reducing drilling pressure; RPM a 、RPM h 、RPM d They respectively represent increasing the speed, maintaining the speed, and decreasing the speed.

[0018] Preferably, the reward value of the reward function R is set as follows:

[0019] r 1 =-1000; r 2 =-10; r 3 =10; r 4 =-1000; r 5 =-1; r 6 =1

[0020] In the formula, r 1 is the reward for drilling parameters exceeding the boundary conditions; r 2 Reward for exceeding stick-slip strength; r 3 A reward for increased drilling speed; 4 The reward is for the continuous reduction of mechanical drilling speed, which means that the mechanical drilling speed is reduced at least 5 times in a row; 5 is the reward for reducing the drilling speed, that is, the reward for a single reduction; r 6 Rewards for exploring different drilling parameters.

[0021] The above-mentioned drilling parameter adaptive control method based on reinforcement learning has the following steps:

[0022] S1. Optimize drilling parameters based on historical drilling data of adjacent wells as the initial state of the intelligent agent.

[0023] Specifically, a mechanical drilling speed prediction model is established based on the historical drilling data of adjacent wells, and the particle swarm algorithm is used to solve the optimal drilling parameters and the corresponding mechanical drilling speed and stick-slip index as the initial state of the intelligent agent under the safety constraints of drilling parameter regulation; the drilling parameters are drilling pressure and rotation speed.

[0024] S2. Implement and evaluate the pre-drilling drilling parameter optimization strategy to control the direction of drilling parameter exploration.

[0025] Specifically, during the drilling process, the pre-drilling drilling parameter optimization strategy is implemented. According to the new state fed back after the implementation, the reward function is used to evaluate the action, control the direction of drilling parameter exploration, and determine the new drilling parameter optimization strategy.

[0026] S3. Use the reward function to control the action, generate a random drilling parameter optimization strategy, and actively explore the impact of different drilling parameters on the mechanical drilling speed.

[0027] S4. Repeat steps S2 and S3 to find the optimal drilling parameter combination under the safety constraints of drilling parameter regulation.

[0028] In the method, the drilling parameters regulate safety constraints, including but not limited to drilling pressure constraints, rotation speed constraints, mechanical drilling speed constraints and stick-slip vibration constraints.

[0029] Compared with the prior art, the present invention is beneficial in that:

[0030] (1) The reinforcement learning-based drilling parameter adaptive control method provided by the present invention effectively integrates drilling engineering with artificial intelligence algorithms, enabling the drilling rig to have the ability to autonomously control drilling parameters, which is of great significance for realizing intelligent drilling.

[0031] (2) The present invention can be directly applied on the basis of existing conventional drilling equipment without the need for additional downhole measurement tools.

[0032] Other advantages, objectives and features of the present invention will be embodied in part through the following description, and in part will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The present invention is a flow chart of the method for adaptively controlling drilling parameters based on reinforcement learning.

[0034] Figure 2 The drilling parameter control results of the intelligent agent autonomously controlling the drilling parameters.

[0035] Figure 3 The mechanical drilling speed optimization results of the intelligent agent autonomously controlling the drilling parameters. DETAILED DESCRIPTION

[0036] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0037] like Figure 1 As shown, the drilling parameter adaptive control method based on reinforcement learning provided by the present invention comprises the following steps:

[0038] S1. A mechanical drilling speed prediction model is established based on the historical drilling data of adjacent wells. The particle swarm algorithm is used to solve the optimal drilling parameters and the corresponding mechanical drilling speed and stick-slip index under the safety constraints of drilling parameter regulation to determine the initial state s of the intelligent agent. t = {ROP t ,TSE t ,WOB t ,RPM t};

[0039] S2. During the drilling process, the initial state of the intelligent agent obtained by the pre-drilling drilling parameter optimization result is implemented to generate a random drilling parameter optimization strategy, action a t ={WOB a ,RPM h} and implement it, according to the new state s fed back after implementationt+1 = {ROP t+1 ,TSE t+1 ,WOB t+1 ,RPM t+1}, use the reward function to evaluate the action, calculate the reward function value, determine the new drilling parameter optimization strategy, and control the direction of drilling parameter exploration;

[0040] The reward value is determined as follows:

[0041] r 1 =-1000; r 2 =-10; r 3 =10; r 4 =-1000; r 5 =-1; r 6 =1

[0042] Among them, r 1 is the reward for drilling parameters exceeding the boundary conditions; r 2 Reward for exceeding stick-slip strength; r 3 A reward for increased drilling speed; 4 A reward for continuous reduction in drilling speed; 5 A bonus for reduced drilling speed; 6 Rewards for exploring different drilling parameters.

[0043] S3. During the drilling process, the reward function is used to control the action, generate a random drilling parameter optimization strategy, and actively explore the impact of different drilling parameters on the mechanical drilling speed. Specifically, the Q-learning algorithm is used to find the optimal action strategy in the environment, update the Q value according to the Bellman equation, and update the Q value of the action selected in the previous step through the current reward and the maximum Q value of the future state. The Q value calculation method is as follows:

[0044] Q(s,a)←Q(s,a)+α×(r+γ×max(Q(s′,a′))-Q(s,a))

[0045] Where max(Q(s′,a′)) is the Q value of the optimal action in the new state s′; α is the learning rate, which controls the update speed of the Q value.

[0046] S4. Repeat steps S2 and S3 to find the optimal drilling parameter combination under the safety constraints of drilling parameter regulation.

[0047] In order to test the drilling parameter adaptive control method of the present invention, a drilling simulation platform is designed in this embodiment to simulate the drilling parameter adaptive control process. The drilling simulation platform contains two different types of formations to be drilled. It is assumed that formation A and formation B are homogeneous formations. The optimal drilling parameters of formation A are drilling pressure of 130 kN, rotation speed of 100 r / min, and maximum mechanical drilling speed of 13.5 m / h. The optimal drilling parameters of formation B are drilling pressure of 185 kN, rotation speed of 100 r / min, and maximum mechanical drilling speed of 37 m / h.

[0048] Before drilling, the agent does not know the distribution of formation types, and formation changes can only be revealed through drilling itself. The goal of the agent is to explore different combinations of drilling parameters as much as possible, find the optimal drilling parameters, and achieve efficient drilling in two formations. For the agent, the optimal drilling parameters are also unknown before drilling. The agent needs to choose different actions to change the drilling parameters during the drilling process and approach the optimal drilling parameters. The simulation parameters are shown in Table 1.

[0049] Table 1 Simulation parameters

[0050] Simulation parameters Learning Rate Discount Factor Initial drilling parameters Drilling pressure range Speed ​​range Stick-slip strength A Stratum 0.01 0.99 [50kN,50r / min] [50kN,150kN] [50r / min,120r / min] [0,1.2] B Stratum 0.01 0.99 [50kN,50r / min] [50kN,200kN] [50r / min,120r / min] [0,1.2]

[0051] The simulation results are as follows:

[0052] The results of the intelligent agent's autonomous control of drilling parameters are shown in the figure below. Figure 2 As shown in the figure, it can be seen that the agent first chooses to increase the drilling pressure and speed. After 13 iterations, the agent finds that continuing to increase the drilling pressure and speed can not get higher rewards. The drilling rig chooses to explore different drilling parameters, and then the drilling pressure fluctuates in the range of 95-115kN, and the speed fluctuates in the range of 105-115r / min. After the formation changes, the agent can quickly respond to the changes in the formation, control the drilling parameters to gradually approach the better drilling parameters, and stabilize after 53 iterations. The drilling rig begins to explore different drilling parameters, and then the drilling pressure fluctuates in the range of 145-160kN, and the speed fluctuates in the range of 110-120r / min.

[0053] The optimization results of mechanical drilling speed by the intelligent agent autonomously controlling drilling parameters are shown in the figure below. Figure 3 As shown in the figure, it can be seen that the average mechanical drilling speed of the A formation intelligent agent autonomously adjusting the drilling parameters is 12.7m / h, and the mechanical drilling speed error relative to the pre-drilling simulation is 5.9%. The average mechanical drilling speed of the B formation intelligent agent autonomously adjusting the drilling parameters is 38.03m / h, which is slightly higher than the mechanical drilling speed of 37m / h in the pre-drilling static simulation, and the mechanical drilling speed error relative to the pre-drilling static simulation is 0.02%.

[0054] Therefore, the simulation results show that the reinforcement learning-based drilling parameter adaptive control method of the present invention can realize the autonomous control of drilling parameters by the drilling rig and quickly adapt to changes in formation conditions.

[0055] The above description is only a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of the technical solution of the present invention.

Claims

1. A drilling parameter adaptive control method based on reinforcement learning, characterized in that: The Markov decision process is used to represent the drilling parameter optimization and control process; the Markov decision process consists of the state space, the action space of the control strategy, the state transition probability, the reward function and the discount factor, which can be expressed as: M=<S,A,P,R,γ> Where, s={s1,s2,...,s n }∈S represents the set of all states perceived by the agent through interaction with the environment, and all state sets constitute the state space; the agent is a decision-making intelligent drilling rig; the environment includes the construction site of all relevant processes including the formation, rock breaking tools, and wellbore; the state is composed of mechanical drilling speed, stick-slip index, drilling pressure, and rotation speed, which can be expressed as: s={ROP,TSE,WOB,RPM} Where, s is the state of the drilling parameter control process; ROP is the mechanical drilling speed, m / h; TSE is the stick-slip index; WOB is the bit pressure, kN; RPM is the rotation speed, r / min; a={a1,a2,...,a n }∈A represents the set state of all actions that the agent can perform; the actions refer to the drilling parameter control strategies including drilling pressure and speed, and the set of drilling parameter control strategies constitutes the action space; P represents the state transition probability, and the state transition represents the state change caused by the action of the intelligent agent during the drilling process; R represents a reward function; the reward function is a means of evaluating the drilling parameter control strategy, and is specifically calculated based on the feedback value after the drilling parameter control strategy is executed; γ∈[0,1] represents the discount factor.

2. The drilling parameter adaptive control method based on reinforcement learning according to claim 1, characterized in that: The action space refers to the set of drilling pressure and speed control strategies. The drilling pressure and speed have three settings: maintain, increase and decrease. After random combination, there are 9 drilling parameter control strategies, corresponding to the 9 dimensions of the action space, as shown below: Where, WOB a 、WOB h 、WOB d Respectively represent increasing drilling pressure, maintaining drilling pressure and reducing drilling pressure; RPM a 、RPM h 、RPM d They respectively represent increasing the speed, maintaining the speed, and decreasing the speed.

3. The drilling parameter adaptive control method based on reinforcement learning according to claim 1, characterized in that: The reward function includes positive rewards, negative rewards and hard boundary rewards. The positive rewards include increased mechanical drilling speed and exploration of different drilling parameters, the negative rewards include continuous reduction of mechanical drilling speed and reduced mechanical drilling speed, and the hard boundary rewards include drilling parameters exceeding the drilling parameter safe operating window and exceeding the stick-slip strength constraint.

4. The drilling parameter adaptive control method based on reinforcement learning according to claim 3 is characterized in that: The reward value of the reward function R is set as follows: r1＝-1000；r2＝-10；r3＝10；r4＝-1000；r5＝-1；r6＝1 Where r1 is the reward for drilling parameters exceeding the boundary conditions; r2 is the reward for exceeding the stick-slip strength; r3 is the reward for increasing the mechanical drilling rate; r4 is the reward for continuously reducing the mechanical drilling rate; r5 is the reward for reducing the mechanical drilling rate; and r6 is the reward for exploring different drilling parameters.

5. The drilling parameter adaptive control method based on reinforcement learning according to any one of claims 1 to 4, characterized in that: The following steps are involved: S1, optimizing drilling parameters based on historical drilling data of adjacent wells as the initial state of the intelligent agent; S2. Implement and evaluate the pre-drilling drilling parameter optimization strategy to control the direction of drilling parameter exploration; S3, using reward function to control actions, generate random drilling parameter optimization strategies, and actively explore the impact of different drilling parameters on mechanical drilling speed; S4. Repeat steps S2 and S3 to find the optimal drilling parameter combination under the safety constraints of drilling parameter regulation.

6. The drilling parameter adaptive control method based on reinforcement learning according to claim 5, characterized in that: Specifically, step S1 is to establish a mechanical drilling speed prediction model based on the historical drilling data of adjacent wells, and use the particle swarm algorithm to solve the optimal drilling parameters and the corresponding mechanical drilling speed and stick-slip index as the initial state of the intelligent agent under the safety constraints of drilling parameter regulation; the drilling parameters are drilling pressure and rotation speed.

7. The drilling parameter adaptive control method based on reinforcement learning according to claim 5, characterized in that: Specifically, step S2 is to implement the pre-drilling drilling parameter optimization strategy during the drilling process, evaluate the action using the reward function according to the new state fed back after the implementation, control the direction of drilling parameter exploration, and determine a new drilling parameter optimization strategy.

8. The drilling parameter adaptive control method based on reinforcement learning according to claim 6, characterized in that: The drilling parameter regulation safety constraint conditions include, but are not limited to, drilling pressure constraint, rotation speed constraint, mechanical drilling speed constraint and stick-slip vibration constraint.

Images