A reservoir-constrained injection-production optimization method based on safety reinforcement learning

Abstract

The present invention discloses a reservoir-constrained injection-production optimization method based on safety reinforcement learning, which belongs to the technical field of reservoir-constrained injection-production optimization, and includes the following steps: constructing a reservoir injection-production ratio-constrained injection-production optimization model and a reservoir injection-production ratio-constrained Markov decision process; constructing a reservoir injection-production ratio-constrained injection-production regulation reinforcement learning agent model based on the WCSAC safety reinforcement learning algorithm; the agent continuously interacts with the reservoir numerical simulation, collects data samples and stores them in the experience playback buffer; trains the reward critic network, the safety critic network, and the strategy network until the maximum number of training rounds is reached; and uses the trained strategy network to output the optimal production system in the current state. The reinforcement learning method of the present invention can design the optimal injection-production plan according to the current state of the reservoir under the premise of satisfying the reservoir injection-production ratio constraint, and realize safe and stable production of the reservoir, and has good scientific and practical application value.

Description

Technical Field

[0001] The invention relates to the technical field of reservoir constrained injection and production, and in particular to a reservoir constrained injection and production optimization method based on safety reinforcement learning. Background Art

[0002] In the actual process of reservoir injection and production optimization, it is not enough to simply pursue high economic benefits. In order to better apply the construction of smart oilfields to field production, a variety of engineering constraints need to be considered. The injection-production ratio is one of the key indicators that affect the development effect and economic benefits of reservoirs. A reasonable injection-production ratio directly affects development indicators such as the formation pressure maintenance level and the water content increase rate, and can enable the oilfield to maintain high production capacity and ultimately achieve a higher recovery rate. However, the geological conditions faced in the process of reservoir development are complex and changeable, and there are many uncertain factors, such as formation heterogeneity, oil-water interface movement, and changes in injection and production processes, which make it difficult for traditional optimization methods to achieve the best results in actual operations.

[0003] Traditional methods for solving constrained injection and production optimization problems mainly rely on numerical simulation and optimization based on machine learning. Based on the numerical simulation method, a mathematical model of the reservoir and a numerical simulation model are established, and a large number of numerical simulation experiments are conducted to evaluate the effects of different injection and production schemes. This method has high accuracy, but the computational cost is huge and it is difficult to apply in real time. Traditional machine learning methods, such as genetic algorithms and particle swarm optimization algorithms, find the optimal solution by simulating the natural evolution process. These algorithms can improve the optimization efficiency to a certain extent, but when dealing with complex constrained problems, they often have problems such as slow convergence and easy to fall into local optimality.

[0004] In contrast, algorithms based on reinforcement learning have shown significant advantages in solving complex engineering problems, especially in the field of reservoir injection and production optimization. However, the current algorithms based on reinforcement learning do not take into account the actual engineering safety constraints, which may shorten the production life of the oil field in actual production applications, thus affecting economic benefits.

[0005] Existing injection-production optimization methods have difficulties in controlling the injection-production ratio and are unable to handle complex engineering constraints, resulting in low optimization efficiency and easy to fall into local optimal solutions. In addition, existing methods are difficult to adjust production strategies in real time to adapt to dynamically changing reservoir conditions, thus affecting the long-term production efficiency and economic benefits of the oil field. Therefore, there is an urgent need for a method that can effectively handle reservoir injection-production ratio constraint optimization and can adjust and optimize production strategies in real time to improve the mining efficiency, economic benefits and production life of the oil field. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a reservoir-constrained injection-production optimization method based on safety reinforcement learning. This method uses the WCSAC safety reinforcement learning algorithm to find the optimal production strategy under the premise of satisfying the injection-production ratio constraint, realize the efficient exploitation of the oil field, and ensure the feasibility and safety in actual production. This method can not only improve economic benefits, but also extend the production life of the oil field, providing important technical support for the development of my country's oil resources.

[0007] The technical solution of the present invention is as follows:

[0008] A reservoir constrained injection and production optimization method based on safety reinforcement learning specifically includes the following steps:

[0009] Step 1: construct an injection-production optimization model with reservoir injection-production ratio constraints and a Markov decision process with reservoir injection-production ratio constraints, and determine optimization variables, constraint variables and objective functions;

[0010] Step 2: Based on the WCSAC safety reinforcement learning algorithm, a reinforcement learning agent model for injection-production regulation with reservoir injection-production ratio constraints is constructed;

[0011] Step 3: Based on the reservoir numerical simulator, the intelligent agent and the reservoir numerical simulation are continuously interacted, and data samples generated during the interaction are collected and stored in the experience playback buffer;

[0012] Step 4: Combine the experience replay buffer data to guide the training and update of the reward critic network and the safety critic network, and use the conditional risk value assessment mechanism to guide the training and update of the policy network;

[0013] Step 5: Repeat steps 3 and 4 until the maximum number of training rounds is reached to obtain a trained policy network.

[0014] Step 6: Input the reservoir pressure field and water saturation field states into the trained strategy network, output the optimal production system under the current state, guide the reservoir injection and production operations according to the production system, and realize injection and production optimization constrained by the injection-production ratio.

[0015] Furthermore, the specific process of step 1 is as follows:

[0016] Step 1.1: Construct an injection-production optimization model with reservoir injection-production ratio constraints:

[0017] (1);

[0018] in, For Find the minimum value of the objective function for the independent variable; is the objective function; represents the production regime, the production volume of production wells and the injection volume of injection wells at each time step; is the economic net present value; and Respectively The lower and upper bounds of the boundary constraints; represents the injection-production ratio constraint; the calculation formula of economic net present value is as follows:

[0019] (2);

[0020] in, is the maximum time step period per round; For block The oil production rate in each time step; represents the price of crude oil; For block Water production rate in time steps; It represents the cost of produced water treatment; For the injection well The water injection rate in each time step; represents the water injection cost; is the annual discount rate; For the The length of the time step in days; Indicates The number of days that a time step has passed;

[0021] Step 1.2: The reservoir injection-production ratio constrained Markov decision process uses a seven-tuple Description, where is the state space, which includes the state variables of reservoir pressure field and water saturation field; is the action space, which includes the production rate of the production well and the injection rate of the water injection well; is the state transition probability; is the reward function; is the cost function; is the injection-production ratio constraint threshold; is the discount factor;

[0022] Step 1.3, specify that the optimization variables are the production rate of the production well and the injection rate of the injection well; the constraint variables are the upper boundary, lower boundary and injection-production ratio constraint of the single well optimization variables; the objective function is the economic net present value, and the goal is to maximize the economic net present value under the premise of satisfying the injection-production ratio constraint.

[0023] Furthermore, the specific process of step 2 is as follows:

[0024] Step 2.1: Obtain the reservoir model and The state of the time step Set to the pressure value and water saturation value for each grid point in the reservoir model as follows:

[0025] (3);

[0026] in, , Respectively Pressure value and water saturation value of each grid point;

[0027] The pressure value is normalized using the min-max normalization function, and the calculation formula is as follows:

[0028] (4);

[0029] in, is the normalization function; For the The pressure value of each grid point; , They are the minimum pressure value and the maximum pressure value of the current pressure state respectively;

[0030] Step 2.2: Actions in time steps Set the injection and production plan for each well at the current time step, including the liquid production rate of all production wells and the water injection rate of all water injection wells, as shown below:

[0031] (5);

[0032] in, For the The liquid production rate of the production well; For the The production rate of the injection well; For the The liquid production rate of the production well; and They represent the minimum and maximum liquid production rates of the production wells, respectively; For the The injection rate of the injection wells; and They represent the minimum and maximum injection rates of the injection wells respectively;

[0033] Step 2.3: The cost function for each time step Set to the extent to which the injection-production ratio constraint is violated at the current time step, as follows:

[0034] (6);

[0035] (7);

[0036] in, Indicates The injection-sampling ratio for each time step; , and Respectively represent The oil production rate of the production wells The water production rate of the production well and the The injection rate of the injection wells; and They represent the minimum and maximum values ​​that do not violate the injection-production ratio constraint;

[0037] Step 2.4: The reward function for time steps Defined as:

[0038] (8);

[0039] in, is the penalty coefficient for violating the injection-production ratio constraint;

[0040] Step 2.5: Determine the initial parameters of the learning environment of the reservoir injection-production ratio constraint injection-production regulation reinforcement learning agent model, including: discount factor , Temperature parameters , policy network learning rate , reward critic network learning rate , Security Critic Network Learning Rate , Injection-production ratio constraint threshold , Risk Level , damping coefficient , Maximum time step period per round .

[0041] Furthermore, the specific process of step 3 is as follows:

[0042] Step 3.1, in In the time step, the agent Output Action After the relevant linear transformation, the actual injection and production control system of the reservoir is obtained. The reservoir numerical simulator eclipse is executed according to the control system to obtain the first Time step state , and calculate the The reward function for time steps , cost function ; The formula for linear transformation is as follows:

[0043] (9);

[0044] in, represents the actual injection-production control system after linear transformation; Indicates action An element in ; , Respectively represent the upper and lower limits of production wells or water injection wells;

[0045] Step 3.2: Add the data samples generated during the interaction to the experience playback buffer The collected data samples are ;

[0046] Step 3.3: In the time step, loop through steps 3.1 to 3.2 and record the cumulative reward of the round .

[0047] Furthermore, the specific process of step 4 is as follows:

[0048] Step 4.1: Predefine a fixed value ,Will Experience Replay Buffer When the number of samples in The number of samples in is greater than When, from Random sampling The reward critic network, safety critic network, and strategy network are updated and trained based on sample data.

[0049] Step 4.2: Use the sampled data to calculate the target reward Q value and update the network parameters of the reward critic by minimizing the loss function; the formula is as follows:

[0050] (10);

[0051] (11);

[0052] in, , Respectively represent The target reward Q value and reward Q value for each time step; Indicates The reward for each time step value; Indicates the expectation of the target value; Indicates A policy network with time steps; represents the reward critic network loss;

[0053] Step 4.3: Use the sampled data to calculate the target safety Q value and target safety variance value, and update the parameters of the safety critic network by minimizing the loss function. The formula is as follows:

[0054] (12);

[0055] (13);

[0056] (14);

[0057] in, , , and Respectively represent The target safety Q value, safety Q value, target safety variance value and safety variance value for each time step; and Respectively represent The safe Q value and safe variance value of each time step; Indicates security critic network loss;

[0058] Step 4.4: Based on the updated reward critic network and safety critic network, combined with the conditional risk value assessment mechanism, guide the policy network update to ensure that the strategy maximizes the return while satisfying the constraints; the formula is as follows:

[0059] (15);

[0060] (16);

[0061] (17);

[0062] in, is an exponential function with base e; Represents the inverse cumulative distribution function of the standard normal distribution; Indicates A policy network with time steps; Indicates the expectation of the target value; represents the weight coefficient for adjusting the safety constraint; represents the conditional value at risk; represents the damping term value; represents the policy network loss.

[0063] The present invention has the following beneficial effects: the present invention combines the WCSAC safety reinforcement learning algorithm, introduces the conditional risk value assessment mechanism and the safety critic network, and effectively handles the reservoir injection-production ratio constraint problem; compared with the traditional method, the present invention significantly reduces the number of iterations of numerical simulation and improves the optimization efficiency. Its main use is to improve the economic benefits and production life of the oil field by accurately controlling the injection-production ratio during the reservoir production process. The method has broad application prospects and promotion value in the construction of intelligent oil fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is the overall design flow chart of the reservoir constrained injection and production optimization method based on safety reinforcement learning of the present invention.

[0065] Figure 2 It is a well location distribution diagram of the three-channel two-dimensional reservoir model in the embodiment of the present invention.

[0066] Figure 3 It is a convergence curve diagram of the economic net present value in the embodiment of the present invention.

[0067] Figure 4 It is a curve chart of the number of times the injection-production ratio constraint is violated every 100 rounds in an embodiment of the present invention.

[0068] Figure 5 It is a comparison diagram of the curves before and after the optimization of the injection-production ratio within a control cycle in an embodiment of the present invention.

[0069] Figure 6 It is a comparison diagram of the curves before and after the optimization of the cumulative oil production in a regulation cycle in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0071] like Figure 1 The reservoir constraint injection and production optimization method based on safety reinforcement learning shown in the figure specifically includes the following steps:

[0072] Step 1: Construct an injection-production optimization model with reservoir injection-production ratio constraints and a Markov decision process with reservoir injection-production ratio constraints, and determine the optimization variables, constraint variables, and objective function. The specific process is as follows:

[0073] Step 1.1, constrained injection-production optimization is to use the injection-production relationship as the independent variable, the economic net present value as the objective function, and establish a mathematical optimization model based on various constraints that need to be met in actual production, and finally combine the optimization method to optimize the solution.

[0074] The reservoir injection-production ratio-constrained injection-production optimization model can be defined as follows:

[0075] (1);

[0076] in, For Find the minimum value of the objective function for the independent variable; is the objective function; represents the production regime, the production volume of production wells and the injection volume of injection wells at each time step; and Respectively The lower and upper bounds of the boundary constraints; represents the injection-production ratio constraint; is the economic net present value. The calculation formula of economic net present value is as follows:

[0077] (2);

[0078] in, is the maximum time step period per round; For block The oil production rate in each time step; represents the price of crude oil; For block Water production rate in time steps; It represents the cost of produced water treatment; For the injection well The water injection rate in each time step; represents the water injection cost; is the annual discount rate; For the The length of the time step in days; Indicates The number of days that a time step has passed.

[0079] Step 1.2: As an effective method for solving sequential decision problems, reinforcement learning is based on Markov decision. Constrained Markov decision process, as a variant of Markov decision process, is also an effective method for solving sequential decision problems, which limits the decision process of the intelligent agent by introducing additional constraints.

[0080] According to the reservoir injection-production ratio constraint optimization model, a reservoir injection-production ratio constraint Markov decision process is constructed, which can be expressed as a seven-tuple Description, where each element is defined as follows:

[0081] is the state space, which includes the state variables of reservoir pressure field and water saturation field;

[0082] is the action space, which includes the production rate of the production well and the injection rate of the water injection well;

[0083] is the state transition probability; definition Indicates taking action After that, from the state Transition to a new state The probability of state transfer is simulated by the reservoir numerical simulator eclipse;

[0084] is the reward function; define Indicates in status Take action The resulting reward function is an instantaneous reward, which is represented by the economic net present value of the current time step and the cost function;

[0085] is the cost function; define Indicates in status Take action The resulting cost function is an instantaneous cost, which is represented by the degree of violation of the injection-production ratio constraint at the current control time step;

[0086] is the injection-production ratio constraint threshold; the injection-production ratio constraint threshold represents the maximum cumulative constraint cost allowed in one round, that is, ; For the The injection-production ratio constraint threshold is used to control the safety during the constraint strategy optimization process.

[0087] is the discount factor; it is a discount factor for future discounted rewards.

[0088] Step 1.3: Based on the above reservoir injection-production ratio constraint injection-production optimization model and reservoir injection-production ratio constraint Markov decision process, the optimization variables are clearly defined as the production rate of production wells and the injection rate of water injection wells; the constraint variables are the upper and lower boundaries of the single well optimization variables and the injection-production ratio constraint; the objective function is the economic net present value The goal is to maximize the .

[0089] Step 2: Based on the WCSAC safety reinforcement learning algorithm (Worst-case Soft Actor Critic for Safety-constrained Reinforcement Learning), a reservoir injection-production ratio constraint injection-production control reinforcement learning agent model is constructed; the core idea of ​​the WCSAC safety reinforcement learning algorithm is to combine the soft actor-critic (SAC) framework with the strategy optimization considering the worst-case performance in reinforcement learning to ensure that the agent maximizes the cumulative reward while satisfying safety constraints. The specific process of step 2 is:

[0090] Step 2.1: Obtain the reservoir model and The state of the time step Set to the pressure value and water saturation value of each grid point in the reservoir model, as shown in formula (3):

[0091] (3);

[0092] in, , Respectively The pressure value and water saturation value of each grid point. The water saturation field is generally between [0,1], and the pressure field data is usually between thousands of orders of magnitude. Directly using these data for reinforcement learning will lead to numerical instability, slow training process, and feature imbalance. By normalizing the pressure field data to the range of [0,1], the stability of numerical calculations can be improved, the convergence speed of the model can be accelerated, and the generalization ability of the model can be enhanced. The normalization process uses the min-max normalization function, and the calculation formula is as follows:

[0093] (4);

[0094] in, is the normalization function; For the The pressure value of each grid point; , They are respectively the minimum pressure value and the maximum pressure value of the current pressure state.

[0095] Step 2.2: time step actions Set as the injection-production scheme of each well at the current time step, including the liquid production rate of all production wells and the water injection rate of all water injection wells, as shown in formula (5):

[0096] (5);

[0097] in, , represents the number of producing wells; For the The liquid production rate of the production well; For the The production rate of the injection well; For the The liquid production rate of the production well; and They represent the minimum and maximum liquid production rates of the production wells, respectively; , represents the number of injection wells; For the The injection rate of the injection wells; and They represent the minimum and maximum injection rates of the injection wells respectively.

[0098] Step 2.3: The cost function for each time step Set to the degree of violation of the injection-production ratio constraint at the current time step. The cost function calculation formula is as follows:

[0099] (6);

[0100] (7);

[0101] in, Indicates The injection-sampling ratio for each time step; , and Respectively represent The oil production rate of the production wells The water production rate of the production well and the The injection rate of the injection wells is in m 3 / d; and They represent the minimum and maximum values ​​that do not violate the injection-production ratio constraint.

[0102] Step 2.4: The reward function for time steps It is defined as shown in formula (8):

[0103] (8);

[0104] in, , and They are crude oil price, water treatment cost and water injection cost, all in RMB / m 3 ; , and Respectively represent The oil production rate of the production wells The water production rate of the production well and the The injection rate of the injection wells is in m 3 / d; is the penalty coefficient for violating the injection-production ratio constraint.

[0105] Step 2.5: Determine the initial parameters of the learning environment of the reservoir injection-production ratio constraint injection-production regulation reinforcement learning agent model, including: discount factor , Temperature parameters , policy network learning rate , reward critic network learning rate , Security Critic Network Learning Rate , Injection-production ratio constraint threshold , Risk Level , damping coefficient , Maximum time step period per round .

[0106] Step 3: Based on the reservoir numerical simulator, the agent continuously interacts with the reservoir numerical simulation, collects data samples generated during the interaction and stores them in the experience playback buffer. The specific process is as follows:

[0107] Step 3.1, in At a time step, the agent Output Action ,at this time is a set with a value range of [-1,1]. After the relevant linear transformation, the actual injection and production control system of the reservoir is obtained. The reservoir numerical simulator eclipse is executed according to the control system to obtain the first Time step state , and calculate the The reward function for time steps , cost function .action The linear transformation formula is as follows:

[0108] (9);

[0109] in, Indicates action An element in ; , Respectively represent the upper and lower limits of production wells or water injection wells; Represents the actual injection-production control system after linear transformation.

[0110] Step 3.2: Add the data samples generated during the interaction to the experience playback buffer The collected data samples are .

[0111] Step 3.3: In the time step, loop through steps 3.1 to 3.2 and record the cumulative reward of the round .

[0112] Step 4: Combine the experience replay buffer data to guide the training and update of the reward critic network and the safety critic network, and use the conditional risk value assessment mechanism to guide the training and update of the policy network; the specific process is as follows:

[0113] Step 4.1: Predefine a fixed value ,Will Experience Replay Buffer When the number of samples in The number of samples in is greater than When, from Random sampling The reward critic network, safety critic network, and policy network are updated and trained based on sample data.

[0114] Step 4.2: Use the sampled data to calculate the target reward Q value, and update the reward critic network parameters by minimizing the loss function. Formula (10) is the target reward Q value calculation, and formula (11) is the reward critic network loss calculation:

[0115] (10);

[0116] (11);

[0117] in, , Respectively represent The target reward Q value and reward Q value for each time step; Indicates The reward for each time step value; represents the discount factor; Indicates the expectation of the target value; represents the temperature parameter; Indicates A policy network with time steps; represents the reward-critic network loss.

[0118] Step 4.3: Use the sampled data to calculate the target security Q value and target security variance value, and update the security critic network parameters by minimizing the loss function. Formula (12) is the target security Q value calculation, formula (13) is the target security variance value calculation, and formula (14) is the security critic network loss calculation:

[0119] (12);

[0120] (13);

[0121] (14);

[0122] in, , , and Respectively represent The target safety Q value, safety Q value, target safety variance value and safety variance value for each time step; and Respectively represent The safe Q value and safe variance value of each time step; Indicates The cost function for each time step; represents the discount factor; Represents security critic network loss.

[0123] Step 4.4: Based on the updated reward critic network and security critic network, combined with the conditional risk value CVaR evaluation mechanism, guide the policy network update to ensure that the strategy maximizes the return while satisfying the constraints. Formula (15) is the conditional risk value CVaR calculation, formula (16) is the damping term calculation, and the policy network uses formula (17) to minimize the loss update:

[0124] (15);

[0125] (16);

[0126] (17);

[0127] in, is an exponential function with base e; Represents the inverse cumulative distribution function of the standard normal distribution; Indicates A policy network with time steps; Indicates the expectation of the target value; represents the weight coefficient for adjusting the safety constraint; represents the conditional value at risk; It represents the value of the damping term, which ensures that the constraints can be stably met during the optimization process by controlling the change range of the strategy; represents the policy network loss.

[0128] Step 5: Repeat steps 3 and 4 until the maximum number of training rounds is reached, and a trained policy network is obtained; the specific process is: repeat steps 3 and 4, and continue to iterate until the preset maximum number of training rounds is reached. In each round of iteration, the policy network is continuously optimized and updated to gradually improve its performance. Through multiple rounds of repeated training and adjustment, the optimal policy network that has been fully trained and optimized is finally obtained.

[0129] Step 6: Input the reservoir pressure field and water saturation field states into the trained strategy network. The strategy network can output the optimal production system under the current state, effectively guide the reservoir injection and production operations in practical applications, and achieve efficient injection-production ratio-constrained injection-production optimization.

[0130] In order to demonstrate the feasibility and superiority of the present invention, the following specific examples are given.

[0131] like Figure 2 As shown, a reservoir model is used as an example to test the method proposed in the present invention. The model is a two-dimensional three-channel single-layer reservoir model, which consists of 25*25*1 grids, with a viscosity of 2.2cP, a porosity of 0.2, an initial pressure of 3800psi, and an initial water saturation of 0.3. The model has 9 production wells (numbered P1 to P9) and 4 water injection wells (numbered I1 to I4) and three high permeability channels. All production wells are controlled by the liquid production rate, and the water injection wells are controlled by the water injection rate. The unit is m 3 / d, the optimized production time of the reservoir is 1800 days, and the regulation is performed once every 180 days, with a total of 10 regulation time steps. The production rate or water injection rate of each production well is regulated in each time step. That is, in the embodiment of the present invention, the number of production wells Set to 9, the minimum liquid production rate of the production well and maximum value 0 and 300 respectively; the number of injection wells is 4, the minimum injection rate of the injection well and maximum value 0 and 700 respectively. The minimum value of the injection-production ratio constraint is not violated and maximum value are set to 0.8 and 1.2 respectively; The length of the time step in days 180 days. Crude oil price , water treatment cost and water injection costs are set to 80, 5, and 0 respectively; the penalty coefficient for violating the injection-production ratio constraint is set to 10. Discount factor Set to 0.99; the initial temperature parameter is set to 0.2; the policy network learning rate , reward critic network learning rate and the security critic network learning rate All are set to 0.0003; the injection-production ratio constraint threshold Set to 0.2; risk level Set to 0.1; Damping coefficient Set to 10. For production wells, , are 300 and 0 respectively; for water injection wells, , In this embodiment, eclipse reservoir numerical simulation software is used for numerical simulation, and a total of 4000 training iterations are performed, with a maximum time step period of each round of Set to 10. During training, Set to 256.

[0132] Figure 3 is a convergence curve diagram of the economic net present value of the embodiment of the present invention as the number of reservoir numerical simulations changes. Figure 4 This is a graph of the number of constraint violations per hundred rounds in an embodiment of the present invention. It can be seen that the method proposed in the present invention can effectively reduce the number of constraint violations, quickly achieve a convergence effect, and can more quickly find the optimal production control system to achieve improved economic benefits.

[0133] Figure 5 It is a comparison diagram of the injection-production ratio curves before and after optimization within a control cycle in an embodiment of the present invention, showing the change of the injection-production ratio with the reservoir development time before and after optimization. It can be seen that the injection-production ratio constraint optimization method proposed in the present invention (corresponding to the optimized curve) makes the injection-production ratio well meet the constraint range of 0.8-1.2.

[0134] Figure 6 It is a comparison diagram of the cumulative oil production curves before and after optimization within a control cycle in an embodiment of the present invention, which shows the change of the cumulative oil production with the reservoir development time before and after optimization. It can be seen that the optimized scheme (corresponding to the optimized curve) of the method proposed in the present invention has a larger cumulative oil production, which effectively achieves the oil increase effect under the premise of the injection-production ratio constraint.

[0135] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A reservoir-constrained injection-production optimization method based on safety reinforcement learning, characterized in that: The specific steps include: Step 1: construct an injection-production optimization model with reservoir injection-production ratio constraints and a Markov decision process with reservoir injection-production ratio constraints, and determine optimization variables, constraint variables and objective functions; Step 2: Based on the WCSAC safety reinforcement learning algorithm, a reinforcement learning agent model for injection-production regulation with reservoir injection-production ratio constraints is constructed; Step 3: Based on the reservoir numerical simulator, the intelligent agent and the reservoir numerical simulation are continuously interacted, and data samples generated during the interaction are collected and stored in the experience playback buffer; Step 4: Combine the experience replay buffer data to guide the training and update of the reward critic network and the safety critic network, and use the conditional risk value assessment mechanism to guide the training and update of the policy network; Step 5: Repeat steps 3 and 4 until the maximum number of training rounds is reached to obtain a trained policy network. Step 6: Input the reservoir pressure field and water saturation field states into the trained strategy network, output the optimal production system under the current state, guide the reservoir injection and production operation according to the production system, and realize injection and production optimization constrained by injection-production ratio; The specific process of step 1 is as follows: Step 1.1: Construct an injection-production optimization model with reservoir injection-production ratio constraints: in, is to find the minimum value of the objective function with x as the independent variable; N(x) is the objective function; x represents the production system, the liquid production of the production well and the injection volume of the injection well at each time step; NPV is the economic net present value; x low and x up They represent the lower and upper limits of the boundary constraints of x respectively; g(x) represents the injection-production ratio constraint; the calculation formula of the economic net present value is as follows: Where T is the maximum time step period per round; Q o,t is the oil production rate of the block at the tth time step; r o represents the crude oil price; Q w,t is the water production rate of the block at the tth time step; r w Indicates the cost of water treatment; Q i,t is the injection rate of the injection well at the tth time step; r i represents the water injection cost; b is the annual discount rate; Δt is the duration of the t-th time step in days; p t Indicates the length of time in days that has passed in the tth time step; Step 1.2: The reservoir injection-production ratio constrained Markov decision process uses a seven-tuple Description, where S is the state space, including the state variables of reservoir pressure field and water saturation field; A is the action space, including the production rate of production wells and the injection rate of water injection wells; is the state transition probability; is the reward function; c is the cost function; d is the injection-production ratio constraint threshold; γ is the discount factor; Step 1.3, specify that the optimization variables are the production rate of the production well and the injection rate of the injection well; the constraint variables are the upper boundary, lower boundary and injection-production ratio constraint of the single well optimization variables; the objective function is the economic net present value, and the goal is to maximize the economic net present value under the premise of satisfying the injection-production ratio constraint; The specific process of step 2 is: Step 2.1: Obtain the reservoir model and convert the state s at the tth time step t Set to the pressure value and water saturation value for each grid point in the reservoir model as follows: s t ={p1,...,p M ;w1,...,w M }(3); Among them, p M 、w M are the pressure value and water saturation value of the Mth grid point respectively; The pressure value is normalized using the min-max normalization function, and the calculation formula is as follows: Where, f(·) is the normalization function; p m is the pressure value of the mth grid point; p min 、p max They are the minimum pressure value and the maximum pressure value of the current pressure state respectively; Step 2.2: Set the action a of the tth time step t Set the injection and production plan for each well at the current time step, including the liquid production rate of all production wells and the water injection rate of all water injection wells, as shown below: in, For N prod The liquid production rate of the production well; For N inj The liquid production rate of the injection well; L i is the liquid production rate of the ith production well; L min and L max Respectively represent the minimum and maximum liquid production rates of the production wells; W j is the injection rate of the jth injection well; W min and W max They represent the minimum and maximum injection rates of the injection wells respectively; Step 2.3: The cost function c of the tth time step t Set to the extent to which the injection-production ratio constraint is violated at the current time step, as follows: Among them, IPR t represents the injection-production ratio at the tth time step; q o,i ,q w,i and q wj,j IPR represents the oil production rate of the i-th production well, the water production rate of the i-th production well, and the water injection rate of the j-th injection well; min and IPR max They represent the minimum and maximum values ​​that do not violate the injection-production ratio constraint; Step 2.4: Change the reward function of the tth time step Defined as: Where δ is the penalty coefficient for violating the injection-production ratio constraint; Step 2.5: Determine the initial parameters of the learning environment of the reservoir injection-production ratio constraint injection-production regulation reinforcement learning agent model, including: discount factor γ, temperature parameter α, strategy network learning rate Reward critic network learning rate λ μ , security critic network learning rate λ η , injection-production ratio constraint threshold d, risk level ρ, damping coefficient λ K , the maximum time step period T per round.

2. The reservoir-constrained injection-production optimization method based on safety reinforcement learning according to claim 1 is characterized in that: The specific process of step 3 is as follows: Step 3.1: At the tth time step, the agent follows the state s t Output action a t , after the relevant linear transformation, the actual injection and production control system of the reservoir is obtained. The reservoir numerical simulator eclipse executes according to the control system to obtain the state s at the t+1th time step t+1 , and calculate the reward function for the tth time step Cost function c t ; The formula for linear transformation is as follows: Where f′(·) represents the actual injection-production control system after linear transformation; k represents action a t An element in t ; Respectively represent the upper and lower limits of production wells or water injection wells; Step 3.2: Add the data samples generated during the interaction to the experience playback buffer Ω. The collected data samples are Step 3.3: At time steps t=0~T, loop through steps 3.1 to 3.2 and record the cumulative reward for each round.

3. The reservoir-constrained injection-production optimization method based on safety reinforcement learning according to claim 2 is characterized in that: The specific process of step 4 is as follows: Step 4.1: predefine a fixed value B, compare B with the number of samples in the experience replay buffer Ω, and when the number of samples in Ω is greater than B, randomly sample B sample data from Ω to update the reward critic network, safety critic network, and policy network; Step 4.2: Use the sampled data to calculate the target reward Q value and update the network parameters of the reward critic by minimizing the loss function; the formula is as follows: Among them, Q target (s t ,a t )、Q(s t ,a t ) represent the target reward Q value and reward Q value of the t-th time step respectively; Q(s t+1 ,a t+1 ) represents the reward Q value at the t+1th time step; represents the expectation of the target value; π(a t+1 |s t+1 ) represents the policy network at the t+1th time step; L reward_critic represents the reward critic network loss; Step 4.3: Use the sampled data to calculate the target safety Q value and target safety variance value, and update the parameters of the safety critic network by minimizing the loss function. The formula is as follows: Q C,target (s t ,a t )=c t +γQ C (s t+1 ,a t+1 )(12); Among them, Q C,target (s t ,a t ), Q C (s t ,a t ), V C,target (s t ,a t ) and V C (s t ,a t ) represent the target safety Q value, safety Q value, target safety variance value and safety variance value at the t-th time step respectively; Q C (s t+1 ,a t+1 ) and V C (s t+1 ,a t+1 ) represent the safety Q value and safety variance value of the t+1th time step respectively; L safety_critic Indicates security critic network loss; Step 4.4: Based on the updated reward critic network and safety critic network, combined with the conditional risk value assessment mechanism, guide the policy network update to ensure that the strategy maximizes the return while satisfying the constraints; the formula is as follows: Where exp(·) is an exponential function with base e; Φ -1 Represents the inverse cumulative distribution function of the standard normal distribution; π(a t |s t ) represents the policy network at the tth time step; represents the expectation of the target value; β represents the weight coefficient for adjusting the safety constraint; V CVaR represents the conditional risk value; K represents the damping term value; L policy represents the policy network loss.

Images