Deep reinforcement learning based injection-production optimization method considering geological uncertainty

Abstract

The present application relates to a kind of oil reservoir development methods, more particularly to a kind of deep reinforcement learning injection-production optimization control method considering geological uncertainty, comprising the following steps: 1) obtain multiple oil reservoir geological data field, establish multiple oil reservoir numerical simulation model as environment submodel;Establish the action collector of experience collection and the learner of strategy learning, and evaluator;2) action collector and each environment submodel interaction iteration, collect sample data through parallel sampling and put into shared experience pool;3) the average value of the reward of each environment submodel is calculated and evaluated, and the learner is updated by gradient through extracting sample data in shared experience pool;4) repeat 2-3 until maximum number, save the network parameter of optimal learner;5) read the current state of the oil reservoir to be optimized, input the network of optimal learner, output optimized injection-production system.The deep reinforcement learning injection-production optimization control method considering geological uncertainty of the present application can optimize the development and production strategy of oil reservoir.

Description

Technical Field

[0001] This invention relates to reservoir development methods, specifically to a deep reinforcement learning-based injection-production optimization and control method that takes into account geological uncertainties. Background Technology

[0002] In traditional oil reservoir development, injection-production optimization has always been an important and complex issue. The geological structure and properties of oil reservoirs often exhibit significant uncertainties, such as uncertainties in parameters like permeability, porosity, and lithology. These geological uncertainties have a substantial impact on injection-production performance. Traditional planning methods and optimization strategies often overlook these geological uncertainties, leading to unsatisfactory results in actual production.

[0003] With the development of deep reinforcement learning, it has demonstrated great potential in the optimization of complex systems. Reinforcement learning algorithms, through interaction with the environment, progressively learn and optimize policies, addressing challenges such as high dimensionality, continuous action spaces, and uncertainty. However, applying reinforcement learning to reservoir injection-production optimization problems requires solving the multi-model modeling problem with multiple posterior permeability fields, as well as the issues of data diversity and reward robustness during sampling. Therefore, a novel injection-production optimization control method is needed that can consider geological uncertainties and overcome technical challenges such as multi-model modeling, data sampling, and reward calculation. This method should be able to train agents under multiple posterior permeability fields to effectively cope with geological uncertainties, thereby achieving a better injection-production optimization control strategy.

[0004] In view of this, there is a need to design a deep reinforcement learning injection-production optimization and control method that takes into account geological uncertainties, which can overcome the above-mentioned technical difficulties and effectively solve or alleviate the above-mentioned technical defects. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a deep reinforcement learning-based injection-production optimization and control method that takes into account geological uncertainties. The agent trained by this method can optimize reservoir development and production strategies, maximize reservoir recovery, and improve injection-production efficiency and output.

[0006] To address the aforementioned technical problems, this invention provides a deep reinforcement learning-based injection-production optimization and control method that considers geological uncertainties, comprising the following steps:

[0007] Step 1: Acquire multiple reservoir geological data fields, and establish multiple posterior reservoir numerical simulation models as posterior environment sub-models; establish an actioner for experience collection, a learner for policy learning, and an evaluator;

[0008] Step 2: The actor interacts and iterates with each of the posterior environment sub-models, and accelerates the sampling and collection of sample data in a distributed parallel manner, putting it into a shared experience pool;

[0009] Step 3: Evaluation is performed by calculating the average reward of each posterior environment sub-model, and the learner updates its gradient by extracting the sample data from the shared experience pool;

[0010] Step 4: Repeat steps 2-3 until the maximum number of iterations is reached, and save the optimal network parameters of the learner;

[0011] Step 5: Read the current state of the reservoir to be optimized, input the optimal network of the learner, and output the optimized injection-production regime.

[0012] In some embodiments, the reservoir geological data field is a post-hoc permeability field.

[0013] In some embodiments, in step 1, the step of acquiring multiple reservoir geological data fields and establishing multiple posterior reservoir numerical simulation models as posterior environmental sub-models includes:

[0014] N posterior permeability fields are generated using a model inference method based on geological information, thereby obtaining N posterior reservoir numerical simulation models. Then, N posterior environment sub-models are created using the posterior reservoir numerical simulation models as input.

[0015] In some embodiments, the generation process of the posterior permeability field is as follows: a geological model is established based on the permeability data, wherein the geological model includes a random field model or a Gaussian process model; the geological model is simulated and inferred through a history fitting process to generate N posterior permeability fields.

[0016] In some embodiments, step 2 includes the following steps:

[0017] Step 2.1: Set time step t and time step t+1 to their initial values;

[0018] Step 2.2: Read the state information S at time step t of the posterior environment sub-model. t And perform normalization processing;

[0019] Step 2.3: Input the read state information S at time step t into the actuator. t Output action A t Wherein action A t For each well, the injection-production regime data is provided at time step t.

[0020] Step 2.4: The posterior environment sub-model executes a reservoir numerical simulator based on the current injection-production regime data, and obtains the net present value (NPV) of reservoir production at this stage through parallel computation. t and the state information S at time step t+1t+1 , will [S t A t NPV t S t+1 This data point is added to the shared experience pool as a sample data point.

[0021] Step 2.5: Set the value of time step t to the value of time step t+1, and the value of time step t+1 is shifted forward accordingly;

[0022] Step 2.6: Determine whether the numerical simulation of a production cycle is complete. If it is complete, calculate the round reward within a production cycle. Otherwise, repeat steps 2.2-2.5 until the final state is reached.

[0023] In some embodiments, in step 2.2, the state information S at time step t... t The reservoir model, including the pressure field P, saturation field So, and permeability field X of the posterior environment sub-model at time step t, is shown below.

[0024] S t =[P,So,X].

[0025] In some embodiments, in step 2.3, the input to the actuator is the read state information S. t The action A output by the actuator t A neural network for the injection and acquisition regime data at the current time step, the neural network including a process for the state information S t It consists of convolutional layers and fully connected linear output layers that make decisions.

[0026] In some embodiments, action A t The decision-making schemes for each well at the current time step are as follows: production wells use bottom hole pressure control, and water injection wells use water injection rate control. Action A... t The formula is shown below.

[0027]

[0028] Among them, BHP prd,i W represents the bottom hole pressure of the i-th production well. inj,i N represents the water injection rate of the i-th injection well. prd N represents the number of production wells. inj This indicates the number of water injection wells.

[0029] In some embodiments, step 3 includes the following steps:

[0030] Step 3.1: Randomly select multiple sample data points from the shared experience pool [S] t At NPV t S t+1 ];

[0031] Step 3.2: The evaluator uses the current time step's state information S from the sample data as a basis. t and the action A t Forecast the projected net present value (NPV) at time step t. t The network parameters of the evaluator are updated using time-series difference error;

[0032] Step 3.3: Update the learner network parameters by performing backpropagation based on the gradient provided by the evaluator;

[0033] Step 3.4: The network of each of the aforementioned actors is copied and updated to resemble the network of the learner.

[0034] In some embodiments, in step 3.2: the timing difference error is expressed as:

[0035] δ t =NPV t +γQ ω (S t+1 A t+1 )-Q ω (S t A t )

[0036] Where, δ t NPV represents the timing difference error. t Let Q be the economic net present value at time step t, γ be the discount factor, and Q be the net present value at time step t. ω (s t ,a t S represents the evaluator network. t A represents the state at time step t. t Indicates an action.

[0037] Through the above technical solution, this invention provides a deep reinforcement learning-based injection-production optimization and control method that considers geological uncertainties. This method establishes a reservoir geological data field based on reservoir geological data and trains an agent based on this data field. Through multiple iterative optimizations, the agent can gradually learn and adopt the optimal injection-production control strategy, thereby effectively improving the reservoir's recovery rate. By using multi-model modeling and reward averaging, the agent can more robustly optimize the control strategy, reduce development risks, and ensure the continuous and efficient development of the reservoir.

[0038] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the overall process of the deep reinforcement learning-based injection-production optimization and control method that considers geological uncertainties in a specific embodiment of the present invention.

[0040] Figure 2 This is a convergence curve of the deep reinforcement learning injection-production optimization control method considering geological uncertainties in a specific embodiment of the present invention.

[0041] Figure 3 A schematic diagram of the production regime after optimizing the production well using a deep reinforcement learning injection-production optimization and control method that considers geological uncertainties, as described in a specific embodiment of the present invention.

[0042] Figure 4 A schematic diagram of the injection volume after optimizing the injection well using the deep reinforcement learning injection-production optimization and control method that considers geological uncertainties in a specific embodiment of the present invention. Detailed Implementation

[0043] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0044] The terms "comprising" or "including" as used in this invention mean that the element preceding the word covers the element listed after the word, and do not exclude the possibility that other elements may also be included.

[0045] All terms used in this invention have the same meaning as understood by one of ordinary skill in the art to which this disclosure pertains, unless otherwise specifically defined. It should also be understood that terms defined in general dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant art, and not as idealized or highly formalized, unless expressly defined herein.

[0046] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0047] See Figure 1 This invention provides a deep reinforcement learning-based injection-production optimization method that considers geological uncertainties, comprising the following steps:

[0048] Step 1: Acquire multiple reservoir geological data fields, and establish multiple posterior reservoir numerical simulation models as posterior environment sub-models; establish an experience collection Actor, a policy learning Learner, and an evaluator Critic.

[0049] Step 2: The Actor interacts and iterates with each posterior environment sub-model, and accelerates the sampling and collection of sample data through a distributed parallel approach, putting it into a shared experience pool;

[0050] Step 3: Evaluation is performed by calculating the average reward of each posterior environment sub-model. The learner updates its gradient by extracting sample data from the shared experience pool.

[0051] Step 4: Repeat steps 2-3 until the maximum number of iterations is reached, and save the network parameters of the optimal learner;

[0052] Step 5: Read the current state of the reservoir to be optimized, input the network of the optimal learner, and output the optimized injection-production regime.

[0053] This deep reinforcement learning-based injection-production optimization method, which considers geological uncertainties, establishes a posterior environment sub-model based on reservoir geological data. It models the injection-production process as a Markov decision process, establishing an Actor for experience collection, a Learner for policy learning, and a Critic. State information S is obtained through iterative interaction between the Actor and each posterior environment sub-model. t The D4PG algorithm is used to collect sample data in parallel and put it into a shared experience pool, which is then sent to the learner for learning. The agent's policy is modeled through a neural network, making full use of computer resources and improving computational efficiency. The average reward of multiple posterior environment sub-models under a given state and action is used as the final reward signal, enhancing the robustness and accuracy of the reward and improving the effect of reinforcement learning. Feedback is used to calculate the reward and update the agent's policy. A gradient optimization algorithm is used to update the policy parameters by maximizing the cumulative reward, ensuring that the agent can more robustly optimize the control strategy, reduce development risks, and ensure the continuous and efficient development of the reservoir. By repeatedly performing steps 2 and 3 up to a set number of times, new sample data is continuously generated through interaction with each environment sub-model for the learner's network to learn, in order to obtain the optimal learner. In some embodiments, the change in the average reward after multiple iterations is as follows: Figure 2 As shown, the state information of the reservoir to be optimized is input into the optimal learner generated by this instance, and the optimized injection-production regime is output. The production regime data of the production wells and injection wells are as follows: Figure 3 and Figure 4 As shown.

[0054] Oil reservoir geological data includes geological information such as porosity, permeability, and lithology. As a preferred implementation, a post-hoc permeability field can be selected as the oil reservoir geological data field. This application mainly uses the post-hoc permeability field as the oil reservoir geological data field and is only used to illustrate and explain the present invention, and is not intended to limit the present invention.

[0055] As a specific implementation method, in step 1, N (N≥2) posterior permeability fields are generated using a model inference method based on geological information, thereby obtaining N posterior reservoir numerical simulation models. Then, N posterior environment sub-models are created using the posterior reservoir numerical simulation models as input to consider the impact of geological uncertainties on the injection-production optimization process. The injection-production optimization problem is modeled as a Markov decision model process. Key elements are defined as: strategy, state S. t Action A t , reward function, where state S t Including the current development status of the reservoir, Action A t Feasible oil production control strategies are defined, and the reward function can be designed based on indicators such as injection-production efficiency, productivity, injection rate, and pressure balance.

[0056] As a specific implementation method, the generation process of the posterior permeability field is as follows: a geological model is established based on permeability data, wherein the geological model includes a random field model or a Gaussian process model; the acquired permeability data is simulated and inferred through a history fitting process to produce N posterior permeability fields.

[0057] As one specific implementation method, step 2 includes the following steps:

[0058] Step 2.1: Set time step t and time step t+1 to their initial values, where time step t+1 is the time step following time step t;

[0059] Step 2.2: Read the state information S at time step t of the posterior environment sub-model. t And perform normalization processing;

[0060] Step 2.3: Input the read state information S at time step t into the Actor. t Output action A t Action A t For each well, the injection-production regime data is provided at time step t.

[0061] Step 2.4: The posterior environment sub-model executes a reservoir numerical simulator based on the current injection-production regime data. Through parallel computation, the net present value (NPV) of reservoir production at this stage is obtained. t and the state information S at time step t+1 t+1 , will [S t At NPV t S t+1 This data point is added to the shared experience pool as a sample data point.

[0062] Step 2.5: Set the value of time step t to the value of time step t+1, and the value of time step t+1 is shifted forward accordingly;

[0063] Step 2.6: Determine whether the numerical simulation of a production cycle is complete. If it is complete, calculate the round reward within a production cycle. Otherwise, repeat steps 2.2-2.5 until the final state is reached.

[0064] The total number of days for reservoir model generation can be divided into multiple time steps. For example, in some embodiments, a reservoir model with a size of 60*60 has 13 wells (9 production wells and 4 injection wells) for development, with a total production period of 1800 days. Each step is set to 180 days, thus the optimization time step is set to 10 steps. The permeability field follows a Gaussian distribution, resulting in 20 posterior permeability fields. The reservoir numerical simulator can obtain actual data such as reservoir pressure field, saturation field, oil production, and fluid production after production based on the set injection-production regime data. The net present value (NPV) is then calculated. t The net present value (NPV) at time step t t The calculation formula is as follows:

[0065]

[0066] in, This represents the cumulative oil production during that period, in STB (standard barrels). Represents the cumulative water production of the block, in STB units; c1 represents the cumulative injection volume, in STB; c2 represents the crude oil price, in RMB / STB; c3 represents the oilfield produced water treatment cost, in RMB / STB; c3 represents the injected water cost, in RMB / STB.

[0067] Step 2: Generate current state information S based on the policy network of the actor. t The following action A t The reservoir numerical simulator is invoked to calculate the response set corresponding to the posterior environment sub-model at the current time step, i.e., the net present value (NPV). t and the state information S of the next time step t+1 The sample data [S] constitutes the sample data. t A t NPV t S t+1The data is stored in the shared economy pool and repeatedly calculated until the numerical simulation of a production cycle is completed, that is, the response set corresponding to each time step is calculated. Moreover, the reservoir numerical simulator can calculate the response set corresponding to multiple posterior environment sub-models in parallel, which greatly speeds up the calculation.

[0068] As a specific implementation method, in step 2.2, the state information S at time step t... t The reservoir model, including the posterior environmental sub-model at time step t, contains the pressure field P, saturation field So, and permeability field X, as shown below.

[0069] S t =[P,So,X].

[0070] In one specific implementation, in step 2.3, the input to the actor is the read state information S. t The action A output by the actor t A neural network for the injection and acquisition regime data at the current time step, the neural network including a process for state information S t It consists of convolutional layers and fully connected linear output layers that make decisions.

[0071] As a specific implementation method, action A t The decision-making schemes for each well are set at the current time step. Production wells use bottomhole pressure control, and water injection wells use water injection rate control. Action A t The formula is shown below.

[0072]

[0073] Among them, BHP prd,i W represents the bottom hole pressure of the i-th production well. inj,i N represents the water injection rate of the i-th injection well. prd N represents the number of production wells. inj This indicates the number of water injection wells.

[0074] As a specific implementation method, step 3 includes the following steps:

[0075] Step 3.1: Randomly select multiple sample data points from the shared experience pool [S] t A t NPV t S t+1 ];

[0076] Step 3.2: The evaluator Critic uses the current time step's state information S from the sample data. t And Action A t Forecast the projected net present value (NPV) at time step t. tThe network parameters of the evaluator Critic are updated using time-series difference error.

[0077] Step 3.3: Update the learner network parameters by performing backpropagation using the gradient provided by the evaluator Critic;

[0078] Step 3.4: The networks of each actor are copied and updated to resemble the network of the learner.

[0079] As a specific implementation method, the timing difference error is expressed as:

[0080] δ t =NPV t +γQ ω (S t+1 A t+1 )-Q ω (S t A t )

[0081] Where, δ t NPV represents the timing difference error. t Let Q be the economic net present value at time step t, γ be the discount factor, and Q be the net present value at time step t. ω (s t ,a t S represents the evaluator network. t A represents the state at time step t. t Indicates an action.

[0082] Step 3 of this embodiment evaluates the agent's performance by calculating the average reward of each posterior environment sub-model. It calculates the reward of each posterior environment sub-model under a given state and action, takes the average as the final reward signal, and uses a gradient optimization algorithm to update the policy parameters by maximizing the cumulative reward. This deep reinforcement learning-based injection-production optimization and control method, which considers geological uncertainties, utilizes the average reward of multiple posterior environment sub-models as the final reward signal, enhancing the robustness and accuracy of the reward, improving the learning effect of reinforcement learning, enabling the agent to more robustly optimize the control strategy, reducing development risks, and ensuring the continuous and efficient development of the reservoir.

[0083] As can be seen from the above description, the advantages of this invention are as follows: First, it uses reinforcement learning algorithms to optimize reservoir development and production control strategies under different geological uncertainties, maximizing reservoir recovery and improving injection-production efficiency and output. Second, it utilizes the D4PG algorithm to collect sample data in parallel and put it into a shared experience pool, making full use of computer resources and improving computational efficiency. Third, it uses the average reward of multiple posterior environment sub-models under a given state and action as the final reward signal, enhancing the robustness and accuracy of the reward and improving the effect of reinforcement learning. Fourth, it solves the multi-model modeling problem by obtaining a posterior reservoir numerical simulation model based on the posterior geological data field and then using the reservoir numerical simulation model as input to create a posterior environment sub-model.

[0084] The optional embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above embodiments. Within the scope of the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all fall within the protection scope of the embodiments of the present invention.

[0085] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the embodiments of the present invention will not describe the various possible combinations separately.

[0086] Furthermore, various different implementations of the present invention can be combined arbitrarily, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed in the present invention.

Claims

1. A deep reinforcement learning-based injection-production optimization and control method considering geological uncertainties, characterized in that, Includes the following steps: Step 1: Acquire multiple reservoir geological data fields, and establish multiple posterior reservoir numerical simulation models as posterior environment sub-models; establish an actioner for experience collection, a learner for policy learning, and an evaluator; Step 2: The actor interacts and iterates with each of the posterior environment sub-models, and accelerates the sampling and collection of sample data in a distributed parallel manner, putting it into a shared experience pool; Step 2 includes the following steps: Step 2.1: Set time step t and time step t+1 to their initial values; Step 2.2: Read the state information S at time step t of the posterior environment sub-model. t And perform normalization processing, where, State information S at time step t t The reservoir model, including the pressure field P, saturation field So, and permeability field X of the posterior environment sub-model at time step t, is shown below. ； Step 2.3: Input the read state information S at time step t into the actuator. t Output action A t Wherein action A t For each well, the injection-production regime data is provided at time step t, where... The input to the actuator is the read state information S. t The action A output by the actuator t A neural network for the injection and acquisition regime data at the current time step, the neural network including a process for the state information S t The convolutional layers and the fully connected linear output layer that makes the decisions; Step 2.4: The posterior environment sub-model executes a reservoir numerical simulator based on the current injection-production regime data, and obtains the economic net present value of reservoir production at this stage through parallel computation. and the state information S at time step t+1 t+1 , will [S t A t , S t+1 This data point is added to the shared experience pool as a sample data point. Step 2.5: Set the value of time step t to the value of time step t+1, and the value of time step t+1 is shifted forward accordingly; Step 2.6: Determine whether the numerical simulation of a production cycle is complete. If it is, calculate the round reward within a production cycle. Otherwise, repeat steps 2.2-2.5 until the final state is reached. Step 3: Evaluation is performed by calculating the average reward of each posterior environment sub-model, and the learner updates its gradient by extracting the sample data from the shared experience pool; Step 4: Repeat steps 2-3 until the maximum number of iterations is reached, and save the optimal network parameters of the learner; Step 5: Read the current state of the reservoir to be optimized, input the optimal network of the learner, and output the optimized injection-production regime.

2. The deep reinforcement learning-based injection-production optimization and control method considering geological uncertainties according to claim 1, characterized in that, The reservoir geological data field is a post-hoc permeability field.

3. The deep reinforcement learning-based injection-production optimization and control method considering geological uncertainties according to claim 2, characterized in that, In step 1, the step of acquiring multiple reservoir geological data fields and establishing multiple posterior reservoir numerical simulation models as posterior environmental sub-models includes: N (N≥2) posterior permeability fields are generated using a model inference method based on geological information, thereby obtaining N posterior reservoir numerical simulation models. Then, N posterior environment sub-models are created using the posterior reservoir numerical simulation models as input.

4. The deep reinforcement learning-based injection-production optimization and control method considering geological uncertainties according to claim 3, characterized in that, The process of generating the posterior permeability field is as follows: a geological model is established based on the permeability data, wherein the geological model is a random field model or a Gaussian process model; the geological model is simulated and inferred through a history fitting process to generate N posterior permeability fields.

5. The deep reinforcement learning-based injection-production optimization and control method considering geological uncertainties according to claim 1, characterized in that, Action A t The decision-making schemes for each well at the current time step are as follows: production wells use bottom hole pressure control, and water injection wells use water injection rate control. Action A... t The formula is shown below. in, Indicates the first Bottom hole pressure of a production well Indicates the first The water injection rate of the injection well. Indicates the number of production wells. This indicates the number of water injection wells.

6. The deep reinforcement learning-based injection-production optimization and control method considering geological uncertainties according to claim 1, characterized in that, Step 3 includes the following steps: Step 3.1: Randomly select multiple sample data points from the shared experience pool [S] t A t , S t+1 ]; Step 3.2: The evaluator uses the current time step's state information S from the sample data as a basis. t and the action A t Forecast the projected net present value at time step t The network parameters of the evaluator are updated using time-series difference error; Step 3.3: Update the learner network parameters by performing backpropagation based on the gradient provided by the evaluator; Step 3.4: The network of each of the aforementioned actors is copied and updated to resemble the network of the learner.

7. The deep reinforcement learning-based injection-production optimization and control method considering geological uncertainties according to claim 6, characterized in that, In step 3.2: the timing difference error is expressed as: in, Indicates timing difference error. Let t be the net present value of the economy at time step t. As a discount factor, S represents the evaluator network. t A represents the state at time step t. t Indicates an action.

Images