Multi-target oil reservoir development strategy automatic generation and optimization system based on reinforcement learning

Abstract

The invention relates to the technical field of oil and gas field development engineering, and discloses a reinforcement learning-based multi-target oil reservoir development strategy automatic generation and optimization system, which comprises a flow field calculation module for executing streamline and full-physical numerical simulation and generating a physical confidence coefficient based on flux deviation; the topology reconstruction module is used for generating virtual control nodes and constructing a dynamic connected graph structure by utilizing hysteresis comparison logic based on streamline geometric features and a physical confidence coefficient; the decision reasoning module is used for extracting structural features of the dynamic connected graph by utilizing a graph neural network and outputting a normalized control action value of a virtual control node; the instruction decoding module maps the normalized control action value into a physical water injection instruction using an action mask generated based on the physical confidence coefficient. Through physical data fusion and action space recombination, the problems of dimensionality disaster and physical consistency of large-scale well pattern optimization are solved.

Description

Technical Field

[0001] This invention relates to the field of oil and gas field development engineering technology, specifically to an automatic generation and optimization system for multi-objective reservoir development strategies based on reinforcement learning. Background Technology

[0002] Water injection is a primary means of maintaining formation energy and improving recovery rates during secondary oil recovery. Its core lies in maximizing economic benefits by adjusting the injection and production operation system in real time based on geological characteristics and production dynamics. Traditional strategy formulation methods based on full physical numerical simulation combined with heuristic optimization algorithms are limited by the massive amount of grid computation. When faced with large-scale well network inversion and optimization tasks, the computation time is too long, making it difficult to meet the real-time decision-making requirements of the production site.

[0003] Existing methods employ an end-to-end black-box control model, directly establishing a mapping relationship between production data and single-well control commands. However, this approach ignores the physical topology of inter-well connectivity, resulting in an excessively high action space dimension when the model deals with large-scale well networks. This not only makes training convergence difficult but also makes it prone to getting trapped in local optima. Furthermore, it is difficult to identify uncertainties in the geological model, and the generated control strategies may pose a risk of water channeling in high-conductivity channel areas, making it difficult to guarantee the safety and reliability of the decision-making scheme in engineering practice. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an automatic generation and optimization system for multi-objective reservoir development strategies based on reinforcement learning. This system solves the problems of existing reservoir development optimization methods, which suffer from difficulty in converging algorithms under large-scale well network conditions due to excessively high dimensionality of the action space, and the problems of poor physical consistency of water injection strategies and the risk of water channeling caused by the lack of physical mechanism constraints in pure data-driven models.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solution: an automatic generation and optimization system for multi-objective reservoir development strategies based on reinforcement learning, including a data interface module, which is used to format reservoir geological and production data into state tensors; The flow field solution module performs streamline and full physical numerical simulations based on the state tensor, and generates physical confidence coefficients based on the deviation between the streamline flux field and the physical flux field. The topology reconstruction module calculates the spatial control correlation of injection wells based on streamline characteristics and physical confidence coefficients, and uses hysteresis comparison logic to generate virtual control nodes and construct a dynamic connected graph structure. The decision reasoning module uses graph neural networks to extract the structural features of a dynamically connected graph and outputs the normalized control action values ​​of virtual control nodes. The instruction decoding module uses an action mask generated based on the physical confidence coefficient to map the normalized control action value into a physical water injection instruction, and feeds back the specific water injection instruction to the flow field calculation module to achieve closed-loop optimization of the multi-objective development strategy.

[0006] This invention provides an automatic generation and optimization system for multi-objective reservoir development strategies based on reinforcement learning. It has the following beneficial effects: 1. This invention introduces a physical confidence coefficient through a flow field calculation module to verify the flux deviation between streamline simulation and full physical simulation in real time. While retaining the advantages of high computational efficiency and intuitive streamline geometry of streamline simulation, it uses full physical simulation to correct the calculation errors of streamline algorithm in complex heterogeneous reservoirs, ensuring that the flow field features input to the graph neural network have clear physical meaning, and improving the interpretability of the reservoir development strategy generation process and the algorithm convergence efficiency.

[0007] 2. This invention utilizes a dynamic connectivity graph structure based on time-varying flow tube clustering to adaptively reorganize the control action space following the dynamic evolution of reservoir seepage channels. By reducing the high-dimensional discrete single-well control to low-dimensional continuous cluster control, the dimensionality of the action space is reduced, solving the problem of convergence difficulty of reinforcement learning algorithms under large-scale well network conditions.

[0008] 3. This invention automatically masks invalid or physically distorted water injection paths through physical confidence masks and uses streamline flight time to construct early warning penalty factors to suppress the risk of water channeling that may occur in high-conductivity channels. It can automatically avoid aggressive water injection operations that may lead to reservoir connectivity failure, and achieve multi-objective synergistic optimization between economic benefit goals and engineering safety constraints. Attached Figure Description

[0009] Figure 1 This is a perspective view of the present invention; Figure 2 This is a schematic diagram of the present invention. Detailed Implementation

[0010] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0011] See attached document Figure 1 The system for automatic generation and optimization of multi-objective reservoir development strategies based on reinforcement learning includes a data interface module, a flow field calculation module, a topology reconstruction module, a decision reasoning module, and an instruction decoding module.

[0012] The data interface module connects to external reservoir geological modeling databases and production history databases, reading static geological model data and dynamic production history data of the target reservoir. Static geological model data includes grid porosity, grid permeability, and fault distribution information; dynamic production history data includes historical injection volume of injection wells, historical fluid production of production wells, and wellhead pressure data.

[0013] The flow field calculation module includes a streamline simulation unit, a full physical simulation unit, and a confidence level verification unit. The streamline simulation unit employs a streamline numerical simulation algorithm to calculate the overall pressure distribution and fluid saturation distribution at the decision time step, extracting the streamline geometry between injection wells and production wells. The full physical simulation unit uses a finite difference numerical simulation algorithm to calculate the reservoir dynamic response and generate a reward signal. The confidence level verification unit compares the streamline flux field generated by the streamline simulation unit with the physical flux field generated by the full physical simulation unit, calculating the physical confidence coefficient.

[0014] The topology reconstruction module receives streamline geometry features and physical confidence coefficients output by the flow field solution module. Internally, the topology reconstruction module integrates a flow tube similarity calculation unit and a hysteresis comparison logic unit. The flow tube similarity calculation unit calculates the spatial correlation of the control effect of different injection wells on the production well group. The hysteresis comparison logic unit determines the aggregation or separation state between injection wells based on preset dual threshold parameters and a time-locked window, mapping the physical injection wells to a dynamically changing number of virtual control nodes.

[0015] The decision reasoning module receives the dynamically connected graph structure constructed by the topology reconstruction module. The decision reasoning module utilizes a graph neural network architecture to extract and aggregate features from the nodes in the dynamically connected graph structure, outputting normalized control action values ​​for the virtual control nodes.

[0016] The command decoding module maps the normalized control action values ​​output by the decision reasoning module back to the specific water injection commands of the physical injection wells. The command decoding module includes a connectivity masking unit, which masks invalid injection paths based on streamline connectivity information. The command decoding module then sends the processed final water injection command to the full physical simulation unit in the flow field calculation module.

[0017] See attached document Figure 2 This invention provides a method for automatically generating and optimizing multi-objective reservoir development strategies based on reinforcement learning, comprising the following steps: The system loads the initial geological model of the target reservoir, defines the physical set of water injection wells and the physical set of production wells, and initializes the network parameters of the decision reasoning module.

[0018] At the current decision time step, the flow field solution module calls the streamline simulation unit to perform streamline tracing calculations, obtains the streamline distribution coefficient and streamline flight time from the injection well to the production well, and reads the grid flux data output by the full physical simulation unit of the previous time step. It then calculates the flux deviation between the current streamline simulation result and the full physical simulation result, and generates the physical confidence coefficient corresponding to the inter-well connection path.

[0019] The topology reconstruction module constructs the flow tube response vector based on the streamline allocation coefficient and streamline flight time, and calculates the similarity matrix between injection wells. It applies hysteresis comparison logic including merging threshold, splitting threshold and time locking window to determine the clustering structure of injection wells in the current time step, generates virtual control nodes, and constructs a directed weighted topology graph with virtual control nodes and physical production wells as graph nodes, effective streamline connections between wells as graph edges, and streamline allocation coefficient, streamline flight time and physical confidence coefficient as edge feature attributes.

[0020] The decision reasoning module takes the directed weighted topology graph as input, and through neural network reasoning, outputs the water injection intensity adjustment ratio for each virtual control node. It also assigns the virtual control node actions to their respective physical water injection wells and applies an action mask based on streamline connectivity. Finally, the control commands are input to the full physical simulation unit for numerical simulation, advancing the simulation time and obtaining oil production and water production data.

[0021] The system calculates the flow field matching reward based on oil production, water production, and streamline distribution coefficients, stores the state transition data in the experience replay buffer, and updates the strategy parameters of the decision reasoning module. The system repeats the above steps until the preset development cycle end time is reached.

[0022] In the hybrid flow field calculation, feature extraction, and physical confidence verification, the flow field solution module performs each decision at each time step. The streamline simulation unit is invoked. Based on the current reservoir pressure field and saturation field, the streamline simulation unit uses a streamline tracing algorithm (such as Pollock's algorithm) based on Darcy's law to trace the trajectory of fluid particles along the pressure gradient direction starting from the injection well grid.

[0023] For water injection wells With production well The well pair combination and streamline simulation unit are used to calculate the water injection distribution coefficient. and average flight time These represent the strength of the wellbore control effect and the average physical time of the injected fluid migration, respectively.

[0024] The confidence verification unit reads the equivalent wellbore flux from the full physical simulation and the streamline flux from the streamline simulation, and calculates the normalized deviation between the two. Based on this normalized deviation, the system uses a Gaussian decay function to calculate the connection path. Physical confidence coefficient As the deviation increases, the confidence level decreases at a Gaussian rate, and this coefficient serves as the feature weights and gating basis for subsequent graph neural network edges.

[0025] See attached document Figure 2 The topology reconstruction module constructs a flow tube response feature vector that includes the water injection allocation coefficient and the flight time coupled by exponential decay, and calculates the spatial control similarity between water injection wells.

[0026] The topology reconstruction module uses a dual-threshold hysteresis comparison logic to determine the clustering state between injection wells: when the similarity is higher than the merging threshold, it is determined to be aggregated; when it is lower than the splitting threshold, it is determined to be separated. Furthermore, the state transition must satisfy the constraint of a time-locked window, thereby mapping physical injection wells to virtual control nodes. .

[0027] In constructing a directed weighted graph When calculating the aggregation attribute for an edge pointing from a virtual control node to a production well: the aggregation assignment coefficient is the sum of the coefficients of the child nodes; the aggregation flight time is the weighted average; and the aggregation confidence coefficient adopts a minimum value strategy. This means taking the minimum value among the physical confidence coefficients corresponding to all physical injection wells within the virtual node to ensure control safety.

[0028] In graph neural network decision reasoning, the decision reasoning module uses a message-passing neural network to process the directed weighted graph. .

[0029] Virtual control node initial state vector Including average bottom hole pressure and cumulative injection volume; production well nodes initial state This includes the liquid production rate and water content.

[0030] Then execute Layered message passing. During the iteration process, the decision reasoning module uses the physical confidence coefficient as a gating weight to adjust the information transmission intensity between graph nodes, and aggregates neighborhood features through gating loop units to update the hidden state of the virtual control node. After the iteration ends, the decision reasoning module maps the hidden state to an action probability distribution and samples to generate normalized water injection intensity action values.

[0031] The instruction decoding module maps virtual actions to physical water-filling instructions. First, a physical constraint mask is introduced. If the well is used for water injection The maximum physical confidence level of all outflow streamlines is below the threshold. ,but ,otherwise .

[0032] for The water injection wells under its jurisdiction Target water injection rate for: ; in, for Apparent water absorption index The calculation formula is: .

[0033] In the reward calculation and closed-loop update steps, the system adopts a local attribution reward mechanism based on flow field matching. The module uses streamline allocation coefficients to attribute changes in oil and water production of production wells to virtual control nodes, and multiplies them by physical confidence coefficients to eliminate reward contributions from physically distorted paths.

[0034] Meanwhile, a flight time penalty factor is introduced: when the actual streamline flight time is less than the theoretical water channeling breakthrough time threshold, the penalty factor increases exponentially, suppressing the risk of water channeling in high-guide channels.

Claims

1. A system for automatically generating and optimizing multi-objective reservoir development strategies based on reinforcement learning, characterized in that, include: The data interface module is used to format reservoir geology and production data into state tensors; The flow field solution module performs streamline and full physical numerical simulations based on the state tensor, and generates physical confidence coefficients based on the deviation between the streamline flux field and the physical flux field. The topology reconstruction module calculates the spatial control correlation of injection wells based on streamline characteristics and physical confidence coefficients, and uses hysteresis comparison logic to generate virtual control nodes and construct a dynamic connected graph structure. The decision reasoning module uses graph neural networks to extract the structural features of a dynamically connected graph and outputs the normalized control action values ​​of virtual control nodes. The instruction decoding module uses an action mask generated based on the physical confidence coefficient to map the normalized control action value into a physical water injection instruction, and feeds back the specific water injection instruction to the flow field calculation module to achieve closed-loop optimization of the multi-objective development strategy.

2. The automatic generation and optimization system for multi-objective reservoir development strategies based on reinforcement learning according to claim 1, characterized in that, The flow field calculation module calculates the normalized deviation between the streamline flux field and the physical flux field, and uses the Gaussian decay function to calculate the physical confidence coefficient based on the normalized deviation.

3. The automatic generation and optimization system for multi-objective reservoir development strategies based on reinforcement learning according to claim 1, characterized in that, The topology reconstruction module sets the merging threshold, the splitting threshold, and the time lock window; When the spatial control correlation is higher than the merging threshold, the injection well is determined to be in an aggregated state. When the spatial control correlation is lower than the splitting threshold, the injection well is determined to be in a separated state. The state transition must meet the constraints of the time locking window.

4. The automatic generation and optimization system for multi-objective reservoir development strategies based on reinforcement learning according to claim 1, characterized in that, The edge characteristic attributes of the dynamic connected graph structure include the aggregation assignment coefficient, the aggregation streamline flight time, and the aggregation confidence coefficient. The polymer confidence coefficient is taken as the minimum value among the physical confidence coefficients of all physical injection wells included in the virtual control node.

5. The automatic generation and optimization system for multi-objective reservoir development strategies based on reinforcement learning according to claim 1, characterized in that, The decision reasoning module is also used for message passing iteration. The decision reasoning module uses the physical confidence coefficient as the gating weight to adjust the information transmission strength between graph nodes, and updates the hidden state of the virtual control node through the gating loop unit.

6. The automatic generation and optimization system for multi-objective reservoir development strategies based on reinforcement learning according to claim 1, characterized in that, The instruction decoding module is also used to generate an action mask with a value of zero to block the water injection action of the physical water injection well when the maximum value of the physical confidence coefficient of all outflow streamlines of the physical water injection well is lower than the confidence cutoff threshold.

7. The automatic generation and optimization system for multi-objective reservoir development strategies based on reinforcement learning according to claim 6, characterized in that, The instruction decoding module calculates the target water injection rate based on the normalized control action value, action mask, and the weight of the apparent water absorption index of the physical injection well within its virtual control node; the apparent water absorption index is determined by the ratio of the water injection rate to the production pressure difference in the previous time step.

8. The automatic generation and optimization system for multi-objective reservoir development strategies based on reinforcement learning according to claim 1, characterized in that, The system also includes a reward calculation module, which uses streamline allocation coefficients to attribute changes in oil production and water production of production wells to virtual control nodes, and uses physical confidence coefficients to eliminate reward contributions from physically distorted paths.

9. The automatic generation and optimization system for multi-objective reservoir development strategies based on reinforcement learning according to claim 8, characterized in that, The reward calculation module introduces a flight time penalty factor. When the actual streamline flight time is less than the theoretical water jet breakthrough time threshold, the flight time penalty factor increases exponentially.

10. The automatic generation and optimization system for multi-objective reservoir development strategies based on reinforcement learning according to claim 1, characterized in that, The system uses state transition data to update the network parameters of the decision reasoning module through a proximal policy optimization algorithm.

Images