An efficient numerical simulation method for carbon dioxide geological storage based on reinforcement learning

Abstract

The present invention discloses an efficient numerical simulation method for carbon dioxide geological storage based on reinforcement learning. This method, which lies at the intersection of oil and gas reservoir numerical simulation and artificial intelligence, comprises the following steps: Step 1: Establishing a fully implicit numerical simulation model for carbon dioxide geological storage; Step 2: Constructing a reinforcement learning environment, including designing a state space, action space, and reward function; Step 3: Offline training of a reinforcement learning agent to learn a time step control strategy; and Step 4: Deploying the trained time step control strategy in actual numerical simulations to achieve real-time step adjustment. By introducing a reinforcement learning agent and constructing a state-action-reward feedback loop, this method achieves intelligent dynamic adjustment of the time step, breaking through the limitations of traditional heuristic step control methods and possessing significant engineering application value.

Description

Technical Field

[0001] The present invention belongs to the technical field of intersection of oil and gas reservoir numerical simulation and artificial intelligence, and specifically relates to an efficient numerical simulation method for geological storage of carbon dioxide based on a reinforcement learning algorithm. Background Art

[0002] Carbon dioxide geological storage (CCS) is an important carbon emission reduction technology. It prevents industrially captured carbon dioxide from entering the atmosphere and mitigates the greenhouse effect by injecting it into geological bodies such as deep saline aquifers and depleted oil and gas reservoirs. With the advancement of global carbon neutrality goals, carbon dioxide geological storage is considered one of the key paths to address climate change in the next few decades. The carbon dioxide geological storage process involves complex multiphase and multicomponent flows and rock-fluid interactions. Predicting the migration behavior, pressure response, and storage safety of injected carbon dioxide underground through numerical simulation is a core tool for project design, risk assessment, and operation optimization. However, compared with traditional oil and gas reservoir development, carbon dioxide geological storage has the following prominent technical difficulties: (1) Lack of geological data and high uncertainty; carbon dioxide storage is often located in deep saline aquifers or abandoned oil and gas reservoirs. Parameters such as geological structure, porosity, and permeability are often difficult to obtain accurately. Model inputs have large uncertainties, and a large number of numerical simulations are required to conduct risk screening and sensitivity analysis of injection schemes. (2) The temporal and spatial scale is much larger than that of conventional oil reservoirs. Carbon dioxide storage projects often involve injection periods of several decades and monitoring periods of hundreds of years. The injection scale may reach over one million tons, and the spatial scope can be expanded to tens or even hundreds of square kilometers. Numerical simulations over long periods of time and large spaces place higher demands on computational efficiency. (3) The computational load is extremely high, especially when conducting uncertainty quantification analysis (such as Monte Carlo simulation and multi-scenario simulation), which requires repeated calculations of a large number of working conditions. The computational speed of existing commercial oil and gas reservoir simulators in carbon dioxide storage applications has become a major bottleneck restricting project design and optimization.

[0003] In existing fully implicit finite volume reservoir simulators, the choice of time step has a crucial impact on numerical solution efficiency and convergence. If the time step is set too large, the nonlinear solver (typically the Newton-Raphson method) is prone to nonconvergence or solution failure, necessitating a forced step size reduction and backoff, resulting in significant computational resource waste. If the time step is set too small, while the number of Newton iterations per step is small, the total number of time steps increases significantly, significantly reducing overall computational efficiency. Currently, the following methods are widely used in the reservoir simulation industry for time step adjustment: The Predictor-Corrector method dynamically adjusts the next step size based on whether the current step size has converged. However, its drawbacks include delayed response and inadequate adjustment. The Newton iteration-based adjustment method uses a preset Newton step threshold, increasing the step size if the number of iterations is low and decreasing it if the number is high, resulting in overly coarse adjustments. The Residual Change Rate Control method adjusts the step size based on the rate of decline of the convergence residual. This method has low sensitivity and is prone to overconservatism or frequent regression. The above-mentioned methods mainly rely on simple heuristic rules, which cannot fully utilize the state information in the historical simulation process. They are also difficult to adapt to the complex, nonlinear and widely varying operating conditions of the carbon dioxide storage process. As a result, the step-size control still has the following prominent problems: the time step control strategy lacks intelligence and foresight; most adjustments are too conservative and fail to fully realize the potential of the solver; and there is a lack of generalization capabilities across operating conditions.

[0004] Therefore, there is an urgent need to develop a new time step adaptive control method based on machine learning, which can combine historical iteration information and real-time state feedback to intelligently and dynamically optimize step selection, thereby significantly improving the computational efficiency and engineering applicability of numerical simulations of carbon dioxide geological storage. Summary of the Invention

[0005] To address the above issues, the present invention proposes an efficient numerical simulation method for carbon dioxide geological storage based on reinforcement learning. By introducing a reinforcement learning agent and constructing a state-action-reward feedback loop, this method achieves intelligent dynamic adjustment of the time step, breaking through the limitations of traditional heuristic step-size control and having significant engineering application value.

[0006] The technical solutions of the present invention are as follows:

[0007] An efficient numerical simulation method for geological storage of carbon dioxide based on reinforcement learning, comprising the following steps:

[0008] Step 1: Establish a fully implicit numerical simulation model for CO2 geological storage;

[0009] Step 2: Build a reinforcement learning environment, including designing the state space, action space, and reward function;

[0010] Step 3: Offline training of the reinforcement learning agent to learn the time step control strategy;

[0011] Step 4: Deploy the trained time step control strategy in actual numerical simulation to achieve real-time step adjustment.

[0012] Furthermore, the specific process of step 1 is as follows:

[0013] Step 1.1: Based on the actual project and existing data of the CO2 geological storage project, select an oil and gas reservoir numerical simulator, import grid data into the numerical simulator, and generate a grid model for numerical simulation; the grid data includes grid type, number of grids, and dimensions;

[0014] Step 1.2: Based on existing geological knowledge, select a geological parameter generation algorithm or an inversion method based on parameter perturbation to randomly generate porosity and permeability fields;

[0015] Step 1.3: Generate a fluid model for CO2 and formation fluid based on the target reservoir depth and geothermal gradient data. The fluid model specifically includes the functional relationship between the fluid physical properties and pressure, as well as the functional relationship between the fluid relative permeability and capillary force and water saturation. The physical properties include density, viscosity, and solubility.

[0016] Step 1.4: Select a mathematical model for CO2 flow and migration based on actual site needs;

[0017] Step 1.5: Set boundary conditions and initial conditions based on the actual formation conditions, and simultaneously determine different CO2 injection strategies. Also, set the reservoir to initially saturated with formation water, and set the initial pressure to the static pressure distribution. The CO2 injection strategy includes injection well location and injection rate.

[0018] Step 1.6: Based on the established grid model, porosity and permeability field data, fluid model, flow and transport mathematical model, and initial and boundary conditions, the finite volume numerical method and the inverse Euler method are used to discretize the mathematical model equations in space and time, respectively, to obtain a fully implicit numerical simulation model. The formula of the fully implicit numerical simulation model is:

[0019] ;

[0020] in, It is a fully implicit numerical simulation model; 、 Represent the current time step and the previous time step respectively; and They represent the unknowns to be solved in the current time step and the unknowns that have been solved in the previous time step respectively; Indicates the control conditions for going down the well at the current time step; Indicates the time step size of the current time step.

[0021] Furthermore, the specific process of step 2 is as follows:

[0022] Step 2.1, state space design; the state vector of the current time step in the state space The time step including the previous time step , the number of Newton iterations in the previous time step , the Newton residual at the current time step , the time step adjustment ratio of the previous time step , and the maximum rate of change of pressure and saturation in the current time step ; Specifically expressed as:

[0023] ;

[0024] in, Contains the maximum rate of change of pressure in the current time step and maximum rate of change of saturation ;

[0025] Step 2.2, action space design; define the discrete action set of the current time step in the action space ;

[0026] Step 2.3: Reward function design; the single-step reward function is designed as follows:

[0027] ;

[0028] Where, is the reward value of the current time step; Indicates the number of Newton iterations in the current time step; Indicates the convergence flag of the current time step. The value is 1 when convergence fails and 0 when convergence is normal. Represents the convergence failure penalty coefficient.

[0029] Furthermore, the specific process of step 3 is as follows:

[0030] Step 3.1. Based on the design of the state space, action space and reward function in step 2, encapsulate the fully implicit numerical simulation model of carbon dioxide geological storage established in step 1 into an environment for interaction of intelligent agents in reinforcement learning, and write the initialization function and state transfer function of the environment; the initialization function completes the initialization of the environment by randomly sampling the permeability field and carbon dioxide injection rate, and performs a simulation calculation for one time step to obtain the initial state of the environment; the state transfer function determines the time step of the next time step according to the output of the intelligent agent, completes the simulation calculation for one time step, and outputs the new state vector, reward value and a signal whether the simulation is over; at the beginning of each round, the environment uses step 1.2 to randomly initialize the porosity and permeability fields, and uses step 1.5 to randomly formulate a carbon dioxide injection strategy within a predetermined range; then use the determined initial time step Perform a calculation for one time step and use the result of the calculation to get the state vector of the first time step ;Then the agent continues to interact with the environment. The agent selects an action based on the current state to get the time step of the next time step. The environment updates the state after running a time step and gives a reward value until the total simulation time is completed.

[0031] Step 3.2: Select a reinforcement learning algorithm to train the reinforcement learning agent. The agent repeatedly interacts with the environment and gradually learns the step adjustment strategy. The training samples include different porosities, different permeability fields, different CO2 injection well locations and injection rates, and CO2 storage scenarios. The training goal is to minimize the total number of Newton iterations for each complete numerical simulation while ensuring the convergence of the entire simulation process.

[0032] Furthermore, the specific process of step 4 is as follows:

[0033] Step 4.1: In the actual numerical simulation of CO2 geological storage, select new formation parameters and CO2 injection strategy, initialize the time step, and input the initial state;

[0034] Step 4.2: After each time step is completed, the current state parameters are obtained in real time, including the time step, number of Newton iterations, convergence residual, and rate of change of physical variables;

[0035] Step 4.3: Call the trained reinforcement learning agent, input the current state, output the time step adjustment action, and update the time step of the next time step;

[0036] Step 4.4: Continue the numerical simulation time advancement, and the agent continues to dynamically adjust the time step until the simulation ends.

[0037] The beneficial technical effects brought about by the present invention: The present invention significantly improves the computational efficiency of numerical simulation of carbon dioxide geological storage by introducing reinforcement learning to dynamically adjust the time step, can effectively reduce the total number of Newton iterations and the time step back frequency, and overcomes the problems of rough adjustment and low efficiency of existing heuristic time step control methods. The proposed time step control strategy has intelligent adaptive capabilities, can autonomously optimize step selection according to the real-time numerical simulation status, reduce manual intervention, reduce operational complexity, and improve the automation level of the simulation process. At the same time, the present invention has good generalization ability and robustness after multi-scenario and multi-condition training, is suitable for different geological conditions and complex injection conditions, and can be easily integrated into existing oil and gas reservoir numerical simulator software, with broad prospects for engineering application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a design flow chart of the efficient numerical simulation method for geological storage of carbon dioxide based on reinforcement learning of the present invention.

[0039] Figure 2 Schematic diagram of the reservoir basic grid model in an embodiment of the present invention.

[0040] Figure 3 Schematic diagram of the horizontal permeability field of the reservoir generated in an embodiment of the present invention.

[0041] Figure 4 Schematic diagram of the vertical permeability field of the reservoir generated in an embodiment of the present invention.

[0042] Figure 5 Schematic diagram of the functional relationship between the density and pressure of carbon dioxide and formation water in an embodiment of the present invention.

[0043] Figure 6 Schematic diagram of the functional relationship between the viscosity of carbon dioxide and formation water and pressure in an embodiment of the present invention.

[0044] Figure 7 Schematic diagram of the phase permeability curve of carbon dioxide and formation water in an embodiment of the present invention.

[0045] Figure 8 Schematic diagram of the well location of an injection well in the center of an oil reservoir in an embodiment of the present invention.

[0046] Figure 9 This is a graph showing the benchmark test results without using reinforcement learning in an embodiment of the present invention.

[0047] Figure 10 This is a diagram of the calculation results after applying reinforcement learning to optimize the time step selection in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0049] like Figure 1 The efficient numerical simulation method for carbon dioxide geological storage based on reinforcement learning shown in the figure specifically includes the following steps:

[0050] Step 1: Establish a fully implicit numerical simulation model for CO2 geological storage. The specific process is as follows:

[0051] Step 1.1: Based on the actual CO2 geological storage project and existing data, select an appropriate reservoir numerical simulator and establish a grid model for the simulation. This includes selecting the grid type, determining the number of grid cells, and determining the grid dimensions. If mesh data for the simulation area has been generated in advance using third-party software such as Petrel, this mesh data can be directly imported into the numerical simulator. Numerical simulators can include commercial simulators such as Eclipse, CMG, or tNavigator, or open-source simulators such as MRST and OPM Flow.

[0052] This embodiment takes an open source reservoir model as an example and selects the open source reservoir numerical simulator MRST to establish a grid model for numerical simulation. The dimensions of the open source reservoir model are: 500 meters in length and width, 28 meters in thickness, and 1000 meters in top depth. The simulation area is discretized using a regular orthogonal grid with a dimension of 60×60×7, where the total number of active grids is 18553. The open source grid data is directly imported into the MRST numerical simulator, and the resulting grid model is as follows: Figure 2 As shown, x, y, and z represent coordinate axes in different directions, and the unit is m.

[0053] Step 1.2: Based on existing geological knowledge, select an appropriate algorithm for generating geological parameters such as porosity and permeability, or use an inversion method based on parameter perturbation to randomly generate porosity and permeability fields. Algorithms for generating geological parameters can employ stochastic modeling-based parameter generation methods (such as Gaussian random function simulation and multi-point geostatistics). Inversion methods based on parameter perturbation include principal component analysis, singular value decomposition parameterization, and the Karhunen-Loève expansion method.

[0054] The porosity of the open source reservoir model in this embodiment is 0.2, and the permeability data includes 101 sets of different artificially generated permeability fields. Figure 3 and Figure 4 A generated horizontal permeability field and vertical permeability field are shown respectively, with the unit of md.

[0055] Step 1.3: Generate a fluid model for CO2 and formation fluid based on data such as target reservoir depth and geothermal gradient. The fluid model specifically includes the functional relationship between fluid physical parameters such as density, viscosity, and solubility and pressure, as well as the functional relationship between the fluid relative permeability and capillary force and water saturation.

[0056] In this example, the reservoir burial depth is 1000m. The formation temperature and initial pressure are determined based on the burial depth and the common low-temperature gradient to obtain the corresponding fluid model. The functional relationship between the density of carbon dioxide and formation water and pressure is as follows: Figure 5 As shown in the figure; the functional relationship between the viscosity of carbon dioxide and formation water and pressure is as follows Figure 6 The relative permeability curve of carbon dioxide and formation water (i.e. the functional relationship curve between relative permeability and water saturation) is shown as follows. Figure 7 This embodiment does not take into account the capillary force between carbon dioxide and formation water.

[0057] Step 1.4: Select a mathematical model for CO2 flow and migration based on actual site requirements, such as a CO2-formation water two-phase flow model, a black oil-like model that considers CO2 dissolution in formation water, a component model for multiphase and multicomponent flow, or a coupled physical field model that simultaneously considers physical processes such as capillary pressure, gravity, interphase dissolution, and chemical reactions.

[0058] This embodiment considers the two-phase seepage process of carbon dioxide and formation water, and considers the dissolution of carbon dioxide in formation water. Therefore, it is determined to select the black oil model in the MRST simulator, regard carbon dioxide as the gas phase, and use the oil phase in the model to simulate formation water, thereby simulating the dissolution of carbon dioxide in formation water.

[0059] Step 1.5: Set boundary conditions and initial conditions close to the actual formation conditions and determine different CO2 injection strategies (including injection well location, injection rate, etc.). Set the reservoir to be initially saturated with formation water and set the initial pressure to be the static pressure distribution. Figure 8 As shown. Carbon dioxide is injected into the reservoir at a constant rate. The specific value of the injection rate ranges from [0.0104 0.0208] m 3 / s. In order to simulate the influence of boundary conditions, the pore volume of all grids located on the vertical boundary was multiplied by 1000 times according to the method in the literature "Impact of relative permeability hysteresis on geological CO2 storage (Juanes, R., Spiteri, EJ, Orr Jr, FM, & Blunt, MJ (2006). Impact of relative permeability hysteresis on geologicalCO2 storage. Water resources research, 42(12).)".

[0060] Step 1.6: Based on the grid model, porosity / permeability field data, fluid model, flow and transport mathematical model, and initial and boundary conditions established in the previous steps, the equations of the mathematical model are discretized in space and time using the finite volume numerical method and the inverse Euler method, respectively, to obtain a fully implicit numerical simulation model. Specifically, at each time step, the discretized model is a set of nonlinear equations that can be expressed as follows:

[0061] ;

[0062] in, It is a fully implicit numerical simulation model; 、 Represent the current time step and the previous time step respectively; and They represent the unknown quantities (pressure / saturation, etc.) to be solved in the current time step and the unknown quantities (pressure / saturation, etc.) that have been solved in the previous time step respectively; Indicates the control conditions for going down the well at the current time step; Represents the time step length of the current time step. The Newton-Raphson method is used to iteratively solve the above nonlinear equations to obtain the values ​​of unknown quantities such as pressure and saturation at the current time step.

[0063] Step 2: Build a reinforcement learning environment, including designing the state space, action space, and reward function. The specific process is as follows:

[0064] Step 2.1, state space design; the state vector of the current time step in the state space The time step including the previous time step , the number of Newton iterations in the previous time step , the Newton residual at the current time step , the time step adjustment ratio of the previous time step , and the maximum rate of change of pressure and saturation in the current time step Specific expression is:

[0065] ;

[0066] in, Contains the maximum rate of change of pressure in the current time step and maximum rate of change of saturation .

[0067] Step 2.2, action space design; define the discrete action set of the current time step in the action space , indicating that the agent can choose to reduce or increase the next time step by a specified ratio.

[0068] Step 2.3: Reward function design; the single-step reward function is designed as follows:

[0069] ;

[0070] Where, is the reward value of the current time step; Indicates the number of Newton iterations in the current time step; Indicates the convergence flag of the current time step. The value is 1 when convergence fails and 0 when convergence is normal. represents the convergence failure penalty coefficient, and is recommended to be between 10 and 50. This reward function encourages reducing the number of Newton iterations while penalizing convergence failure. In this embodiment, The value is 20.

[0071] Step 3: Offline training of the reinforcement learning agent to learn the time step control strategy. The specific process is as follows:

[0072] Step 3.1. Based on the design of the state space, action space and reward function in step 2, encapsulate the fully implicit numerical simulation model of carbon dioxide geological storage established in step 1 into an environment for interaction of intelligent agents in reinforcement learning, and write the initialization function and state transfer function of the environment. The initialization function completes the initialization of the environment by randomly sampling the permeability field and the carbon dioxide injection rate, and performs a simulation calculation for one time step to obtain the initial state of the environment; the state transfer function determines the time step of the next time step according to the output of the intelligent agent, completes the simulation calculation for one time step, and outputs the new state vector, reward value and a signal whether the simulation is over. At the beginning of each round, the environment uses step 1.2 to randomly initialize the geological parameter fields such as porosity and permeability fields, and uses step 1.5 to randomly formulate a carbon dioxide injection strategy within a predetermined range. Then use the determined initial time step Perform a calculation for one time step and use the result of the calculation to get the state vector of the first time step The agent then continues to interact with the environment. The agent selects an action based on the current state to obtain the time step of the next time step. The environment updates the state after running a time step and gives a reward value until the total simulation time is completed.

[0073] In step 3.2, a mature and robust reinforcement learning algorithm, such as the Soft-Actor-Critic (SAC) algorithm or the Proximal Policy Optimization (PPO) algorithm, is selected to train a reinforcement learning agent. The agent repeatedly interacts with the environment and gradually learns a step-size adjustment strategy. Training samples include CO2 storage scenarios with varying porosity, permeability, CO2 injection well locations, and injection rates to ensure good generalization of the agent. The training objective is to minimize the total number of Newton iterations for each complete numerical simulation while ensuring convergence of the entire simulation process. The SAC reinforcement learning algorithm was selected for this test case. Specifically, the established SAC reinforcement learning agent consists of an actor network and two critic networks. The actor network uses fully connected layers and Relu activation functions. Its input is the environment state, and its output is the probability distribution of each discrete action. Action sampling is implemented using the rlDiscreteCategoricalActor function in the MATLAB reinforcement learning toolbox. The two critic networks have the same structure: a dual-Q network. Their input is the environment state, and their output is the Q-values ​​of all discrete actions, which is used to reduce the overestimation bias of Q-values. Both dual-Q networks are defined using the rlVectorQValueFunction function in the Reinforcement Learning Toolbox provided by MATLAB. Both the actor and critic networks use the Adam optimizer with independent learning rates and L2 regularization coefficients. The agents use experience replay, batch updates, and adjustable target smoothing coefficients and entropy weights, which are used to encourage policy exploration.

[0074] Step 4: Deploy the trained time step control strategy in the actual numerical simulation to achieve real-time step adjustment. The specific process is as follows:

[0075] Step 4.1: In the actual numerical simulation of CO2 geological storage, select new formation parameters and CO2 injection strategy, initialize the time step, and input the initial state.

[0076] Step 4.2: After each time step is completed, the current state parameters are obtained in real time, including the time step, number of Newton iterations, convergence residual, rate of change of physical variables, etc.

[0077] Step 4.3: Call the trained reinforcement learning agent, input the current state, output the time step adjustment action, and update the time step of the next time step.

[0078] Step 4.4: Continue the numerical simulation time advancement, and the agent continues to dynamically adjust the time step until the simulation ends.

[0079] The entire process described above does not require human intervention, and the step size adjustment is entirely based on the agent's autonomous decision-making.

[0080] In order to demonstrate the feasibility and superiority of the present invention, the following comparative experiments are given.

[0081] The comparative analysis of the final optimization results of the experiment is as follows:

[0082] Figure 9 The following table shows the results of a benchmark without reinforcement learning. As a baseline, without using reinforcement learning to optimize the time step selection, the total simulation time is 50 years, the total number of time steps is 51, the first time step is 30 days, and the remaining 50 time steps are divided equally. Figure 9 The bar graph in shows the time step length and the corresponding number of Newton iterations at each time step.

[0083] Figure 10 The figure shows the calculation results after optimizing the time step selection using reinforcement learning. It can be seen that reinforcement learning dynamically amplifies the time step size in the early stages, rapidly increasing it to a larger step size and maintaining it later, significantly reducing the number of time steps. The number of Newton iterations (red bars) is significantly lower than that of the baseline solution, with most time steps kept to around 4. This results in higher iteration efficiency, a reduction in the overall number of simulation steps, and a significantly better total number of iterations than the unoptimized solution.

[0084] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the scope of protection of the present invention.

Claims

1. An efficient numerical simulation method for carbon dioxide geological storage based on reinforcement learning, characterized in that: The steps include: Step 1: Establish a fully implicit numerical simulation model for CO2 geological storage. The specific process is as follows: Step 1.1: Based on the actual project and existing data of the CO2 geological storage project, select an oil and gas reservoir numerical simulator, import grid data into the numerical simulator, and generate a grid model for numerical simulation; the grid data includes grid type, number of grids, and dimensions; Step 1.2: Based on existing geological knowledge, select a geological parameter generation algorithm or an inversion method based on parameter perturbation to randomly generate porosity and permeability fields; Step 1.3: Generate a fluid model for CO2 and formation fluid based on the target reservoir depth and geothermal gradient data. The fluid model specifically includes the functional relationship between the fluid physical properties and pressure, as well as the functional relationship between the fluid relative permeability and capillary force and water saturation. The physical properties include density, viscosity, and solubility. Step 1.4: Select a mathematical model for CO2 flow and migration based on actual site needs; Step 1.5: Set boundary conditions and initial conditions based on the actual formation conditions, and simultaneously determine different CO2 injection strategies. Also, set the reservoir to initially saturated with formation water, and set the initial pressure to the static pressure distribution. The CO2 injection strategy includes injection well location and injection rate. Step 1.6: Based on the established grid model, porosity and permeability field data, fluid model, flow and transport mathematical model, and initial and boundary conditions, the finite volume numerical method and the inverse Euler method are used to discretize the mathematical model equations in space and time, respectively, to obtain a fully implicit numerical simulation model. The formula of the fully implicit numerical simulation model is: ； in, It is a fully implicit numerical simulation model; 、 Represent the current time step and the previous time step respectively; and They represent the unknowns to be solved in the current time step and the unknowns that have been solved in the previous time step respectively; Indicates the control conditions for going down the well at the current time step; Indicates the time step of the current time step; Step 2: Build a reinforcement learning environment, including designing the state space, action space, and reward function; Step 3: Offline training of the reinforcement learning agent to learn the time step control strategy; the specific process is as follows: Step 3.

1. Based on the design of the state space, action space and reward function in step 2, encapsulate the fully implicit numerical simulation model of carbon dioxide geological storage established in step 1 into an environment for interaction of intelligent agents in reinforcement learning, and write the initialization function and state transfer function of the environment; the initialization function completes the initialization of the environment by randomly sampling the permeability field and carbon dioxide injection rate, and performs a simulation calculation for one time step to obtain the initial state of the environment; the state transfer function determines the time step of the next time step according to the output of the intelligent agent, completes the simulation calculation for one time step, and outputs the new state vector, reward value and a signal whether the simulation is over; at the beginning of each round, the environment uses step 1.2 to randomly initialize the porosity and permeability fields, and uses step 1.5 to randomly formulate a carbon dioxide injection strategy within a predetermined range; then use the determined initial time step Perform a calculation for one time step and use the result of the calculation to get the state vector of the first time step ;Then the agent continues to interact with the environment. The agent selects an action based on the current state to get the time step of the next time step. The environment updates the state after running a time step and gives a reward value until the total simulation time is completed. Step 3.2: Select a reinforcement learning algorithm to train the reinforcement learning agent. The agent repeatedly interacts with the environment and gradually learns the step size adjustment strategy. The training samples include CO2 storage scenarios with different porosities, different permeability fields, and different CO2 injection well locations and injection rates. The training goal is to minimize the total number of Newton iterations for each complete numerical simulation while ensuring convergence of the entire simulation process. Step 4: Deploy the trained time step control strategy in actual numerical simulation to achieve real-time step adjustment.

2. The efficient numerical simulation method for geological storage of carbon dioxide based on reinforcement learning according to claim 1 is characterized in that: The specific process of step 2 is: Step 2.1, state space design; the state vector of the current time step in the state space The time step including the previous time step , the number of Newton iterations in the previous time step , the Newton residual at the current time step , the time step adjustment ratio of the previous time step , and the maximum rate of change of pressure and saturation in the current time step ; Specifically expressed as: ； in, Contains the maximum rate of change of pressure in the current time step and maximum rate of change of saturation ; Step 2.2, action space design; define the discrete action set of the current time step in the action space ; Step 2.3: Reward function design; the single-step reward function is designed as follows: ； Where, is the reward value of the current time step; Indicates the number of Newton iterations in the current time step; Indicates the convergence flag of the current time step. The value is 1 when convergence fails and 0 when convergence is normal. Represents the convergence failure penalty coefficient.

3. The efficient numerical simulation method for geological storage of carbon dioxide based on reinforcement learning according to claim 2 is characterized in that: The specific process of step 4 is as follows: Step 4.1: In the actual numerical simulation of CO2 geological storage, select new formation parameters and CO2 injection strategy, initialize the time step, and input the initial state; Step 4.2: After each time step is completed, the current state parameters are obtained in real time, including the time step, number of Newton iterations, convergence residual, and rate of change of physical variables; Step 4.3: Call the trained reinforcement learning agent, input the current state, output the time step adjustment action, and update the time step of the next time step; Step 4.4: Continue the numerical simulation time advancement, and the agent continues to dynamically adjust the time step until the simulation ends.

Images