Oil field injection-production collaborative optimization method and system fusing physical information reinforcement learning

Abstract

The invention discloses an oil field injection-production collaborative optimization method and system fusing physical information reinforcement learning, and belongs to the crossing field of petroleum engineering and intelligent control. The method comprises the following steps: firstly, collecting oil field historical production data and carrying out normalization preprocessing; secondly, constructing a bidirectional long-short-term memory network based on an attention mechanism as an oil reservoir digital twin environment model for predicting a future production state; meanwhile, a physical information neural network based on a reservoir capillary force-saturation physical limit curve is innovatively introduced to serve as a physical constraint model, and the nonlinear constitutive relation between the water injection driving pressure and the theoretical water holding capacity of the micro-pores is analyzed; further constructing a composite reward function containing physical consistency penalty, and explicitly embedding the physical security boundary into a strategy optimization process of a depth deterministic strategy gradient algorithm; and finally, through interaction of the intelligent agent, the digital twin environment and the physical constraint model, a water injection strategy considering yield maximization and geological safety is output. According to the method, a data driving algorithm and an oil reservoir seepage mechanism are fused, so that the risk that a pure AI model easily generates physical illusion under sparse data and causes non-Darcy flowing water channeling is effectively avoided, and intelligent and safe collaborative optimization of an oil field injection and production system is realized.

Description

Technical Field

[0001] This invention belongs to the field of intelligent oilfield construction and intelligent control technology for complex systems. Specifically, it relates to a closed-loop control method for reservoir development driven by a fusion of data-driven and physical mechanism approaches. Specifically targeting reservoirs in the mid-to-late stages of water injection development, it involves real-time dynamic control and multi-well collaborative optimization of production parameters such as water injection pressure, water injection volume, and fluid production volume. It belongs to the application of deep reinforcement learning in industrial process control, particularly utilizing algorithms to handle optimization problems in continuous action spaces and combining LSTM neural networks to construct a digital twin environment model. It also specifically involves physical information neural network technology in scientific machine learning, specifically embedding the capillary force-saturation relationship from porous media seepage mechanics as a physical constraint into the reward function and decision loop of reinforcement learning. This addresses the problems of poor generalization ability and lack of physical consistency in traditional pure data-driven models under sparse data. Background Technology

[0002] In the mid-to-late stages of oilfield development, water injection is the main means of replenishing formation energy and improving oil recovery. In existing technologies, water injection control mainly relies on manual experience or traditional PID control methods. The manual experience method relies heavily on the personal experience of engineers, is slow to adjust, and is difficult to cope with complex and ever-changing downhole geological conditions. It often leads to excessive water injection causing water channeling, or insufficient water injection causing formation pressure deficit. Traditional PID control: Although it achieves automation, it is a single-input, single-output feedback control, which is difficult to handle the nonlinear coupling relationship between injection and production well networks, and cannot predict and plan for future production status. Purely data-driven AI methods: In recent years, deep reinforcement learning (such as DDPG) has been used for water injection optimization, but existing purely data-driven models have the "black box" defect, often ignoring rock physics mechanisms (such as the seepage law of porous media), and are prone to outputting radical strategies that violate physical common sense (such as causing violent water channeling), which poses a great risk in practical engineering applications. Existing technologies generally lack deep integration of geophysical mechanisms (such as relative permeability curves and capillary force curves) with intelligent algorithms, making it impossible to maximize production while ensuring geological safety. Summary of the Invention

[0003] Based on the shortcomings of the prior art, the main objective of this invention is to provide a collaborative optimization control method for oilfield water injection and production based on physical information reinforcement learning. This method aims to construct a closed-loop control system that deeply integrates data-driven algorithms with reservoir seepage mechanisms. By introducing the reservoir capillary force-saturation characteristic curve, which characterizes the microscopic seepage law of porous media, as prior physical knowledge, a physical constraint model is constructed. Real-time physical consistency constraints are applied to the decision space of the intelligent agent. This mechanism can achieve dynamic optimal control of the injection and production operation system under the premise of strictly ensuring geological safety boundaries (i.e., preventing violent water channeling of non-Darcy flows), ultimately achieving the goals of increasing cumulative oil production, reducing overall water cut, and extending the life cycle of oil wells. To achieve the above objectives, the technical solution adopted by this invention includes the following specific construction modules and implementation steps: This invention constructs a dual-model collaborative drive control architecture of "perception-prediction-verification-decision". This architecture consists of four core modules: Data preprocessing module: used to clean production data and perform normalization mapping of key features; Digital twin environment module: used to simulate reservoir dynamics and predict future production status; Physical constraint module: acting as a "physical safety valve," it calculates the water saturation limit based on rock physics theory; Intelligent decision-making module: Based on a physics-guided reward function, it outputs the optimal water injection strategy.

[0004] S1. Construct and preprocess a multi-dimensional time-series production dataset for the oilfield. Collect historical production data from water injection wells and connected production wells in the oilfield, construct sliding time window samples, and perform feature engineering processing on the data, including: performing magnitude detection and forced normalization on water cut data to map percentage data to the 0-1 interval to prevent numerical explosion; performing quantile-based truncation on water injection data to remove outlier noise; and performing Z-Score standardization on other continuous features. S2. Construct a reservoir digital twin environment model based on attention mechanism. Establish a deep neural network containing a bidirectional long short-term memory network and a multi-head attention mechanism as an environment simulator. The model receives the current state and water injection action, and outputs the predicted oil production and predicted water cut of each oil well at the next moment. The introduction of attention mechanism aims to capture the time step features that have the greatest impact on the current state in the long historical sequence. S3. Construct a physical constraint model for the reservoir capillary force-saturation curve. Use a physical information neural network to pre-train a physical model that can characterize the water-holding capacity of porous media. This model establishes a mapping relationship between injection pressure and theoretical water saturation. Specifically, it includes: mapping the real-time injection pressure of the oilfield to the matrix potential (pF value) in porous media physics; taking the static rock properties of the reservoir as input conditions; and outputting the theoretical equilibrium water saturation of rock pores under the current pressure conditions as a physical safety boundary. S4. Construct a physical-guided composite reward function. Design a composite reward function that includes oil production reward, water content penalty, physical consistency penalty and action smoothness penalty. The physical consistency penalty term is obtained by comparing the actual water content predicted in step two with the theoretical water saturation calculated in step three. When the predicted water content exceeds a certain threshold of the theoretical water saturation, a penalty with dynamic weights is applied to force the agent to find a strategy within the physically permissible range. S5. Based on the improved deep deterministic policy gradient algorithm, a policy optimization is carried out by constructing an agent containing an Actor network and a Critic network. The agent is trained using a priority experience replay mechanism and Ornstein-Uhlenbeck noise. During the training process, the agent interacts with the digital twin environment and the physical oracle. By maximizing the accumulated physical guidance reward, the agent learns the optimal water injection control policy. S6. Online closed-loop control and model co-evolution: The trained Actor network is deployed for real-time water injection control. Simultaneously, the digital twin environment model is periodically fine-tuned using newly generated interactive data to achieve co-evolution between the policy network and the environment model, adapting to the dynamic changes in reservoir development characteristics.

[0005] The beneficial effects of this invention are as follows: it significantly improves the physical interpretability and geological safety of the control strategy; it innovatively introduces a physical oracle based on the reservoir capillary force-saturation physical limit curve, transforming the constitutive relationship of porous media seepage into explicit physical constraints. This effectively overcomes the shortcomings of traditional pure data-driven AI models, which are prone to "physical illusions" and output extreme water injection commands in sparse data regions, ensuring that the operating state of the injection-production system is always strictly limited within the geological and physical safety boundary, fundamentally avoiding the risk of violent water channeling in non-Darcy flows; and it achieves a dynamic and precise balance between production benefits and geological risks through the physical guidance reward function. With a dynamic weight adjustment mechanism, the agent successfully learned an advanced strategy of "maximizing the injection-production ratio within the physical critical safety domain." This strategy avoids both the waste of production capacity caused by excessive conservatism and the fingering phenomenon induced by aggressive water injection, achieving the global optimization of reservoir development economic benefits. A high-fidelity reservoir digital twin training environment was constructed, and a high-precision reservoir digital twin model was built using a bidirectional long short-term memory network with a multi-head attention mechanism. This model accurately captures the lag and nonlinear characteristics of water injection response in the time dimension, providing a high-confidence interactive environment for the reinforcement learning agent's strategy iteration and significantly reducing the cost of on-site trial and error. Attached Figure Description

[0006] Figure 1 This is a flowchart illustrating the overall process architecture of the method of this invention; Figure 2 This is a schematic diagram of the digital twin environment model structure based on LSTM and Attention mechanism in this invention; Figure 3 This is a schematic diagram of the physical mapping and constraint principle of the physical constraint model in this invention; Figure 4 This is a graph showing the total reward convergence curve of the training process in this embodiment of the invention; Figure 5 This is a histogram of agent action distribution in an embodiment of the present invention; Figure 6 This is a reward component decomposition diagram in an embodiment of the present invention (showing the dynamic game between oil production rewards and physical penalties). Detailed Implementation

[0007] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment selects an actual water injection well group (including 1 water injection well and 4 interconnected oil production wells) in an oilfield as the object, and applies the method described in this invention for water injection optimization control. The system operating environment is based on the PyTorch deep learning framework.

[0008] S1. Data Construction and Preprocessing: The system collects daily water injection volume, water injection pressure, and temperature data from water injection wells, as well as daily oil production, water cut, bottom hole flowing pressure, and crude oil saturation data from four oil production wells, forming a 20-dimensional state vector. The time window size is set to 7 days. In the data preprocessing stage, an automatic correction logic is designed to address the potential dimension issues in the water cut data: the maximum value of the water cut column is detected. If the maximum value is greater than 1.0, it is determined to be percentage data and is forcibly divided by 100 to normalize it to the [0,1] interval. This step completely solves the numerical explosion problem caused by the square term of water cut in the subsequent reward calculation. The water injection volume data is truncated using 1% to 99% quantiles to eliminate abnormal fluctuations from the sensors. S2. Reservoir Digital Twin Environment Model: Construct a deep neural network as an environment simulator to replace the real reservoir for low-cost trial and error. a. Model Structure: The input layer receives tensors of dimension (Batch, 7, 20). The backbone network contains two layers of bidirectional LSTM with 256 hidden units, used to extract bidirectional temporal features. The LSTM output is fed into a multi-head attention mechanism to capture the feature weights of key time steps in long sequences. b. Output layer: Eight values ​​are output through the fully connected layer, namely the predicted oil production of four wells and the predicted water cut of four wells; c. Activation function: The Softplus activation function is used to ensure non-negativity for oil production prediction; the Sigmoid activation function is used to strictly limit the water cut prediction to the range of 0-1. d. Training: The mean squared error loss function is used for oil production, and the binary cross-entropy loss function is used for water cut. The two are weighted and summed for backpropagation training. S3. Physical constraint model: A pre-trained physical constraint model based on reservoir capillary force-saturation characteristic curve is used. This model has converged in 200 epochs and has high-precision physical fitting capability. a. Physical Mapping: This invention innovatively maps the "water injection pressure" in petroleum engineering to the "matrix potential (pF)" in soil physics, with the mapping formula set as follows: in, The normalized injection pressure is represented by the formula, which reflects that the greater the injection pressure, the stronger the corresponding driving force, which is equivalent to the smaller the capillary suction (pF), and the more saturated the rock tends to be. b. Inference process: The calculated pF value and geological features representing the static properties of the reservoir (normalized values ​​such as Sand=1.2, Clay=-0.8, etc.) are input into PINN, and the model outputs the theoretical equilibrium water saturation. ; S4. Design of the reward function for physics-guided learning: In the reinforcement learning environment PhysicsGuideEnv, the following composite reward function R is designed: The specific calculation logic is as follows: Oil production incentives ( ): Encourage agents to maximize oil production; water content penalty ( ): Apply a basic penalty to areas with high moisture content; Physical consistency penalty ( This is the core constraint. Calculate the difference between the actual predicted moisture content and the theoretical value. ,like If the value is greater than 0.1 (physical tolerance threshold), a penalty will be triggered. In this embodiment, after comparative experiments (weight 5.0 leads to conservatism, 2.0 leads to oscillation), the optimal physical weight is 1.0. Under this weight, the system achieves the best balance between safety and efficiency, and smooth action penalty (…). ): Used to suppress drastic valve fluctuations and protect equipment; S5. Agent training and control strategy: Construct an improved DDPG agent. Both the Actor and Critic networks use LayerNorm normalization layers to accelerate convergence. During training, a priority experience replay mechanism is introduced to prioritize learning samples with larger TD errors. In the training loop, the agent outputs actions, the LSTM environment predicts the next state, and the physical constraint model calculates the physical boundary and applies constraints to the reward function. At the same time, every 100 episodes, the LSTM predictor is fine-tuned using newly generated interaction data to prevent the environment model from drifting. S6. Implementation effect analysis, such as Figures 4 to 6 As shown in the image (corresponding to the training results provided by the user), after 1000 training rounds: a. Total reward curve ( Figure 4 The model exhibits a clear step-like rise and eventually converges to a high plateau (approximately -294.6), proving that the model has effectively learned the optimal policy. b. Action distribution ( Figure 5The pattern exhibits a unique "asymmetric bimodal" shape. The main peak is located in the 0.2-0.5 range, indicating that the agent adopts a mild water injection strategy to maintain production most of the time; a secondary peak exists at the extreme value on the left (-1.2), indicating that the agent has learned to perform a "sudden stop" well shutdown when the physical constraint model warns of high risk. c. Reward components ( Figure 6 The physical penalty term (orange line) exhibits a regular sawtooth oscillation, indicating that the agent always performs precise operations near the physical safety boundary, maximizing the formation's oil production potential while avoiding continuous violations. In summary, this invention overcomes the limitations of the separation between "physical mechanism" and "data-driven" approaches in traditional oilfield development control. By innovatively introducing a physical information neural network based on the reservoir capillary force-saturation characteristic curve, which characterizes the microscopic seepage law of porous media, as a physical constraint model, and deeply coupling the physical consistency constraint of porous media seepage mechanics to the reward function and decision loop of the deep deterministic strategy gradient, this invention successfully endows the black-box agent with the ability to perform strategy optimization within the geological safety boundary. This not only fundamentally solves the problem of "physical illusion" and numerical instability easily generated by pure data-driven models under sparse data, but also achieves a dynamic and precise balance between "maximizing oil production benefits" and "minimizing water channeling risk" in complex injection-production nonlinear coupling environments. This provides a new solution for water injection development in smart oilfields that combines physical interpretability, operational robustness, and decision-making accuracy.

Claims

1. A method and system for coordinated optimization of oilfield injection and production integrating physical information reinforcement learning, characterized in that, Includes the following steps: Step S1: Construct a digital twin environment model of the oil reservoir and establish a time series prediction model based on a deep neural network. The model is used to receive the current multidimensional state feature vector of the oilfield and the water injection action command, and to deduce and predict the oil production and water cut values ​​of each connected oil well at the next moment. Step S2: Use physical information neural network to learn and construct a physical constraint model based on reservoir capillary force-saturation curve, and calculate the theoretically reasonable upper limit of water saturation under the current state under a given injection pressure. Step S3: Construct a physically guided composite reward function. The reward function includes an oil production reward term that encourages increased production, a water cut penalty term that inhibits water cut, and a physical consistency penalty term. The physical consistency penalty term is dynamically generated by calculating the deviation between the predicted water cut derived in step S1 and the theoretical equilibrium water saturation boundary output in step S2. Step S4: Construct a deep reinforcement learning agent. Using the digital model in Step S1 and the physical information neural network model in Step S2 as a virtual environment, automatically learn a set of optimal strategies that produce high oil and prevent water channeling. Apply this strategy to the control terminal at the oilfield site to automatically adjust the water injection volume of the water injection equipment and achieve automatic closed-loop control.

2. The method according to claim 1, characterized in that, In step S2, the construction process of the physical constraint model and the physical mapping logic specifically include: A reservoir capillary force-saturation model of micro-seepage mechanism under porous media is established using physical information neural network. The model is constrained by capillary force curve measured by rock physics experiments. This curve represents the nonlinear relationship between the critical displacement pressure required for non-wet phase fluid to be displaced by wet phase fluid and the pore fluid saturation under a specific pore structure. A conversion formula between water injection pressure and matrix potential is set up to map the normalized water injection pressure of the oilfield into the matrix potential (pF value) in porous media physics. This conversion formula characterizes the negative correlation physical mechanism between macroscopic water injection driving force and microscopic capillary suction force. That is, the higher the water injection pressure, the lower the corresponding matrix potential, and the stronger the tendency of rock pores to be displaced by water and reach saturation. The calculated matrix potential and the geological feature vector representing the static properties of the reservoir are input into the physical information neural network. The geological feature vector includes at least the normalized values ​​of sand content, clay content and rock bulk density. The theoretical equilibrium water saturation boundary corresponding to the output of the physical information neural network represents the upper limit of water saturation that can be physically allowed in the formation pores under the current injection pressure and without the occurrence of violent water channeling without Darcy flow.

3. The method according to claim 2, characterized in that, In step S3, the specific calculation logic for the physical consistency penalty term is as follows: Calculate the difference between the predicted water content obtained from step S1 and the theoretical equilibrium water saturation boundary output from step S2; Set a physical tolerance threshold to allow for small fluctuations within the model's prediction error range; Determine whether the difference exceeds the physical tolerance threshold; If the physical tolerance threshold is exceeded, it is determined that the current water injection action has caused a non-physical risk of fluid fingering or water crossing along the large channel, and a dynamic penalty value that is positively correlated with the difference is applied. The dynamic penalty value is calculated in an exponential growth or linear amplification manner to force the agent's strategy to return to the physical safety boundary. If the physical tolerance threshold is not exceeded, the current state is determined to conform to the physical laws of porous media seepage, and the physical consistency penalty term is zero.

4. The method according to claim 1, characterized in that, In step S1, the reservoir digital twin environment model adopts a deep neural network architecture that includes a bidirectional long short-term memory network and a multi-head attention mechanism; The bidirectional long short-term memory network is used to simultaneously extract the forward and backward temporal dependency features of historical oilfield production data; The multi-head attention mechanism layer is connected after the bidirectional long short-term memory network and is used to calculate the attention weight of each time step to the current state within the historical time window, so as to capture the lag characteristics of the water injection response in the time dimension. The output layer of the model contains specific activation functions, where the oil production prediction uses the Softplus activation function to ensure non-negativity, and the water cut prediction uses the Sigmoid activation function to limit it to the range of 0 to 1.

5. The method according to claim 1, characterized in that, In step S4, the deep reinforcement learning agent employs an improved deep deterministic policy gradient algorithm, which includes a policy network and a value network. The strategy network is used to directly output continuous water injection action values ​​based on the current state; The value network is used to evaluate the Q-value of state-action pairs; Step S4 further includes a co-evolution mechanism: while the agent updates the strategy network parameters, a specific update cycle is set; whenever the update cycle is reached, newly generated real interaction data during the online closed-loop control process is collected; the new data is used to fine-tune the reservoir digital twin environment model in step S1 to correct the prediction bias of the environment model when exploring unknown areas, so as to realize the synchronous evolution of the control strategy and the environment model.

6. The method according to claim 4, characterized in that, In step S3, the formula for calculating the physical guidance composite reward function is as follows: Where R represents the total reward. As an incentive for oil production, As a water-containing penalty, Penalty for physical consistency For smooth action penalty; the oil production reward The water content penalty is proportional to the average predicted oil production; The penalty is proportional to the square of the predicted moisture content, thus increasing the severity of the penalty for high moisture content conditions; the penalty is smoothed out. It is proportional to the absolute value or square of the difference between the current action value and the action value at the previous moment.

7. The method according to claim 1, characterized in that, The method also includes steps for magnitude restoration and preprocessing of the input data: Before inputting data into the model, the numerical range of moisture content characteristic data is automatically detected; If the maximum value of the moisture content data is detected to be greater than 1.0, the data is determined to be in percentage format, and a division by 100 is forcibly performed to normalize it to the numerical range of [0,1], so as to match the output dimension of the physical constraint model and prevent numerical explosion when calculating the reward function. Historical water injection data is bidirectionally truncated using a preset quantile threshold to remove sensor noise and outliers. Z-Score standardization was performed on continuous feature data other than moisture content.

8. A collaborative optimization control system for oilfield water injection and oil production based on physical information reinforcement learning, used to implement the method described in any one of claims 1 to 6, characterized in that, include: The digital twin module contains a pre-trained Bi-LSTM neural network, which is used to simulate reservoir dynamics and extrapolate future production indicators based on historical production sequences. The physical constraint module includes a pre-trained PINN neural network, which is used to calculate the theoretical water saturation boundary based on the reservoir capillary force-saturation physical limit curve principle according to the real-time water injection pressure. The intelligent decision-making and control module includes an Actor network and a Critic network, which are used to calculate the optimal water injection action based on the current state and a composite reward function that includes a physical consistency penalty. The module is also equipped with a communication interface, which is used to convert the calculated optimal water injection action into control commands for the on-site water injection pump or regulating valve and send them to the actuator.

9. An electronic device, characterized in that, The method includes a processor and a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 6.

Images