Intelligent optimization method for multi-type well seam joint control fine injection-production mode

Abstract

The invention discloses an intelligent optimization method for a multi-type well seam joint control fine injection-production mode, and relates to the technical field of oil-gas field development. The method comprises the following steps: setting a well seam joint control fine injection-production mode, establishing an oil reservoir numerical simulation model in oil reservoir numerical simulation software, obtaining multiple groups of oil reservoir injection-production schemes based on a Latin hypercube sampling method, performing simulation according to each group of oil reservoir injection-production schemes by utilizing the oil reservoir numerical simulation model, generating multiple pieces of sample data, and establishing a sample database; a deep learning agent model is established, after the sample database is utilized to train and train the deep learning agent model, a particle swarm optimization algorithm is adopted to carry out single-target pre-search global optimization to obtain a preferred reference strategy, a reinforcement learning dynamic decision model is established, and a reinforcement learning agent is obtained through training based on a PPO near-end strategy optimization algorithm; and the optimal injection-production development scheme of the oil reservoir is obtained by utilizing the reinforcement learning agent, so that rapid optimization and decision support of the oil reservoir injection-production scheme in a new multi-type well seam joint control mode are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of oil and gas field development, in particular to a multi-type well-fracture joint control fine injection-production mode intelligent optimization method. BACKGROUND

[0002] Unconventional oil and gas resources are rich and have great development potential, and are an important replacement force to ensure oil and gas energy security, but they are limited by poor reservoir physical properties and low energy, and need to rely on large-scale fracturing to improve oil and gas seepage conditions and realize effective development of unconventional oil and gas.

[0003] Considering that the post-fracturing reservoir matrix-fracture heterogeneity is significantly enhanced, the global adaptability of the traditional macroscopic injection-production well pattern is poor, and the injected fluid is difficult to direct sweep, resulting in that a large amount of remaining oil between wells and fracture segments is difficult to be produced, and the water injection sweep efficiency is poor, which greatly limits the post-fracturing stimulation effect of the reservoir. The well-fracture joint control method realizes the directional guidance and fine regulation and control of the displacement flow field in the heterogeneous reservoir by independently controlling the injection and production properties of the fractures of each segment of the horizontal well, and accurately produces the remaining oil between wells and fracture segments, and improves the sweep efficiency.

[0004] The asynchronous injection-production scheme between well segments and the staggered injection-production scheme between well segments as two new well-fracture joint control fine injection-production modes can effectively expand the swept range of the injected fluid, but the actual application effect depends on the design of a reasonable injection-production scheme, however, the traditional numerical simulation method for injection-production scheme optimization has limitations such as long operation time and low efficiency, and the calculation cost of screening the optimal injection-production scheme through massive simulation is huge, and at present, the injection-production scheme optimization method for the well-fracture joint control scene is not mature.

[0005] Therefore, it is urgent to propose a multi-type well-fracture joint control fine injection-production mode intelligent optimization method with high efficiency and accuracy and certain self-adaptive ability to realize the rapid optimization and decision support of the injection-production scheme under the multi-type well-fracture joint control new mode. SUMMARY

[0006] The present application aims to solve the above problems, and proposes a multi-type well-fracture joint control fine injection-production mode intelligent optimization method, which designs the well-fracture joint control fine injection-production mode of the reservoir, establishes an intelligent algorithm coupled with single-target pre-search and reinforcement learning, dynamically fine-tunes the benchmark injection-production system scheme of the reservoir, obtains the optimal injection-production development scheme of the reservoir, realizes the rapid optimization and decision support of the injection-production scheme of the reservoir under the multi-type well-fracture joint control new mode, and provides technical support for the development strategy of the actual reservoir.

[0007] The present application adopts the following technical scheme: A multi-type well-fracture joint control fine injection-production mode intelligent optimization method, comprising the following steps: Step 1: Set up a well-fracture joint control fine injection and production mode. Combine reservoir physical property parameters and horizontal well design parameters to establish a reservoir numerical simulation model in the reservoir numerical simulation software to simulate the injection and production process of the fractured horizontal well. Step 2: Set the core sampling parameters and their ranges. Based on the Latin hypercube sampling method, obtain multiple reservoir injection and production schemes. Use the reservoir numerical simulation model to simulate each reservoir injection and production scheme to obtain the state, action and reward of the reservoir numerical simulation model at each time step when each reservoir injection and production scheme is adopted. Generate multiple sample data and establish a sample database. Step 3: Build a deep learning agent model based on the LSTM network structure, and train and validate the deep learning agent model using a sample database to obtain the validated deep learning agent model. Step 4: Replace the reservoir numerical simulation model with the validated deep learning surrogate model, and use the particle swarm optimization algorithm for single-objective pre-search global optimization. Determine the optimal benchmark strategy through fast search to obtain the benchmark injection-production scheme. Step 5: Establish a reinforcement learning dynamic decision-making model based on the Actor-Critic dual network structure of policy gradient, train the reinforcement learning dynamic decision-making model based on the PPO proximal policy optimization algorithm, and obtain a reinforcement learning agent for fine-tuning the benchmark injection and extraction system scheme. Step 6: Obtain the initial formation state, total development time, and time step of the actual reservoir. Input the benchmark injection-production regime scheme obtained in Step 4 into the reinforcement learning agent. Use the reinforcement learning agent to obtain the injection flow rate and production pressure of all fractures in the actual reservoir at all time steps within the total development time. Generate an intelligent injection-production regime table containing the injection flow rate sequence and production pressure sequence to obtain the optimal injection-production development scheme for the actual reservoir.

[0008] Preferably, in step 1, the well-fracture joint control precision injection and production mode includes an asynchronous injection and production scheme between the same well section and an interleaved injection and production scheme between different well sections. The asynchronous injection and production scheme between the same well section is designed for a single fractured horizontal well, and the injection and production development of the fractured horizontal well is carried out by alternately arranging production fractures and injection fractures in the horizontal well section of the single fractured horizontal well. The interleaved injection and production scheme between different well sections is designed for multiple fractured horizontal wells, and the injection and production development of the fractured horizontal well is carried out by alternately setting fractures between the same well sections of two adjacent fractured horizontal wells.

[0009] Preferably, step 2 includes the following sub-steps: Step 2.1: Based on the actual working conditions of the reservoir, set the total development time and initial formation pressure of the reservoir numerical simulation model, and set the core sampling parameters and their value ranges. The core sampling parameters include the mean and fluctuation range of the injected fracture flow rate, the mean and fluctuation range of the produced fracture pressure, and the adjustment frequency. Step 2.2: Based on the Latin hypercube sampling method, hierarchical sampling is performed in the parameter space of each core sampling parameter to obtain multiple sets of non-overlapping core sampling parameters. According to the preset threshold judgment rule, the collected core sampling parameters are converted into injection and sampling parameters to obtain multiple injection and sampling schemes. Step 2.3: Based on each injection-production scheme and the preset time step, simulate the reservoir using the reservoir numerical simulation model in the reservoir numerical simulator to obtain the reservoir state, actions, and rewards obtained from the reservoir numerical simulation model at each time step, and then... The state of the reservoir at each time step ,action and rewards and the The state of the reservoir at each time step As sample data, the state is set as the average pressure and water saturation around the fracture, the action is set as the average injection volume and average production pressure of the fracture, and the reward is set as the step-by-step net present value. Step 2.4: Preprocess the sample data obtained from the simulation. First, remove the sample data that failed the simulation and contained outliers. Then, use the Z-Score standardization method to normalize the sample data. Randomly allocate the normalized sample data to the training set and the validation set to construct the sample database.

[0010] Preferably, step 3 includes the following sub-steps: Step 3.1: Establish a deep learning agent model based on the LSTM network structure, and set the maximum number of iterations and loss function of the deep learning agent model; Step 3.2: Randomly select multiple sample data from the training set and input them into the deep learning proxy model. Train the deep learning proxy model to predict the production index of the reservoir numerical simulation model at the current moment and the state of the reservoir numerical simulation model at the next moment based on the state and action of the reservoir numerical simulation model in the sample data. Calculate the loss function value of the deep learning proxy model during the training process. Step 3.3: If the current iteration count has reached the maximum number of iterations, stop training the deep learning proxy model and proceed to step 3.4. Otherwise, compare the calculated loss function value with the preset loss function value. If the calculated loss function value exceeds the preset loss function value, adjust the hyperparameters of the deep learning proxy model and return to step 3.2 to continue training the deep learning proxy model. If the calculated loss function value does not exceed the preset loss function value, stop training the deep learning proxy model and proceed to step 3.4. Step 3.4: Randomly select multiple sample data from the test set and input them into the trained deep learning agent model. Evaluate the performance of the trained deep learning agent model based on the coefficient of determination and the mean absolute percentage error. If the coefficient of determination of the trained deep learning agent model is greater than the preset coefficient of determination value and the mean absolute percentage error is less than the preset error value, proceed to step 3.5. Otherwise, adjust the hyperparameters of the deep learning agent model and return to step 3.2 to continue training the deep learning agent model. Step 3.5: Output the validated deep learning agent model.

[0011] Preferably, the deep learning proxy model is built based on an LSTM network structure, including an input layer, a hidden layer, a fully connected layer, and an output layer. The input layer is used to input the state and actions of the reservoir numerical simulation model at the current time step. The hidden layer uses multiple stacked LSTM units to learn the pressure propagation characteristics and saturation front advancement characteristics during reservoir seepage through a gating mechanism. The fully connected layer is set between the hidden layer and the output layer to map the high-dimensional feature data output by the hidden layer to the physical space. The output layer is used to output the predicted state of the reservoir numerical simulation model at the next time step and the production indicators at the current time step. The mapping function of the deep learning agent model is: ; In the formula, For the first Predicted values ​​of reservoir state at each time step; , , The first Predicted production indicators, status, and actions of the reservoir at each time step; LSTM network weight parameters The proxy model mapping function; The hidden state memory passed to the LSTM network; The loss function of the deep learning agent model is: ; In the formula, For the loss function of the deep learning agent model; These are the weight parameters of the LSTM network; The sample data sequence number; The total number of sample data used in training; , These are all weighting coefficients of the loss function; , The first The reservoir status and production indicators at each time step; For the first Predicted values ​​of reservoir production indicators at each time step; It is an L2 norm.

[0012] Preferably, step 4 includes the following sub-steps: Step 4.1 sets maximizing the cumulative net present value over the entire cycle as the objective function of the pre-search process, expressed as: ; in, ; In the formula, It is a function for maximizing the value; The objective function is... This refers to the time step number; This represents the total number of time steps. For the first The stepwise net present value of the reservoir at each time step; , , These are crude oil price, water injection cost, and produced water treatment cost, respectively. , , The first The cumulative oil production, cumulative water injection, and water production of the reservoir at each time step; The annual discount rate; For time step; Using production regime parameters, including injection volume and production pressure, as optimization variables, the deep learning surrogate model is set with the following parameters: The crack in the first The actions at each time step are: ; In the formula, The crack number; For the first The crack in the first Injected traffic at each time step; For the first The crack in the first The extraction pressure at each time step; For the first The average injection flow rate of the crack; For the first Average bottom hole flowing pressure of the fracture; For the first The amplitude of the crack fluctuation; It is a sine function; Step 4.2: Set the initial parameters of the particle swarm optimization algorithm, including the total number of particles, the maximum number of iterations, the inertia weight, and the learning factor; Step 4.3: Using the particle swarm to represent the injection and collection regime scheme, the position and velocity of each particle in the particle swarm are randomly set. The velocity of the particle is the rate of change of the particle position, which is used to represent the magnitude of the injection and collection parameter adjustment. The position of the particle is the injection and collection regime parameter. For each particle in the particle swarm, the injection-extraction scheme represented by the particle is input into the validated deep learning proxy model for prediction, so as to obtain the production indicators and cumulative net present value corresponding to the injection-extraction scheme, and use them as the fitness value of the particle. Step 4.4: Update the optimal value of each particle, making the particle with the highest fitness the global optimal particle, and update the velocity and position of each particle in the particle swarm; Step 4.5: Determine whether the current iteration count has reached the preset maximum iteration count. If it has, proceed to Step 4.6; otherwise, update the iteration count and return to Step 4.4 to continue particle swarm optimization. Step 4.6: Output the injection and extraction regime scheme corresponding to the globally optimal particle as the preferred benchmark strategy to obtain the benchmark injection and extraction regime scheme.

[0013] Preferably, step 5 includes the following sub-steps: Step 5.1: Establish a reinforcement learning dynamic decision-making model based on the Actor-Critic dual network structure of policy gradient, and connect the reinforcement learning dynamic decision-making model and the validated deep learning agent model based on the Markov decision process. Use the reinforcement learning dynamic decision-making model to predict the cumulative net present value of the reservoir within a specified time in the future based on the current injection and production scheme. Step 5.2: Train the reinforcement learning dynamic decision-making model using the PPO (Proximity Policy Optimization) algorithm, and update the network parameters of the reinforcement learning dynamic decision-making model. Step 5.3: Save the network parameters of the reinforcement learning dynamic decision-making model to obtain the reinforcement learning agent.

[0014] Preferably, the reinforcement learning dynamic decision-making model is constructed based on an Actor-Critic dual-network structure with policy gradients, including a policy network and a value network. The policy network is used to adjust the injection-production regime, including an input layer, a fully connected hidden layer, and an output layer connected in sequence. It is used to determine the normal distribution function of the adjustment coefficient based on the current state of the reservoir, and the adjustment coefficient is determined by random sampling. The value network is used to evaluate the merits of the injection-production regime adjustment, including an input layer, a fully connected hidden layer, and an output layer connected in sequence. It is used to determine the state value function based on the current state of the reservoir, and the value function is calculated to guide the policy network update. The state value function is used to predict the cumulative net present value of the reservoir over a specified future period based on the current reservoir state and the injection-production scheme. The expression is: ; In the formula, The state value function; For the first The state of the reservoir at each time step; For the first The state of the reservoir at each time step; Expected based on experience; Discount factor; For the first The reward for each time step of the reservoir; The Markov Decision Process (MDP) is used for parameter passing in reinforcement learning dynamic decision-making models and deep learning agent models, by establishing a continuous state-action MDP quintuple. ,in, This represents the state space, which serves as the input to the reinforcement learning dynamic decision-making model. This is the action space, used to represent adjustments to the injection and extraction system; This is a state transition model, responsible for receiving the current state of the reservoir and the adjusted actions, and calculating the state of the reservoir in the next time step. For the reward function; This is the discount factor.

[0015] Preferably, step 5.2 includes the following sub-steps: Step 5.2.1: Initialize the PPO proximal policy optimization algorithm parameters, setting the learning rate, truncation parameter, discount factor, generalized advantage estimation parameter, maximum number of training epochs, and composite total loss function; Step 5.2.2: During iterative training, the policy network of the reinforcement learning dynamic decision-making model is first used to identify the current state of the reservoir and output adjustment coefficients. The adjusted injection-production scheme is then input into the validated deep learning agent model. The validated deep learning agent model is used to obtain the reward for the current time step and the reservoir state for the next time step, acquiring a reservoir database composed of reservoir data from all time steps. The data in the reservoir database are then input into the reinforcement learning dynamic decision-making model. The value network of the reinforcement learning dynamic decision-making model outputs the state value function for each time step, determining the reward and advantage function of the reservoir at each time step. The calculation formula is as follows: ; ; In the formula, For the first The return on investment for a reservoir at a given time step; For the first The reward for each time step of the reservoir; Discount factor; For the first The return on investment for a reservoir at a given time step; For the first The dominant function of a reservoir at each time step; For summation index; This represents the total number of time steps. This refers to the time step number; For parameters to estimate the generalized dominance; For the first The reward for each time step of the reservoir; For the first The state of the reservoir at each time step; For the first The state of the reservoir at each time step; Step 5.2.3: Based on the gradient optimization algorithm, calculate the gradient of the composite total loss function value relative to the policy network weight parameters and the value network weight parameters, and update the network parameters of the reinforcement learning dynamic decision model by combining the preset learning rate. Step 5.2.4: Continue iterative calculations until the current number of iterations reaches the preset maximum number of training rounds or the preset reward convergence condition is met, then end the training of the reinforcement learning dynamic decision model.

[0016] Preferably, the composite total loss function is set as follows: ; in, ; ; ; In the formula, This is the composite total loss function; To maximize the strategy's reward function; The policy entropy function; , All are weighting coefficients of the composite total loss function; These are the policy network weight parameters; These are the weight parameters of the value network; The current injection / collection strategy is determined by the strategy network weight parameters; The predicted value is the expected value based on experience; It is a minimum value function; The new and old injection and extraction system schemes were proposed in the [number]th [year]. Select an action within a time step The probability ratio; The dominant function; This is a truncation function; To truncate parameters; The value network loss function; This is an estimate of the state value; For the first The state of the reservoir at each time step; For the first The return on investment for a reservoir at a given time step; The crack number; To optimize the total number of cracks; It is a natural constant; For the first The standard deviation of the crack action; It is a logarithmic function with base 10.

[0017] The present invention has the following beneficial effects: (1) This invention proposes an intelligent optimization method for fine injection and production mode with multi-type well fracture joint control. Based on the Latin hypercube sampling and physical constraint mapping method, high-quality sample data is obtained, a sample database is constructed, and the sample data in the sample database is used to train and verify a deep learning proxy model based on LSTM network structure. The deep learning proxy model is used to replace the reservoir numerical simulation model to obtain simulation results, which effectively solves the problem that traditional numerical simulation methods are time-consuming and difficult to support high-frequency iterative optimization.

[0018] Meanwhile, the method of this invention introduces physical boundary constraints to ensure the generalization ability of the data-driven model to complex working conditions. It uses the unique gating mechanism of the LSTM network structure to capture the temporal characteristics of reservoir pressure transmission and material balance, reducing the time cost of simulating a single injection and production scheme. This provides an efficient and high-fidelity environment for realizing large-scale, long-cycle intelligent injection and production optimization, and significantly improves the optimization efficiency of injection and production schemes.

[0019] (2) This invention proposes an intelligent optimization method for a multi-type fracture joint control fine injection-production mode, and establishes an intelligent optimization algorithm framework coupled with single-objective pre-search and reinforcement learning. Addressing the strong heterogeneity and dynamic interference problems existing in multiple fractures and between multiple wells, it solves the difficulties of single heuristic algorithms in handling complex time-varying reservoir conditions, and the challenges of slow startup and training convergence when directly applying reinforcement learning for reservoir injection-production system optimization. This method rapidly searches for the optimal benchmark strategy globally through single-objective pre-search, providing high-quality initial values ​​for the reinforcement learning dynamic decision model. By introducing a residual control mechanism into the reinforcement learning of the dynamic decision model, the optimal benchmark strategy is dynamically fine-tuned based on the real-time formation pressure and water saturation state, resulting in the optimal injection-production development scheme for the reservoir.

[0020] The method of this invention not only ensures the global optimization of the injection and production system from a macroscopic perspective, but also has the ability to adjust in real time to deal with sudden reservoir conditions such as local water channeling and pressure exhaustion. It enables rapid optimization and decision support of injection and production schemes under the new model of multi-type well and fracture joint control, effectively improving the optimization efficiency of reservoir injection and production development schemes. Attached Figure Description

[0021] Figure 1 This is a flowchart of an intelligent optimization method for a multi-type well fracture joint control fine injection and production mode according to the present invention.

[0022] Figure 2 This is a schematic diagram of the asynchronous injection and production scheme between the same well sections according to the present invention.

[0023] Figure 3 This is a schematic diagram of the staggered injection and production scheme between different well sections according to the present invention.

[0024] Figure 4 This is the relative permeability curve of oil and water.

[0025] Figure 5 This is a table showing the dynamic injection and production work schedule for the asynchronous injection and production scheme between the same well sections in this invention.

[0026] Figure 6 The figure shows the reservoir parameter variation curves based on the asynchronous injection and production scheme between the same well sections. In the figure, (a) is the curve of cumulative oil production of the reservoir over time, (b) is the curve of cumulative water production of the reservoir over time, (c) is the curve of water cut of the reservoir over time, and (d) is the curve of net present value of the reservoir over time.

[0027] Figure 7 The table shows the dynamic injection and production work schedule for the staggered injection and production scheme between different well sections of the present invention. In the figure, (a) is the dynamic injection and production work schedule for the first horizontal well in the staggered injection and production scheme between different well sections, and (b) is the dynamic injection and production work schedule for the second horizontal well in the staggered injection and production scheme between different well sections.

[0028] Figure 8 The figure shows the reservoir parameter variation curves based on the staggered injection and production scheme between different well sections. In the figure, (a) is the curve of cumulative oil production of the reservoir over time, (b) is the curve of cumulative water production of the reservoir over time, (c) is the curve of water cut of the reservoir over time, and (d) is the curve of net present value of the reservoir over time. Detailed Implementation

[0029] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0030] This invention proposes an intelligent optimization method for multi-type well fracture joint control and fine injection-production mode, such as... Figure 1 As shown, this method is mainly used for intelligent design of injection and production schemes in fractured horizontal wells. By coupling single-objective pre-search with reinforcement learning, the injection and production operation system is dynamically controlled and optimized. Specifically, it includes the following steps: Step 1: Set up a well-fracture joint control fine injection and production mode. Combine reservoir physical parameters and horizontal well design parameters to establish a reservoir numerical simulation model in the reservoir numerical simulation software to simulate the injection and production process of the fractured horizontal well.

[0031] Specifically, to improve the sweep efficiency of injected fluids after reservoir fracturing and enhance the production properties of traditional single fractures, this embodiment sets up two well-fracture joint control precision injection and production modes: an asynchronous injection and production scheme between the same well section and a staggered injection and production scheme between different well sections. The asynchronous injection and production scheme between the same well section is designed for a single fracturing horizontal well, such as... Figure 2 As shown, fracturing horizontal well injection-production development is carried out by alternately arranging production fractures and injection fractures in the horizontal section of a single fracturing horizontal well; the staggered injection-production scheme between different well sections is designed for multiple fracturing horizontal wells, such as... Figure 3 As shown, horizontal well injection and production development is carried out by staggering fractures between the same well sections of two adjacent fractured horizontal wells.

[0032] In this embodiment, a reservoir numerical simulation model is established in the reservoir numerical simulation software based on the asynchronous injection-production scheme between the same well sections. The size of the reservoir numerical simulation model is set to... , set the grid cell size to The length of the horizontal well model in the reservoir numerical simulation model is set to The crack spacing is set to Crack half length set The initial pressure in the basic physical properties of the formation is set to Porosity set to Permeability set to The initial water saturation is set to 0.2. The relative permeability data used in the fluid model of the reservoir numerical simulation model are as follows: Figure 4 As shown.

[0033] A reservoir numerical simulation model is established in the reservoir numerical simulation software based on the staggered injection and production scheme between different well sections. The size of the reservoir numerical simulation model is set as follows: , set the grid cell size to Two horizontal well models were established in the reservoir numerical simulation model, namely the first horizontal well model and the second horizontal well model. The length of the horizontal well models was set to... The crack spacing is set to The spacing between fractures in the horizontal direction of the horizontal well model is set as follows: The fracture spacing at the fracture tip along the vertical direction of the horizontal well model is set to... Crack half length set The initial pressure in the basic physical properties of the formation is set to Porosity set to Permeability set to The initial water saturation is set to 0.2. The relative permeability data used in the fluid model of the reservoir numerical simulation model are as follows: Figure 4 As shown.

[0034] Step 2: Set the core sampling parameters and their value ranges. Based on the Latin hypercube sampling method, obtain multiple reservoir injection-production schemes. Use a reservoir numerical simulation model to simulate each scheme, obtaining the state, actions, and rewards of the reservoir numerical simulation model at each time step when using each scheme. Generate multiple sample data points and establish a representative sample database that fully covers the parameter space. This includes the following sub-steps: Step 2.1: Based on the actual working conditions of the reservoir, set the total development time for the reservoir numerical simulation model. and initial formation pressure Set the core sampling parameters and their value range, wherein the core sampling parameters include the average value of the injected fracture flow rate. With fluctuation range Mean value of extracted fracture pressure With fluctuation range Adjustment frequency The adjustment frequency The average injection fracture flow rate is set as the reciprocal of the total number of days in the injection-production regime. Fluctuation range of injected fracture flow Average pressure of produced fractures Range of pressure fluctuations in the produced fracture and adjustment frequency .

[0035] Step 2.2: Based on the Latin hypercube sampling method, hierarchical sampling is performed in the parameter space of each core sampling parameter to obtain multiple sets of non-overlapping core sampling parameters. The collected produced fracture pressure, injected fracture flow rate and time are judged according to the preset threshold judgment rules. The collected core sampling parameters are transformed into injection and production parameters that meet the actual requirements of the field, and multiple injection and production schemes are obtained.

[0036] In this embodiment, the threshold determination rule is set as follows: ; In the formula, The collected pressure from the extracted fracture; The collected injection fracture flow rate; For time; Total development time.

[0037] In the threshold judgment rule, when the extracted fracture pressure and injected fracture flow rate have no negative values, the values ​​of the extracted fracture pressure and injected fracture flow rate are set to 0. In actual engineering, the corresponding fracture is in a closed state and no injection or extraction is performed. At the same time, the extracted fracture pressure is limited to not exceeding the initial formation pressure, and the total duration of the injection and production scheme constructed by sampling does not exceed the total development time.

[0038] Step 2.3: Based on each injection and extraction plan, and in conjunction with the preset time step... The reservoir numerical simulation model in the reservoir numerical simulator is used to simulate the reservoir state, actions, and rewards obtained from the reservoir numerical simulation model at each time step, and the results are then used to simulate the reservoir state, actions, and rewards at each time step. The state of the reservoir at each time step ,action and rewards and the The state of the reservoir at each time step As sample data, the state is set as the average pressure and water saturation around the fracture, the action is set as the average injection volume and average production pressure of the fracture, and the reward is set as the step-by-step net present value.

[0039] Specifically, the formula for calculating the step-by-step net present value is as follows: ; In the formula, For the first The stepwise net present value of the reservoir at each time step; , , These are crude oil price, water injection cost, and produced water treatment cost, respectively. The crude oil price... Water injection cost Produced water treatment costs All units are ; , , The first The cumulative oil production, cumulative water injection, and water production of the reservoir at each time step, the first... Cumulative oil production of reservoir at each time step Cumulative water injection volume and water production All units are ; The annual discount rate; For time step.

[0040] In this embodiment, reservoir numerical simulation models are established in reservoir numerical simulation software based on asynchronous injection-production schemes within the same well section and staggered injection-production schemes between different well sections, respectively. The total development time of the reservoir numerical simulation model is set to 1080 days, the average injection fracture flow rate is set to 10~30 MPa, the fluctuation range of the injection fracture flow rate is set to 5~15 MPa, the probability of variation ranges from 0.01 to 0.1, indicating that the working system adjustment will occur within 10~100 days, the time step is set to 30 days, the annual discount rate is set to 0.1, and the crude oil price is set to 3600. The cost of water injection is set at 50. The cost of produced water treatment is set at 80. .

[0041] Step 2.4: Preprocess the sample data obtained from the simulation. First, remove the sample data that failed the simulation and contained outliers. Then, use the Z-Score standardization method to normalize the sample data. After normalization, randomly distribute the sample data to the training set and the validation set in an 8:2 ratio to construct the sample database.

[0042] Step 3: To achieve rapid simulation of reservoir dynamics, a deep learning surrogate model is established based on the LSTM (Long Short-Term Memory Network) network structure. The sample database established in Step 2 is used to train and validate the deep learning surrogate model, resulting in a validated deep learning surrogate model. This validated surrogate model is then used to replace the reservoir numerical simulation model to obtain simulation results, significantly improving the efficiency of subsequent pre-search and reinforcement learning optimization. Specifically, this includes the following sub-steps: Step 3.1: Establish a deep learning agent model based on the LSTM network structure, and set the maximum number of iterations and loss function of the deep learning agent model.

[0043] In this embodiment, the deep learning agent model is built based on an LSTM network structure, including an input layer, a hidden layer, a fully connected layer, and an output layer. The input layer is used to input the state and actions of the reservoir numerical simulation model at the current time step. The hidden layer uses multiple stacked LSTM units to learn the pressure propagation characteristics and saturation front advancement characteristics during reservoir seepage through a gating mechanism. The fully connected layer is set between the hidden layer and the output layer. The fully connected layer is used to map the high-dimensional feature data output by the hidden layer to the physical space. The output layer is used to output the predicted state of the reservoir numerical simulation model at the next time step and the production indicators at the current time step.

[0044] The mapping function of the deep learning agent model is: ; In the formula, For the first Predicted values ​​of reservoir state at each time step; , , The first Predicted production indicators, status, and actions of the reservoir at each time step; LSTM network weight parameters The proxy model mapping function; The hidden state memory passed to the LSTM network.

[0045] In this embodiment, when training the deep learning agent model, the gradient-based Adam optimization algorithm is used to update the network parameters of the deep learning agent model, and Dropout units are introduced into the LSTM network to prevent overfitting. To simultaneously measure the prediction accuracy of state and output, the mean squared error is used to construct the loss function of the deep learning agent model, resulting in: ; In the formula, For the loss function of the deep learning agent model; These are the weight parameters of the LSTM network; The sample data sequence number; The total number of sample data used in training; , These are all weighting coefficients of the loss function, used to determine the degree of fit between the equilibrium state and the output. , The first The reservoir status and production indicators at each time step; For the first Predicted values ​​of reservoir production indicators at each time step; As an L2 norm, combined with the outer square sign, this term represents the mean square error as a whole.

[0046] Step 3.2: Randomly select multiple sample data from the training set and input them into the deep learning proxy model. Train the deep learning proxy model to predict the production indicators of the reservoir numerical simulation model at the current moment and the state of the reservoir numerical simulation model at the next moment based on the state and actions of the reservoir numerical simulation model in the sample data. Calculate the loss function value of the deep learning proxy model during the training process.

[0047] Step 3.3: If the current iteration count has reached the maximum iteration count, stop training the deep learning proxy model and proceed to step 3.4. Otherwise, compare the calculated loss function value with the preset loss function value. If the calculated loss function value exceeds the preset loss function value, adjust the hyperparameters of the deep learning proxy model and return to step 3.2 to continue training the deep learning proxy model. If the calculated loss function value does not exceed the preset loss function value, stop training the deep learning proxy model and proceed to step 3.4.

[0048] Step 3.4: Randomly select multiple sample data from the test set and input them into the trained deep learning agent model. Evaluate the performance of the trained deep learning agent model based on the coefficient of determination and mean absolute percentage error. If the coefficient of determination of the trained deep learning agent model is greater than 0.9 and the mean absolute percentage error is less than 10%, proceed to step 3.5. Otherwise, adjust the hyperparameters of the deep learning agent model and return to step 3.2 to continue training the deep learning agent model.

[0049] Specifically, the formula for calculating the coefficient of determination is as follows: ; In the formula, The coefficient of determination; The total number of sample data used in the validation; The true value of the sample data; The predicted value output by the deep learning agent model; This is the arithmetic mean of the sample data used for validation.

[0050] The formula for calculating the mean absolute percentage error is: ; In the formula, The mean absolute percentage error.

[0051] Step 3.5: Output the validated deep learning agent model.

[0052] Step 4: Replace the reservoir numerical simulation model with the validated deep learning surrogate model, and use the particle swarm optimization algorithm for single-objective pre-search global optimization. Determine the optimal benchmark strategy through fast search to obtain the benchmark injection-production scheme. This includes the following sub-steps:

[0053] Step 4.1 sets maximizing the cumulative net present value over the entire cycle as the objective function of the pre-search process, expressed as: ; In the formula, It is a function for maximizing the value; The objective function is... This refers to the time step number; This represents the total number of time steps. For the first The stepwise net present value of the reservoir at each time step.

[0054] Production regime parameters, including injection volume and production pressure, are used as optimization variables. To reduce dimensionality, a combination of mean and fluctuation range is used to construct the injection-production regime scheme. The third parameter in the deep learning proxy model is set. The crack in the first The actions at each time step are: ; In the formula, The crack number; For the first The crack in the first Injected traffic at each time step; For the first The crack in the first The extraction pressure at each time step; For the first The average injection flow rate of the crack; For the first Average bottom hole flowing pressure of the fracture; For the first The amplitude of the crack fluctuation; It is a sine function.

[0055] In this embodiment, the total number of time steps of the deep learning proxy model is set. The value is 36, and the average injected flow rate is set to 36. The average bottom hole flowing pressure was set to 10~30MPa, and the fluctuation range was set to 10%~30%.

[0056] Step 4.2: Set the initial parameters of the particle swarm optimization algorithm, including the total number of particles, the maximum number of iterations, the inertia weight, and the learning factor. In this embodiment, the total number of particles is set to 80, the maximum number of iterations is set to 100, the inertia weight adopts a linear decreasing strategy, decreasing from 0.9 to 0.4 with the number of iterations, and the learning factor is set to 1.5.

[0057] Step 4.3: Using particles from the particle swarm to represent the injection-progression regime, randomly set the position and velocity of each particle in the particle swarm. The particle velocity is the rate of change of the particle position, used to represent the magnitude of the injection-progression parameter adjustment. The particle position is the injection-progression regime parameter, expressed as: ; In the formula, For the first The position of each particle; This represents the total number of time steps. For the first The fluctuation range of the crack.

[0058] For each particle in the particle swarm, the injection-procurement scheme represented by the particle is input into the validated deep learning proxy model for prediction, so as to obtain the production indicators and cumulative net present value corresponding to the injection-procurement scheme, which are used as the fitness value of the particle.

[0059] Step 4.4: Update the optimal value of each particle, and select the particle with the highest fitness as the globally optimal particle, resulting in: ; In the formula, For the first During the nth iteration calculation The optimal fitness value of each particle; For the first During the nth iteration calculation The optimal fitness value of each particle; For the first During the nth iteration calculation The position of each particle; This is the fitness value.

[0060] The formula for updating the velocity and position of each particle in the particle swarm is as follows: ; ; In the formula, For the first During the nth iteration calculation The velocity of each particle; Inertial weights; , All are learning factors; , All values ​​are random numbers, ranging from [0,1]. By introducing random perturbations into the particle velocity update, population diversity is increased, preventing premature entry into local optima. , The first During the nth iteration calculation The velocity and position of each particle; This is the globally optimal value.

[0061] Step 4.5: Determine whether the current iteration count has reached the preset maximum iteration count. If it has, proceed to step 4.6; otherwise, update the iteration count and return to step 4.4 to continue particle swarm optimization.

[0062] Step 4.6: Output the injection and extraction regime scheme corresponding to the globally optimal particle as the preferred benchmark strategy to obtain the benchmark injection and extraction regime scheme.

[0063] Step 5: Establish a reinforcement learning dynamic decision-making model based on the Actor-Critic dual network structure with policy gradient. Train the reinforcement learning dynamic decision-making model using the PPO proximal policy optimization algorithm to obtain a reinforcement learning agent, which is used for fine-tuning the benchmark injection and collection scheme. This step includes the following sub-steps:

[0064] Step 5.1: Establish a reinforcement learning dynamic decision-making model based on the Actor-Critic dual network structure of policy gradient, and connect the reinforcement learning dynamic decision-making model and the validated deep learning agent model based on the Markov decision process. Use the reinforcement learning dynamic decision-making model to predict the cumulative net present value of the reservoir within a specified time in the future according to the current injection and production scheme.

[0065] Specifically, the Markov decision process is used for parameter passing in reinforcement learning dynamic decision models and deep learning agent models by establishing a continuous state-action MDP quintuple. ,in, The state space is specifically a state vector composed of the local formation pressure and water saturation around each fracture at all time steps, which serves as the input to the reinforcement learning dynamic decision-making model. This is the action space, used to represent adjustments to the injection-production regime, by introducing an adjustment coefficient relative to the benchmark injection-production regime. The adjusted injection and collection system is as follows: ; In the formula, For the adjusted number The actions of the reservoir at each time step; For the first The actions of the reservoir at each time step; To adjust the coefficient, the adjustment coefficient in this embodiment is... The value range is set to [-0.2, 0.2].

[0066] The state transition model, namely the deep learning agent model verified in step 3, is responsible for receiving the current state of the reservoir and the adjusted actions, and calculating the state of the reservoir in the next time step. The reward function is the same as the step-by-step net present value calculation formula in step 2; This is a discount factor used to balance current returns with long-term returns.

[0067] In this embodiment, the reinforcement learning dynamic decision-making model is constructed based on an Actor-Critic dual-network structure with policy gradients, including a policy network and a value network. The policy network, used to adjust the injection-production regime, includes an input layer, a fully connected hidden layer, and an output layer connected sequentially, used to adjust the current reservoir state based on the input. Determine the adjustment coefficient The normal distribution function is used to determine the adjustment coefficient through random sampling. The value network, used to evaluate the merits of adjustments to the injection-production regime, comprises an input layer, a fully connected hidden layer, and an output layer connected in sequence, used to evaluate the current reservoir state based on the input data. Determine the state value function and calculate the advantage function. Guidance strategy network update.

[0068] The state value function is used to determine the current state of the reservoir. The cumulative net present value of a reservoir over a specified future time period is predicted by the injection-production scheme, expressed as: ; In the formula, The state value function; For the first The state of the reservoir at each time step; For the first The state of the reservoir at each time step; For the first The state of the reservoir at each time step; Expected based on experience; Discount factor; For the first The reward for each time step of the reservoir.

[0069] Step 5.2: Train the reinforcement learning dynamic decision-making model using the PPO (Proximity Policy Optimization) algorithm, and update the network parameters of the reinforcement learning dynamic decision-making model. The specific process is as follows: Step 5.2.1: Initialize the PPO proximal policy optimization algorithm parameters, setting the learning rate, truncation parameter, discount factor, generalized advantage estimation parameter, maximum number of training epochs, and composite total loss function.

[0070] Specifically, in this embodiment, the learning rate is set to 0.0003, the truncation parameter is set to 0.2, the discount factor is set to 0.99, the generalized advantage estimation parameter is set to 0.95, the maximum number of training epochs is set to 1000, and the composite total loss function is set as follows: ; in, ; ; ; In the formula, This is the composite total loss function; To maximize the strategy's reward function; The policy entropy function; , These are all weighting coefficients of the composite total loss function. In this embodiment, the weighting coefficients of the composite total loss function are... The value is set to 0.5, and the weighting coefficient of the composite total loss function is... The value is set to 0.01; These are the policy network weight parameters; These are the weight parameters of the value network; The current injection / collection strategy is determined by the strategy network weight parameters; The predicted value is the expected value based on experience; It is a minimum value function; The new and old injection and extraction system schemes were proposed in the [number]th [year]. Select an action within a time step The probability ratio; The dominant function; This is a truncation function, and its value range is... This is used to prevent excessively large parameter updates; To truncate parameters; The value network loss function; This is an estimate of the state value; For the first The state of the reservoir at each time step; For the first The return on investment for a reservoir at a given time step; The crack number; To optimize the total number of cracks; It is a natural constant; For the first The standard deviation of the crack action; It is a logarithmic function with base 10.

[0071] Step 5.2.2: During the iterative training process, the policy network of the reinforcement learning dynamic decision-making model is first used to identify the current state of the reservoir. And output adjustment coefficient The adjusted injection and collection scheme is then input into the validated deep learning agent model, which is used to obtain the reward for the current time step. and the state of the reservoir at the next time step. A reservoir database consisting of reservoir data from all time steps is obtained. The data from this database are then input into a reinforcement learning dynamic decision model. The value network of the dynamic decision model outputs the state-value function for each time step, determining the reward and advantage function for each reservoir at each time step. The calculation formula is as follows: ; ; In the formula, For the first The return on investment for a reservoir at a given time step; For the first The reward for each time step of the reservoir; Discount factor; For the first The return on investment for a reservoir at a given time step; For the first The dominant function of a reservoir at each time step; This is a summation index used to indicate the number of future delay steps; This represents the total number of time steps. This refers to the time step number; For parameters to estimate the generalized dominance; For the first The reward for each time step of the reservoir; For the first The state of the reservoir at each time step; For the first The state of the reservoir at each time step.

[0072] Step 5.2.3: Based on the gradient optimization algorithm and the preset learning rate, calculate the composite total loss function value relative to the policy network weight parameters. and value network weight parameters The gradient is used to update the network parameters of the reinforcement learning dynamic decision model.

[0073] Step 5.2.4: Continue iterative calculations until the current number of iterations reaches the preset maximum number of training rounds or the preset reward convergence condition is met, then end the training of the reinforcement learning dynamic decision model.

[0074] In this embodiment, the reward convergence condition is used to evaluate the steady-state characteristics of the cumulative net present value over the entire cycle. By introducing a window size based on the moving average method to smooth the training curve of the reinforcement learning dynamic decision-making model, the relative growth rate of the average step-by-step net present value in the current window relative to the previous window is calculated. When the relative growth rate is less than the preset growth rate threshold of 0.005, it is determined that the reinforcement learning dynamic decision-making model has converged to the optimal strategy, and the iterative calculation is stopped at this time.

[0075] Step 5.3: Save the network parameters of the reinforcement learning dynamic decision-making model to obtain the reinforcement learning agent.

[0076] Step 6: Obtain the initial formation state, total development time, and time step of the actual reservoir. Input the benchmark injection-production regime scheme obtained in Step 4 into the reinforcement learning agent obtained in Step 5. Use the reinforcement learning agent to generate the optimal dynamic injection-production control scheme for the entire life cycle of the actual reservoir. Obtain the injection flow rate and production pressure of all fractures in the actual reservoir at all time steps within the total development time. Generate an intelligent injection-production regime table containing the injection flow rate sequence and production pressure sequence to obtain the optimal injection-production development scheme for the actual reservoir.

[0077] In this embodiment, a dynamic injection-production work schedule table for asynchronous injection-production schemes between the same well section is obtained after optimization based on the method of the present invention, as follows: Figure 5As shown, Figure 5 The negative values ​​on the vertical axis represent the injection flow rate of the injection joint, while the positive values ​​represent the pressure of the production joint. Figure 6 To obtain the reservoir parameter variation curves based on the optimized dynamic injection-production scheme after asynchronous injection-production schemes between the same well sections, the following steps were taken: Figure 6 The curves of cumulative oil production, cumulative water production, water cut, and net present value of the reservoir over time were obtained when an asynchronous injection and production scheme was implemented between the same well sections.

[0078] In this embodiment, a dynamic injection-production work schedule table for the inter-well section staggered injection-production scheme, optimized based on the method of the present invention, is obtained, as follows: Figure 7 As shown, Figure 7 The negative values ​​on the vertical axis represent the injection flow rate of the injection joint, while the positive values ​​represent the pressure of the production joint. Figure 8 To obtain the reservoir parameter variation curves based on the optimized dynamic injection-production scheme of inter-well sections, the following steps were taken: Figure 8 The curves of cumulative oil production, cumulative water production, water cut, and net present value of the reservoir over time were obtained when the injection and production development was carried out based on the staggered injection and production scheme between different well sections.

[0079] In summary, the method of this invention addresses the strong heterogeneity and dynamic disturbances present in multi-fracture and multi-well environments by combining single-objective pre-search and reinforcement learning dynamic decision-making. By introducing a residual control mechanism into the reinforcement learning dynamic decision-making model, the optimal baseline strategy is dynamically fine-tuned based on real-time formation pressure and water saturation. This yields the optimal injection-production development scheme for reservoirs under different well-fracture joint control fine-tuning modes. This effectively solves the limitations of traditional numerical simulation methods for optimizing injection-production schemes, which suffer from long computation times and low efficiency. It reduces the computational cost of massive simulations to screen for the optimal injection-production scheme, enabling rapid optimization and decision support for reservoir injection-production schemes under new multi-type well-fracture joint control modes, and providing technical support for actual reservoir development strategies.

[0080] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A multi-type well seam joint control fine injection and production mode intelligent optimization method, characterized in that, The method comprises the following steps: Step 1, setting a well fracture joint control fine injection-production mode, combining reservoir physical property parameters and horizontal well design parameters, establishing a reservoir numerical simulation model in a reservoir numerical simulation software for simulating the injection-production process of the fractured horizontal well; Step 2, setting core sampling parameters and value ranges, obtaining multiple sets of reservoir injection-production schemes based on a Latin hypercube sampling method, simulating the reservoir numerical simulation model according to each set of reservoir injection-production scheme to obtain the state, action and reward of the reservoir numerical simulation model at each time step when each set of reservoir injection-production scheme is used, generating multiple sample data and establishing a sample database; Step 3, establishing a deep learning agent model based on an LSTM network structure, training and verifying the deep learning agent model by using the sample database to obtain a verified deep learning agent model; Step 4, using the verified deep learning agent model to replace the reservoir numerical simulation model, performing single-target pre-search global optimization by using a particle swarm optimization algorithm, determining an optimal benchmark strategy by fast search to obtain a benchmark injection-production system scheme; Step 5, establishing a reinforcement learning dynamic decision-making model based on an Actor-Critic double network structure, training the reinforcement learning dynamic decision-making model based on a PPO proximal policy optimization algorithm to obtain a reinforcement learning intelligent agent for fine regulation and control of the benchmark injection-production system scheme; Step 6, obtaining the initial formation state, total development time and time step of the actual reservoir, inputting the benchmark injection-production system scheme obtained in step 4 into the reinforcement learning intelligent agent, obtaining the injection flow rate and production pressure of all fractures at all time steps within the total development time of the actual reservoir by using the reinforcement learning intelligent agent, generating an intelligent injection-production work system table containing an injection flow rate sequence and a production pressure sequence, and obtaining an optimal injection-production development scheme of the actual reservoir.

2. The intelligent optimization method for multi-type well fracture associated control fine injection and production mode according to claim 1, characterized in that, In the step 1, the well fracture joint control fine injection-production mode comprises an inter-well section asynchronous injection-production scheme and an inter-well section staggered injection-production scheme, wherein the inter-well section asynchronous injection-production scheme is designed for a single fractured horizontal well, and the injection-production development of the fractured horizontal well is performed by alternately arranging production fractures and injection fractures in the horizontal well section of the single fractured horizontal well; the inter-well section staggered injection-production scheme is designed for multiple fractured horizontal wells, and the injection-production development of the fractured horizontal well is performed by staggeredly arranging fractures between the same well sections of adjacent two fractured horizontal wells.

3. The intelligent optimization method for multi-type well fracture associated control fine injection and production mode according to claim 1, characterized in that, In the step 2, the following sub-steps are included: Step 2.1, setting the total development time and the initial formation pressure of the reservoir numerical simulation model according to the actual working condition of the reservoir, setting the core sampling parameters and the value ranges, wherein the core sampling parameters include the mean value and fluctuation range of the injection fracture flow rate, the mean value and fluctuation range of the production fracture pressure, and the adjustment frequency; Step 2.2, based on the Latin hypercube sampling method, stratified sampling is performed in the parameter space of each core sampling parameter to obtain multiple sets of core sampling parameters which are not repeated, and the core sampling parameters collected are converted into injection-production parameters according to a preset threshold judgment rule to obtain multiple sets of injection-production schemes. Step 2.3, according to each group injection-production scheme, combined with the preset time step, the numerical reservoir simulator is used to simulate the state, action and reward of the reservoir obtained by the numerical reservoir simulation model at each time step, and the state, action and reward of the reservoir at the first time step and the state of the reservoir at the second time step are taken as sample data, wherein the state is set as the average fracture surrounding pressure and water saturation, the action is set as the average fracture injection and average production pressure, and the reward is set as the step net present value. state of the reservoir at the first time step action and reward and the state of the reservoir at the second time step . ​ Step 2.4, preprocessing the analog obtained sample data, removing analog failure and sample data containing abnormal values, then normalizing the sample data by using Z-Score standardization method, and randomly distributing the normalized sample data to the training set and the validation set to construct a sample database.

4. The intelligent optimization method for multi-type well fracture association control fine injection and production mode according to claim 1, characterized in that, In the step 3, the following sub-steps are included: Step 3.1, establishing a deep learning agent model based on the LSTM network structure, setting the maximum iteration number and the loss function of the deep learning agent model; Step 3.2, randomly extracting multiple sample data from the training set and inputting them into the deep learning agent model, training the deep learning agent model to predict the production index of the current time step of the reservoir numerical simulation model and the state of the next time step of the reservoir numerical simulation model according to the state and action of the sample data, and calculating the loss function value of the deep learning agent model in the training process; Step 3.3, if the current iteration calculation number reaches the maximum iteration number, stop training the deep learning agent model and enter step 3.4, otherwise, compare the calculated loss function value with the preset loss function value, if the calculated loss function value exceeds the preset loss function value, adjust the hyperparameters of the deep learning agent model, return to step 3.2 to continue training the deep learning agent model, if the calculated loss function value does not exceed the preset loss function value, stop training the deep learning agent model and enter step 3.4; Step 3.4, randomly extracting multiple sample data from the test set and inputting them into the trained deep learning agent model, and evaluating the performance of the trained deep learning agent model based on the determination coefficient and the average absolute percentage error, if the determination coefficient of the trained deep learning agent model is greater than the preset determination coefficient value and the average absolute percentage error is less than the preset error value, enter step 3.5, otherwise, adjust the hyperparameters of the deep learning agent model and return to step 3.2 to continue training the deep learning agent model; Step 3.5, outputting the verified deep learning agent model.

5. The intelligent optimization method for multi-type well fracture association control fine injection and production mode according to claim 4, characterized in that, The deep learning agent model is established based on the LSTM network structure and includes an input layer, a hidden layer, a fully connected layer and an output layer, wherein the input layer is used to input the state and action of the current time step reservoir numerical simulation model, the hidden layer adopts a plurality of LSTM units stacked, learns the pressure propagation characteristics and saturation front advance characteristics in the reservoir seepage process through the gating mechanism, the fully connected layer is arranged between the hidden layer and the output layer, and is used to map the high-dimensional feature data output by the hidden layer to the physical space, and the output layer is used to output the predicted state of the next time step reservoir numerical simulation model and the production index of the current time step; The mapping relationship function of the deep learning agent model is: ； In the formula, is the predicted value of the reservoir state at the first time step; , , are respectively the predicted value of the production index, the state and the action of the reservoir at the first time step; is the proxy model mapping function of the LSTM network weight parameter ; is the hidden state memory passed by the LSTM network; The loss function of the deep learning agent model is: ； In the formula, For the loss function of the deep learning agent model; These are the weight parameters of the LSTM network; The sample data sequence number; The total number of sample data used in training; , These are all weighting coefficients of the loss function; , The first The reservoir status and production indicators at each time step; For the first Predicted values ​​of reservoir production indicators at each time step; It is an L2 norm.

6. The intelligent optimization method for multi-type well fracture association control fine injection and production mode according to claim 1, characterized in that, In the step 4, the following sub-steps are included: Step 4.1, setting the maximum full-cycle cumulative net present value as the objective function of the pre-search process, the expression is: ； wherein, ； wherein, is a maximum function; is an objective function; is a time step number; is a total number of time steps; is a stepwise net present value of the reservoir at the time step; , , are a crude oil price, a water injection cost, and a produced water treatment cost, respectively; , , are a cumulative oil production, a cumulative water injection, and a water production of the reservoir at the time step, respectively; is an annual discount rate; is a time step length; The production regime parameters including injection volume and production pressure are taken as the optimization variables, and the depth learning agent model is set in the first The action of the bar crack at the first time step is: ； In the formula, The crack number; For the first The crack in the first Injected traffic at each time step; For the first The crack in the first The extraction pressure at each time step; For the first The average injection flow rate of the crack; For the first Average bottom hole flowing pressure of the fracture; For the first The amplitude of the crack fluctuation; It is a sine function; Step 4.2, set the initial parameters of the particle swarm optimization algorithm, including the total number of particles, the maximum number of iterations, the inertia weight and the learning factor; Step 4.3, use the particles in the particle swarm to represent the injection-production system scheme, and randomly set the position and velocity of each particle in the particle swarm, wherein the velocity of the particle is the change rate of the position of the particle, which is used to represent the adjustment amplitude of the injection-production parameter, and the position of the particle is the injection-production system parameter; For each particle in the particle swarm, input the injection-production system scheme represented by the particle into the verified deep learning agent model for prediction to obtain the production index and cumulative net present value corresponding to the injection-production system scheme, and use them as the fitness value of the particle; Step 4.4, update the optimal value of each particle, take the particle with the highest fitness value among all particles as the global optimal particle, and update the velocity and position of each particle in the particle swarm; Step 4.5, judge whether the current iteration count reaches the preset maximum iteration count, if yes, go to step 4.6, otherwise, update the iteration count and return to step 4.4 to continue the particle swarm optimization; Step 4.6, output the injection-production system scheme corresponding to the global optimal particle as the preferred benchmark strategy to obtain the benchmark injection-production system scheme.

7. The intelligent optimization method for multi-type well fracture association control fine injection and production mode according to claim 1, characterized in that, In step 5, the following sub-steps are included: Step 5.1, establish a reinforcement learning dynamic decision-making model based on the Actor-Critic double network structure of policy gradient, and connect the reinforcement learning dynamic decision-making model and the verified deep learning agent model based on Markov decision process, and use the reinforcement learning dynamic decision-making model to predict the cumulative net present value of the reservoir within a specified future time according to the current injection-production system scheme; Step 5.2, train the reinforcement learning dynamic decision-making model using the PPO proximal policy optimization algorithm, and update the network parameters of the reinforcement learning dynamic decision-making model, Step 5.3, save the network parameters of the reinforcement learning dynamic decision-making model to obtain the reinforcement learning agent.

8. The intelligent optimization method for multi-type well fracture association control fine injection and production mode according to claim 7, characterized in that, The reinforcement learning dynamic decision-making model is constructed based on the Actor-Critic double network structure of policy gradient, including a policy network and a value network, wherein the policy network is used to adjust the injection-production system, including an input layer, a fully connected hidden layer and an output layer connected in sequence, for determining a normal distribution function of adjustment coefficient according to the input current state of the reservoir, and determining the adjustment coefficient by random sampling; the value network is used to evaluate the pros and cons of injection-production system adjustment, including an input layer, a fully connected hidden layer and an output layer connected in sequence, for determining a state value function according to the input current state of the reservoir, and guiding the policy network update by calculating the advantage function; The state value function is used to predict the cumulative net present value of the reservoir within a specified future time according to the current state of the reservoir and the injection-production system scheme, and the expression is: ； wherein is the state value function; is the state of the reservoir at the time step; is the state of the reservoir at the time step; is the empirical expectation; is the discount factor; is the reward of the reservoir at the time step; The Markov decision process is used for parameter transfer between a reinforcement learning dynamic decision model and a deep learning agent model, by establishing a MDP five tuple of continuous state and action wherein, is a state space, an input of the reinforcement learning dynamic decision model; is an action space, used to represent the adjustment of the injection-production system; is a state transition model, used to receive the current state of the reservoir and the adjusted action and calculate the state of the reservoir at the next time step; is a reward function; is a discount factor.

9. The intelligent optimization method for multi-type well fracture association control fine injection and production mode according to claim 8, characterized in that, Step 5.2 includes the following sub-steps: Step 5.2.1, initialize the PPO proximal policy optimization algorithm parameters, set the learning rate, the truncation parameter, the discount factor, the generalized advantage estimation parameter, the maximum training round and the compound total loss function; In the iterative training process of step 5.2.2, the policy network of the reinforcement learning dynamic decision model is used to identify the current state of the reservoir and output the adjustment coefficient, and the adjusted injection-production system scheme is input into the verified deep learning agent model, the reward of the current time step and the state of the reservoir at the next time step are obtained by using the verified deep learning agent model, the reservoir database composed of reservoir data at all time steps is obtained, and each data in the reservoir database is input into the reinforcement learning dynamic decision model, the value network of the reinforcement learning dynamic decision model is used to output the state value function at each time step, the return and advantage function of the reservoir at each time step are determined, and the calculation formula is: ； ； In the formula, For the first The return on investment for a reservoir at a given time step; For the first The reward for each time step of the reservoir; Discount factor; For the first The return on investment for a reservoir at a given time step; For the first The dominant function of a reservoir at each time step; For summation index; This represents the total number of time steps. This refers to the time step number; For parameters to estimate the generalized dominance; For the first The reward for each time step of the reservoir; For the first The state of the reservoir at each time step; For the first The state of the reservoir at each time step; In step 5.2.3, based on the gradient optimization algorithm, the gradient of the composite total loss function value compared with the weight parameters of the policy network and the weight parameters of the value network is calculated combined with the preset learning rate, and the network parameters of the reinforcement learning dynamic decision model are updated; In step 5.2.4, the iterative calculation is continued until the current iteration reaches the preset maximum training round or meets the preset reward convergence condition, and the training of the reinforcement learning dynamic decision model is ended.

10. The intelligent optimization method for multi-type well fracture association control fine injection and production mode according to claim 9, characterized in that, The composite total loss function is set as: ； wherein, ； ； ； wherein, is a composite total loss function; is a maximization policy reward function; is a policy entropy function; , are weight coefficients of the composite total loss function; is a policy network weight parameter; is a value network weight parameter; is a current injection-production policy determined by the policy network weight parameter; is an experience-expected prediction value; is a minimum function; is a probability ratio of selecting an action of a new and old injection-production system scheme in the th time step; is an advantage function; is a truncation function; is a truncation parameter; is a value network loss function; is a state value estimate; is a state of the reservoir in the th time step; is a return of the reservoir in the th time step; is a fracture number; is an optimized total number of fractures; is a natural constant; is a standard deviation of the th fracture action; is a logarithm function with base 10.

Images