A Reinforcement Learning Method for Horizontal Well Regulation Assisted by Evolutionary Algorithm

Abstract

The present invention discloses a horizontal well regulation reinforcement learning method assisted by an evolutionary algorithm, belonging to the technical field of reservoir production optimization, and comprising the following steps: Step 1, determine the position of the segmentation point, the valve opening degree, and the injection-production variables to be optimized, and construct a hybrid action space combining discrete and continuous regulation measures; Step 2, construct a hybrid deep reinforcement learning agent model and a recent experience prioritized replay pool; Step 3, interact the hybrid deep reinforcement learning agent model with a numerical simulator, and train the agent through the generated experience samples; Step 4, perform replacement, selection, crossover, and mutation operations of the evolutionary algorithm to optimize the hybrid deep reinforcement learning agent model; Step 5, repeatedly execute Step 3 to Step 4 until the maximum number of episodes is reached. The present invention combines the evolutionary algorithm with the hybrid deep reinforcement learning agent model, can effectively optimize the well section division and valve opening degree of the horizontal well, and adjust the production system of each well in real time.

Description

Technical Field

[0001] The present invention belongs to the technical field of reservoir production optimization, and particularly relates to a horizontal well control and regulation reinforcement learning method assisted by an evolutionary algorithm. Background Art

[0002] During the development of oil and gas fields, horizontal well technology is widely used in oil and gas extraction under complex geological conditions. However, due to the heterogeneity and multi-layer structure of the reservoir, there are significant differences in the liquid production and fluid flow characteristics of each well section during the extraction process. To improve the oil and gas recovery efficiency, it is necessary to carry out refined sectional management of oil wells and accurately control the valve opening of each well section to ensure the balanced extraction of oil and gas resources. With the dynamic changes in reservoir conditions, the production regime of each well also needs to be optimized and adjusted in real time. Traditional oil well control and regulation methods mostly rely on manual experience and manual adjustment. This method is not only inefficient but also difficult to quickly respond to the dynamic changes in the reservoir. In the face of complex and unpredictable reservoir conditions, it is often difficult to achieve optimal production control.

[0003] As a powerful optimization algorithm, reinforcement learning can achieve adaptive regulation of the production system under the conditions of dynamic changes in the reservoir by continuously learning and adjusting strategies. However, although reinforcement learning can optimize production control to a certain extent, its efficiency and effect are limited by the quality of training data and the convergence speed of the algorithm. In practical applications, the training of reinforcement learning models often requires a large number of samples and computing resources, and the exploration ability during the model training process is limited, making it difficult to effectively avoid local optimal solutions. To improve the efficiency and global search ability of reinforcement learning models, an auxiliary optimization scheme combined with an evolutionary algorithm has become an effective choice. The evolutionary algorithm simulates the process of natural selection and conducts searches within a large range through genetic and selection operations to optimize the decision-making process. The diversity and randomness of the evolutionary algorithm enable it to conduct searches globally and avoid the dilemma of reinforcement learning falling into local optima. In addition, the population characteristics of the evolutionary algorithm can promote the comprehensive exploration of the search space through the evolution and adaptive update of the population, further improving the performance of the reinforcement learning model. Through this method, the valve opening and production regime of each well section can be optimized in real time under the dynamic changes in the reservoir, thereby achieving the maximum recovery of oil and gas resources and having broad application prospects. Summary of the Invention

[0004] To solve the above problems, the present invention proposes a horizontal well control reinforcement learning method assisted by an evolutionary algorithm. This method combines a hybrid deep reinforcement learning agent model with an evolutionary algorithm to optimize the training and exploration process of reinforcement learning through the evolutionary algorithm, thereby accelerating the learning process and improving the global search ability of the model. Through this method, the horizontal well section division can be determined according to the reservoir state, and the valve opening of each section and the production regime of each well can be adjusted in real time to ensure the maximization of oil and gas recovery rate and quickly adapt to different production requirements under dynamically changing reservoir conditions. Finally, the optimal control scheme for each well is obtained, thus greatly improving the oilfield exploitation efficiency and economic benefits.

[0005] The technical solution of the present invention is as follows:

[0006] A horizontal well control reinforcement learning method assisted by an evolutionary algorithm, comprising the following steps:

[0007] Step 1, determine the position of the segmentation point, valve opening and injection-production variables to be optimized, and construct a hybrid action space combining discrete and continuous control measures;

[0008] Step 2, construct a hybrid deep reinforcement learning agent model and a prioritized replay buffer for recent experiences;

[0009] Step 3, interact the hybrid deep reinforcement learning agent model with a numerical simulator, and train the agent through the generated experience samples;

[0010] Step 4, perform replacement, selection, crossover, and mutation operations of the evolutionary algorithm to optimize the hybrid deep reinforcement learning agent model;

[0011] Step 5, repeatedly execute Step 3 to Step 4 until the maximum number of episodes is reached.

[0012] Furthermore, the specific process of Step 1 is as follows:

[0013] Step 1.1, determine the horizontal well segmentation points, valves and the number of oil and water wells to be controlled according to the target reservoir. The variables to be optimized include the position of the segmentation point, valve opening and injection-production variables. The oil and water wells include production wells and injection wells, and the injection-production variables include production rate and injection rate; specifically expressed as:

[0014] ;

[0015] In the formula, is the control time step is the variable to be optimized at is the position of the th segmentation point at the current time step; is the opening of the th valve at the current time step; is the production rate of the th production well at the current time step; is the injection rate of the th injection well at the current time step;

[0016] Step 1.2, construct the discrete action space is:

[0017] ;

[0018] Step 1.3, construct the continuous action space is:

[0019] ;

[0020] Step 1.4, the hybrid action space combining discrete and continuous control measures is represented as:

[0021] ;

[0022] In the formula, is the hybrid action space.

[0023] Furthermore, in the said Step 2, the hybrid deep reinforcement learning agent model includes a policy network, an action-value network and a target action-value network; among them, the policy network consists of two parts: a discrete policy network and a continuous policy network; the discrete policy network consists of a convolutional neural network and a fully connected neural network, the input is the reservoir saturation field and pressure field, the data first extracts the reservoir data features through the convolutional neural network, and then outputs the discrete action probability distribution through the fully connected neural network; the continuous policy network consists of a convolutional neural network and a fully connected neural network, the input is the reservoir saturation field, pressure field and discrete action probability distribution, the reservoir saturation field and pressure field first extract the reservoir data features through the convolutional neural network, and after the extracted reservoir data features are concatenated with the discrete action probability distribution, they are input into the fully connected neural network, and the Gaussian distribution of the continuous action is output;

[0024] The network structures of the action-value network and the target action-value network are the same, both consisting of a convolutional neural network and a fully connected neural network; the input of the action-value network is the reservoir saturation field, pressure field and continuous action at a given time step, and the input of the target action-value network is the reservoir saturation field, pressure field and continuous action at the next time step of a given time step; the reservoir saturation field and pressure field first extract the reservoir data features through the convolutional neural network, and after the extracted reservoir data features are concatenated with the continuous action, they are input into the fully connected neural network, the action-value network outputs the Q value of the action at a given time step, and the target action-value network outputs the Q value of the action at the next time step of a given time step;

[0025] Construct a recent experience prioritized replay pool The sampling ratio factor and the actual sampling range are as follows:

[0026] ;

[0027] ;

[0028] In the formula, is the sampling ratio factor; is the current update count; is the total update count; is the hyperparameter controlling the sampling bias; is the actual sampling range; is the number of experiences in the current experience pool.

[0029] Furthermore, in step 3, the agent is an individual for reinforcement learning; the specific process of step 3 is as follows:

[0030] Step 3.1: Set the reward value at the current time step to the economic net present value at the current time step:

[0031] ;

[0032] In the formula, is the current time step; is the reward value at the current time step; is the economic net present value at the current time step; is the number of production wells; is the number of injection wells; is the unit price of crude oil sales; is the production well water treatment cost; is the injection well water injection cost; , are respectively the daily oil production and daily water production of the th production well at the current time step; is the th injection well injection rate;

[0033] Step 3.2: Take the saturation field and pressure field data of the reservoir model at the current time step as the state and input it into the policy network. The policy network outputs the discrete action at the current time step and the continuous action at the current time step; Calculate the reward value at the current time step and the state at the next time step through the numerical simulator. If the current time step is the last time step, the end flag is 1, otherwise it is 0; Take the trajectory of the experience sample Stored in the recent experience prioritized replay buffer ;

[0034] Step 3.3: Use the experience samples in the recent experience prioritized replay buffer to train the action value loss, specifically as follows:

[0035] ;

[0036] ;

[0037] ;

[0038] In the formula, is the Q value of the target action; is the discount factor; represents the discrete action probability distribution; is to find the minimum value; is the th target action value network; represents the continuous action at the next time step; represents the temperature coefficient of the continuous action; represents the continuous action probability distribution; represents the discrete action at the next time step; represents the temperature coefficient of the discrete action; is the th loss function of the action value network, and the serial number of the action value network corresponds to that of the target action value network; represents calculating the expectation; is the action value network; represents the total loss function of the action value network;

[0039] Step 3.4: Use the experience samples in the recent experience prioritized replay buffer to train the policy network parameters, specifically as follows:

[0040] ;

[0041] ;

[0042] ;

[0043] Among them, is the loss function of the discrete policy network; is to calculate the expectation; is the loss function of the continuous policy network; is the total loss function of the policy network;

[0044] Step 3.5: Design the loss functions for the temperature coefficients of continuous actions and discrete actions, and update and as follows:

[0045] ;

[0046] ;

[0047] where, is the loss function for the temperature coefficient of continuous actions; is the loss function for the temperature coefficient of discrete actions; and represent the entropies of the continuous policy network and the discrete policy network respectively; and represent the target entropies of the continuous policy network and the discrete policy network respectively.

[0048] Furthermore, the specific process of Step 4 is as follows:

[0049] Step 4.1: The evolutionary algorithm has individuals, and each individual is a policy network; the sum of all rewards within one episode of the policy network is recorded as the initial fitness value of the individual; a penalty term is set, and if the continuous action output by the policy network is equal to the boundary value, the fitness value of the policy network is deducted accordingly; calculate the fitness values of all individuals, and the formula is:

[0050] ;

[0051] In the formula, is the fitness value of individual ; is the initial fitness value of individual ; is the penalty term of individual , and the penalty term increases by 1 for each boundary value taken by the individual;

[0052] Calculate the minimum tolerance of all individuals:

[0053] ;

[0054] In the formula, is the minimum tolerance of individual ; is the previous-round fitness value of individual ; is the tolerance threshold;

[0055] If the fitness value of the current individual is the lowest or lower than the minimum tolerance of the individual, replace the current individual with the reinforcement learning individual;

[0056] Step 4.2: The individual with the highest fitness value directly enters the elite pool, and then the roulette wheel algorithm is used to select several individuals to enter the elite pool. The higher the fitness value of an individual, the greater the probability of being selected. The probability calculation formula is as follows:

[0057] ;

[0058] ;

[0059] In the formula, is the normalized fitness value of individual ; is the fitness value; is a constant; is the probability that individual is selected;

[0060] Step 4.3: All individuals that do not enter the elite pool enter the remaining evolution pool, and the three-source crossover operation is performed on all individuals in the remaining evolution pool; this operation performs crossover on the convolutional layer, fully connected layer, and bias term, where the convolution kernel, row, and element are used as the basic units of crossover respectively; during the crossover process of each basic unit, the probability that the target individual retains itself is , the probability of selecting an individual from the elite pool is , and the probability of selecting an individual from the remaining evolution pool is , satisfying the following formula:

[0061] ;

[0062] Step 4.4: The mutation operation is performed on all individuals in the remaining evolution pool. Mutation means adjusting the value of the parameters of an individual by adding noise based on the standard normal distribution is the value of the parameter of individual participating in the mutation, is the variance; each individual triggers a mutation event according to the mutation probability . When a mutation is triggered, for the multi-dimensional parameter vector of the individual, according to the preset ratio of the parameters of the normal mutation and the ratio of the parameters of the super-strong mutation , parameters are selected by sampling without replacement to perform normal mutation and parameters are selected to perform super-strong mutation; where is the number of parameters of individual , is rounding up; define that the variance parameter of the normal mutation is less than the variance parameter of the super-strong mutation, and Satisfy the following formula:

[0063] .

[0064] Furthermore, the specific process of step 5 is as follows: Completing one round of step 3 and step 4 is called one round. In each round, step 3 is executed times, and step 4 is executed once; After step 4 ends, the individual with the highest fitness replaces the reinforcement learning individual in step 3, and the tolerance threshold and mutation probability change with the number of rounds. The formula is:

[0065] ;

[0066] ;

[0067] In the formula, is the minimum value of the tolerance threshold; is the maximum value of the tolerance threshold; is the minimum value of the mutation probability; is the maximum value of the mutation probability; is the current number of rounds; is the maximum number of rounds;

[0068] Repeat step 3.1 to step 4.4 for a total of rounds to finally obtain the optimal horizontal well sectioning point position, valve opening, and optimal injection-production system plan.

[0069] The beneficial technical effects brought by the present invention: The present invention proposes a reinforcement learning method based on an evolutionary algorithm for the refined regulation and production system optimization of horizontal wells. By combining the evolutionary algorithm with a hybrid deep reinforcement learning agent model, the present invention can effectively optimize the well section division and valve opening of horizontal wells, adjust the production system of each well in real time, thereby improving the oil and gas recovery rate and quickly adapting to the dynamic changes of the reservoir. This method can accelerate the learning process, enhance the global search ability, and improve the economic benefits of oilfield exploitation through automated regulation, with significant technical advantages and wide application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 is the design flow chart of the reinforcement learning method for horizontal well regulation based on an evolutionary algorithm according to the present invention.

[0071] Figure 2 is the comparison diagram of the economic net present value convergence curves between the reinforcement learning method for horizontal well regulation based on an evolutionary algorithm according to the present invention and the theoretical maximum economic net present value of the horizontal well without sectioning and only regulating the production system.

[0072] Figure 3 This is a comparison chart of cumulative oil production between the horizontal well regulation reinforcement learning method assisted by the evolutionary algorithm and the theoretical maximum economic net present value of the horizontal well without segmentation but only with regulation system in the present invention.

[0073] Figure 4 This is a comparison chart of cumulative water production between the horizontal well regulation reinforcement learning method assisted by the evolutionary algorithm and the theoretical maximum economic net present value of the horizontal well without segmentation but only with regulation system in the present invention.

[0074] Figure 5 This is the optimized oil saturation and well location map of the reservoir in the present invention.

[0075] Figure 6 This is a schematic diagram of the optimized sectional point position of production well P1 in the present invention.

[0076] Figure 7 This is a schematic diagram of the optimized sectional point position of production well P2 in the present invention.

[0077] Figure 8 This is a schematic diagram of the optimized sectional point position of production well P3 in the present invention. Detailed implementation manners

[0078] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners:

[0079] Taking a certain reservoir model as an example in the present invention, the size of the model is 25*25*3, the porosity is 0.2, the initial pressure is 6000 psi, and the initial water saturation is 0.2. There are a total of three production wells (numbered P1~P3) and four injection wells (numbered I1~I4), and all production wells are horizontal wells. The possible sectional point positions of production wells P1, P2, and P3 are 9, 10, and 9 respectively, and the maximum number of sectional points for each well is 3, 4, and 3 respectively. The upper limit of the valve opening is 100%, and the lower limit is 0%. The upper limit of the production rate of the production well is 178.61 m 3 / d, and the lower limit is 0 m 3 / d. The injection rate of each injection well is 178.61 m 3 / d.

[0080] As Figure 1 shown, the horizontal well regulation reinforcement learning method assisted by the evolutionary algorithm specifically includes the following steps:

[0081] Step 1, determine the sectional point positions, valve openings, and injection-production variables to be optimized, and construct a hybrid action space combining discrete and continuous regulation measures; the specific process is as follows:

[0082] Step 1.1. Determine the horizontal well section points, valves, and the number of oil and water wells (including production wells and injection wells) to be regulated. The variables to be optimized include the section point positions, valve openings, and injection-production variables (including production rate and injection rate), which can be specifically expressed as:

[0083] ;

[0084] In the formula, is the variable to be optimized at the regulation time step ; the number of all possible section points is 28; the number of valves is 10; the number of optimized production wells is 3; the number of optimized injection wells is 0; is the regulation time step. Each regulation time step is an optimization cycle, and there are a total of 5 regulation time steps; is the position of the th section point at the current time step, taking values of 0 or 1, which is only valid for the first time step; is the opening of the th valve at the current time step, %; is the production rate of the th production well at the current time step, m 3 / d; is the injection rate of the th injection well at the current time step, 178.61 m 3 / d.

[0085] Step 1.2. Construct a discrete action space which is:

[0086] ;

[0087] Set to represent the dimension of the discrete action space, with a total of dimensions;

[0088] Step 1.3. Construct a continuous action space which is:

[0089] ;

[0090] Set to represent the dimension of the continuous action space, with a total of dimensions;

[0091] Step 1.4. The hybrid action space combining discrete and continuous regulation measures can be expressed as:

[0092] ;

[0093] In the formula, is the mixed action space.

[0094] Step 2: Construct a hybrid deep reinforcement learning agent model and a prioritized replay buffer for recent experiences, which specifically includes the following steps:

[0095] The hybrid deep reinforcement learning agent model includes a policy network, an action-value network, and a target action-value network; among them, the policy network consists of two parts: a discrete policy network and a continuous policy network. The discrete policy network consists of a convolutional neural network and a fully connected neural network. The input is the reservoir saturation field and pressure field. The data first extracts the reservoir data features through the convolutional neural network, and then outputs the discrete action probability distribution through the fully connected neural network; the continuous policy network consists of a convolutional neural network and a fully connected neural network. The input is the reservoir saturation field, pressure field, and discrete action probability distribution. The reservoir saturation field and pressure field first extract the reservoir data features through the convolutional neural network. After the extracted reservoir data features are concatenated with the discrete action probability distribution, they are input into the fully connected neural network, and the Gaussian distribution of the continuous action is output;

[0096] The network structures of the action-value network and the target action-value network are the same, and both consist of a convolutional neural network and a fully connected neural network. The input of the action-value network is the reservoir saturation field, pressure field, and continuous action at a given time step. The input of the target action-value network is the reservoir saturation field, pressure field, and continuous action at the next time step of a given time step. The reservoir saturation field and pressure field first extract the reservoir data features through the convolutional neural network. After the extracted reservoir data features are concatenated with the continuous action, they are input into the fully connected neural network. The action-value network outputs the Q value of the action at a given time step, and the target action-value network outputs the Q value of the action at the next time step of a given time step;

[0097] Construct a prioritized replay buffer for recent experiences , which is used to store the experience samples generated during the agent's exploration; the sampling ratio factor and the actual sampling range of the prioritized replay buffer for recent experiences are as follows:

[0098] ;

[0099] ;

[0100] In the formula, is the sampling ratio factor; is the current number of updates; is the total number of updates, which is 50,000 times; is the hyperparameter for controlling the sampling bias, taking 0.4; is the actual sampling range; is the number of experience in the current experience pool. The sampling range of the subsequent step 3 is the latest data entries.

[0101] Step 3: Interact the hybrid deep reinforcement learning agent model with the numerical simulator, and train the agent through the generated experience samples. In the present invention, the agent is a reinforcement learning individual. The specific steps are as follows:

[0102] Step 3.1: Set the reward value at the current time step as the economic net present value at the current time step:

[0103] ;

[0104] In the formula, is the current time step; is the reward value at the current time step; is the economic net present value at the current time step; the number of production wells is 3; the number of injection wells is 4; is the unit price of crude oil sales, 3522 yuan / m 3 ; is the production well water treatment cost, 220 yuan / m 3 ; is the injection well water injection cost, 132 yuan / m 3 ; 、 are respectively the daily oil production and daily water production of the th production well at the current time step, m 3 / d; is the th injection well water injection rate, 178.61 m 3 / d.

[0105] Step 3.2: Input the saturation field and pressure field data of the reservoir model at the current time step as the state into the policy network. The policy network outputs the discrete action at the current time step and the continuous action at the current time step; Calculate the reward value at the current time step and the state at the next time step through the numerical simulator. If the current time step is the last time step, the end flag is 1, otherwise it is 0. Save the trajectory of the experience sample in the recent experience prioritized replay pool .

[0106] Step 3.3: Use the recent experience prioritized replay pool The training action value loss of the empirical samples therein is specifically as follows:

[0107] ;

[0108] ;

[0109] ;

[0110] In the formula, is the Q value of the target action; is the discount factor, indicating the relative importance of the current reward and future rewards, with a value of 0.99; represents the discrete action probability distribution; represents the state at the next time step; is to find the minimum value; is the th target action value network. In the present invention, two target action value networks and two action value networks are set. The first target action value network corresponds to the first action value network, and the second target action value network corresponds to the second action value network. Therefore, here ; represents the continuous action at the next time step; represents the temperature coefficient of the continuous action; represents the continuous action probability distribution; represents the discrete action at the next time step; represents the temperature coefficient of the discrete action; is the th loss function of the action value network; represents calculating the expectation; is the action value network; represents the total loss function of the action value network;

[0111] Step 3.4. Use the experience samples in the recent experience prioritized replay pool to train the policy network parameters, specifically as follows:

[0112] ;

[0113] ;

[0114] ;

[0115] Among them, [[ID=scribd]]is the loss function of the discrete policy network; is to calculate the expectation; is the loss function of the continuous policy network; is the total loss function of the policy network;

[0116] Step 3.5: Design the loss functions for the temperature coefficients of continuous actions and discrete actions, and update the temperature coefficients of continuous actions and discrete actions and as follows:

[0117] ;

[0118] ;

[0119] where is the loss function of the temperature coefficient of continuous actions; is the loss function of the temperature coefficient of discrete actions; and represent the entropies of the continuous policy network and the discrete policy network respectively; and represent the target entropies of the continuous policy network and the discrete policy network respectively, with values of -13 and 0.416.

[0120] Step 4: Perform the replacement, selection, crossover, and mutation operations of the evolutionary algorithm to optimize the hybrid deep reinforcement learning agent model. Specifically, it includes the following steps:

[0121] Step 4.1: The evolutionary algorithm has individuals, which is 19, and each individual is a policy network. The sum of all rewards of a policy network in one episode is recorded as the initial fitness value of the individual. Set a penalty term. If the continuous action output by the policy network is exactly equal to the boundary value, it is considered that the network has a risk of damaged network structure, and the fitness value of the network is deducted accordingly. Calculate the fitness values of all individuals, and the calculation formula is:

[0122] ;

[0123] In the formula, is the fitness value of individual ; is the initial fitness value of individual ; is the penalty term of individual , and the penalty term increases by 1 for each boundary value taken by the individual, with an initial value of 0.

[0124] Calculate the minimum tolerance of all individuals:

[0125] ;

[0126] In the formula, is individual The lowest tolerance; For an individual The fitness value of the previous round; Is the tolerance threshold, and the value range is ;

[0127] If the fitness value of the current individual is the lowest or lower than the individual's lowest tolerance, it is replaced by a reinforcement learning individual;

[0128] Step 4.2: The individual with the highest fitness value directly enters the winning pool, and then the roulette wheel algorithm is used to select several individuals to enter the winning pool. Individuals with higher fitness values have a greater probability of being selected. The probability calculation formula is as follows:

[0129] ;

[0130] ;

[0131] In the formula, Is the fitness value of individual After normalization; Is the fitness value; Is the minimum value among the fitness values of all individuals; Is a constant, with a very small value, to avoid the normalized fitness value being zero, and take 1e-8. For individual The probability of being selected; Is the sum of the normalized fitness values of all individuals.

[0132] Step 4.3: All individuals that do not enter the winning pool enter the remaining evolution pool, and the three-source crossover operation is performed on all individuals in the remaining evolution pool. This operation performs crossover on parameters such as convolutional layers, fully connected layers, and bias terms, where the convolution kernel, row, and element are used as the basic units of crossover respectively. During the crossover process of each basic unit, the probability that the target individual retains itself is The probability of selecting an individual from the winning pool is The probability of selecting an individual from the remaining evolution pool is Satisfies the following formula:

[0133] ;

[0134] Among them, Is 0.5, Is 0.3, Is 0.2.

[0135] Step 4.4: The mutation operation is performed on all individuals in the remaining evolution pool. Mutation means adjusting its value by adding noise Based on the standard normal distribution to the parameters of the individual, For an individual The value of the parameter participating in mutation is the variance, representing the average squared distance by which the data deviates from the mean. Each individual triggers a mutation event according to the mutation probability When a mutation is triggered, for the multi-dimensional parameter vector of that individual, according to the ratio of the parameters of the preset normal mutation and the ratio of the parameters of the super-strong mutation selects respectively by sampling without replacement parameters to perform normal mutation and parameters to perform super-strong mutation. Among them is the individual the number of parameters of is rounding up, the variance parameter of normal mutation is less than the variance parameter of super-strong mutation , and satisfy the following formula

[0136] ;

[0137] wherein is 0.2, is 10, is 0.1, is 0.02.

[0138] Step 5. Repeat Steps 3 to 4 until the maximum number of rounds is reached. Specifically, it includes the following steps

[0139] Completely executing one round of Steps 3 and 4 is called one round. In each round, Step 3 is executed times, and Step 4 is executed once, is 200. After Step 4 ends, the individual with the highest fitness replaces the reinforcement learning individual in Step 3, and the tolerance threshold and the mutation probability change with the number of rounds. The formula is

[0140] ;

[0141] ;

[0142] In the formula is the minimum value of the tolerance threshold, which is 0.6; is the maximum value of the tolerance threshold, which is 0.9; is the minimum value of the mutation probability, which is 0.1; is the maximum value of the mutation probability, which is 0.8; is the current number of rounds; is the maximum number of rounds, which is 250.

[0143] Repeat steps 3.1 to 4.4 for a total of rounds to finally obtain the optimal horizontal well sectional point position, valve opening, and optimal injection and production regime plan.

[0144] The comparative analysis of the final optimization results is as follows:

[0145] Figure 2 It is a comparative graph of the economic net present value convergence curves of the theoretical maximum economic net present value of the horizontal well regulation and control enhanced learning method assisted by an evolutionary algorithm (abbreviated as evolutionary-assisted reinforcement learning) and the horizontal well without sectionalization but only regulation and control regime in the embodiments of the present invention. According to the curve convergence situation, it can be seen that the method proposed by the present invention can obtain a higher economic net present value within the same regulation and control time steps.

[0146] Figure 3 It is a comparative graph of the cumulative oil production of the theoretical maximum economic net present value of the horizontal well regulation and control enhanced learning method assisted by an evolutionary algorithm (abbreviated as evolutionary-assisted reinforcement learning) and the horizontal well without sectionalization but only regulation and control regime in the embodiments of the present invention. It can be seen that the method proposed by the present invention has a larger cumulative oil production, effectively realizing oil production increase.

[0147] Figure 4 It is a comparative graph of the cumulative water production of the theoretical maximum economic net present value of the horizontal well regulation and control enhanced learning method assisted by an evolutionary algorithm (abbreviated as evolutionary-assisted reinforcement learning) and the horizontal well without sectionalization but only regulation and control regime in the embodiments of the present invention. It can be seen that the method proposed by the present invention has less cumulative water production, effectively realizing water control.

[0148] Figure 5 It is the oil saturation and well location map of the reservoir after optimization, showing the specific positions of three production wells (numbered P1 - P3) and four injection wells (numbered I1 - I4).

[0149] Figure 6 、 Figure 7 、 Figure 8 They are respectively schematic diagrams of the sectional point positions of production well P1, production well P2, and production well P3 after optimization. The coordinates of each grid are shown in the figure. For example, (4, 4, 1) is the coordinate of a grid. From Figures 6 - 8 it can be seen that each well has been effectively sectioned and all satisfy the constraint of the maximum number of sections per well.

[0150] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions, or substitutions made by those skilled in the art within the essence of the present invention should also fall within the protection scope of the present invention.

Claims

1. A horizontal well control reinforcement learning method assisted by an evolutionary algorithm, characterized in that It includes the following steps: Step 1: Determine the positions of the segmentation points, valve openings, and injection-production variables to be optimized, and construct a hybrid action space that combines discrete and continuous control measures; the injection-production variables include production rate and injection rate; Step 2: Construct a hybrid deep reinforcement learning agent model and a prioritized experience replay pool for recent experiences; The hybrid deep reinforcement learning agent model includes a policy network, an action-value network, and a target action-value network; The policy network consists of two parts: a discrete policy network and a continuous policy network; the discrete policy network consists of a convolutional neural network and a fully connected neural network. The input is the reservoir saturation field and pressure field. The data first extracts the reservoir data features through the convolutional neural network, and then outputs the discrete action probability distribution through the fully connected neural network; the continuous policy network consists of a convolutional neural network and a fully connected neural network. The input is the reservoir saturation field, pressure field, and discrete action probability distribution. The reservoir saturation field and pressure field first extract the reservoir data features through the convolutional neural network. After the extraction, the reservoir data features are concatenated with the discrete action probability distribution and then input into the fully connected neural network to output the Gaussian distribution of the continuous action; The network structures of the action-value network and the target action-value network are the same, and both consist of a convolutional neural network and a fully connected neural network; the input of the action-value network is the reservoir saturation field, pressure field, and continuous action at a given time step, and the input of the target action-value network is the reservoir saturation field, pressure field, and continuous action at the next time step of a given time step; the reservoir saturation field and pressure field first extract the reservoir data features through the convolutional neural network. After the extraction, the reservoir data features are concatenated with the continuous action and then input into the fully connected neural network. The action-value network outputs the Q value of the action at a given time step, and the target action-value network outputs the Q value of the action at the next time step of a given time step; Step 3: Interact the hybrid deep reinforcement learning agent model with the numerical simulator, and train the agent through the generated experience samples; Step 4: Perform replacement, selection, crossover, and mutation operations of the evolutionary algorithm to optimize the hybrid deep reinforcement learning agent model; Step 5: Repeat Steps 3 to 4 until the maximum number of episodes is reached.

2. The horizontal well control reinforcement learning method assisted by an evolutionary algorithm according to claim 1, wherein The specific process of Step 1 is as follows: Step 1.1: Determine the horizontal well segmentation points, valves, and the number of oil production and injection wells to be controlled according to the target reservoir. The variables to be optimized include the positions of the segmentation points, valve openings, and injection-production variables. The oil production and injection wells include production wells and injection wells; the injection-production variables are specifically expressed as: ； Wherein, is the variable to be optimized at the control time step ; is the position of the th segmentation point at the current time step; is the opening degree of the th valve at the current time step; is the production rate of the th production well at the current time step; is the injection rate of the th injection well at the current time step; Step 1.

2. Construct a discrete action space It is as follows: ； Step 1.3, construct a continuous action space It is: ； The hybrid action space that combines discrete and continuous control measures is expressed as: ； In the formula, is the mixed action space.

3. The horizontal well control reinforcement learning method assisted by an evolutionary algorithm according to claim 2, wherein In the above-mentioned step 2, a recent experience prioritized replay pool is constructed The sampling scale factor and the actual sampling range are as follows: ； ； Wherein, is the sampling scale factor; is the current number of updates; is the total number of updates; is the hyperparameter for controlling sampling bias; is the actual sampling range; is the number of experiences in the current experience pool.

4. The horizontal well control enhanced learning method assisted by an evolutionary algorithm according to claim 3, characterized in that In Step 3, the agent is a reinforcement learning individual; the specific process of Step 3 is as follows: Step 3.1: Set the reward value at the current time step as the economic net present value at the current time step: ； Wherein, is the current time step; is the reward value at the current time step; is the economic net present value at the current time step; is the number of production wells; is the number of injection wells; is the unit price of crude oil sales; is the treatment cost of produced well water; is the injection cost of injection wells; and are respectively the daily oil production and daily water production of the th production well at the current time step; is the th injection rate of the injection well; Step 3.2: Use the saturation field and pressure field data of the reservoir model at the current time step as the state and input them into the policy network. The policy network outputs the discrete action at the current time step and the continuous action at the current time step ; Calculate the reward value at the current time step and the state at the next time step using a numerical simulator. If the current time step is the last time step, the end flag is 1, otherwise it is 0; Save the trajectory of the experience sample in the recent experience prioritized replay pool . Step 3.3: Use the recent experience priority replay pool to train the action value loss with the experience samples, specifically as follows: ； ； ； Wherein, is the Q value of the target action; is the discount factor; represents the discrete action probability distribution; is to find the minimum value; is the th target action value network; represents the continuous action at the next time step; represents the temperature coefficient of the continuous action; represents the continuous action probability distribution; represents the discrete action at the next time step; represents the temperature coefficient of the discrete action; is the th loss function of the action value network, and the serial number of the action value network corresponds to that of the target action value network; represents calculating the expectation; is the action value network; represents the total loss function of the action value network; Step 3.4: Use the recent experience prioritized replay pool to train the parameters of the policy network with the experience samples, specifically as follows: ； ； ； Among them, is the loss function of the discrete policy network; is used to calculate expectation; is the loss function of the continuous policy network; is the total loss function of the policy network; Step 3.

5. Design a loss function for the temperature coefficients of continuous actions and discrete actions, and update and as follows: and ​ ； ； Among them, is the loss function of the continuous action temperature coefficient; is the loss function of the discrete action temperature coefficient; and respectively represent the entropies of the continuous policy network and the discrete policy network; and respectively represent the target entropies of the continuous policy network and the discrete policy network.

5. The horizontal well control reinforcement learning method assisted by an evolutionary algorithm according to claim 4, characterized in that The specific process of Step 4 is as follows: Step 4.

1. The evolutionary algorithm has a total of individuals, and each individual is a policy network; the sum of all rewards within one episode of the policy network is recorded as the initial fitness value of the individual; a penalty term is set, and if the continuous actions output by the policy network are equal to the boundary values, the fitness value of the policy network is correspondingly deducted; calculate the fitness values of all individuals, and the formula is: ； wherein, is the fitness value of an individual ; is the initial fitness value of an individual ; is the penalty term of an individual . For each boundary value taken by the individual, the penalty term is incremented by 1; Calculate the minimum tolerance of all individuals: ； In the formula, is the minimum tolerance of individual ; is the fitness value of the previous round of individual ; is the tolerance threshold; If the fitness value of the current individual is the lowest or lower than the minimum tolerance of the individual, then replace the current individual with the reinforcement learning individual; Step 4.2: The individual with the highest fitness value directly enters the winning pool, and then the roulette wheel algorithm is used to select several individuals to enter the winning pool. The probability of selecting an individual with a high fitness value is high, and the probability calculation formula is as follows: ； ； Wherein, is the individual after normalized fitness value; is the fitness value; is a constant; is the probability that the individual is selected; Step 4.3: All individuals that do not enter the winning pool enter the remaining evolution pool, and the three-source crossover operation is performed on all individuals in the remaining evolution pool; this operation performs crossover on the convolutional layer, fully connected layer, and bias term, where the convolution kernel, row, and element are used as the basic units of crossover respectively; during the crossover process of each basic unit, the probability that the target individual retains itself is , the probability of selecting an individual from the winning pool is , the probability of selecting an individual from the remaining evolution pool is , satisfying the following formula: ； Step 4.4: Perform mutation operations on all individuals in the remaining evolutionary pool. Mutation means adjusting the values of the parameters of an individual by adding noise based on the standard normal distribution to adjust its value, for the individual the value of the parameter participating in mutation, is the variance; each individual triggers a mutation event according to the mutation probability When a mutation is triggered, for the multi-dimensional parameter vector of the individual, according to the proportion of the parameters of the preset normal mutation and the proportion of the parameters of the super-strong mutation respectively select through sampling without replacement parameters to perform normal mutation and parameters to perform super-strong mutation; where is the individual the number of parameters, is rounding up; define the variance parameter of the normal mutation is less than the variance parameter of the super-strong mutation , and satisfy the following formula: 。 6. The method for enhancing reinforcement learning of horizontal well control assisted by evolutionary algorithm according to claim 5, wherein The specific process of step 5 is as follows: Completing one round of step 3 and step 4 is called a round. In each round, step 3 is executed times, and step 4 is executed once; after step 4 ends, the individual with the highest fitness replaces the reinforcement learning individual in step 3, and the tolerance threshold and mutation probability change with the number of rounds. The formula is: ； ； Wherein, is the minimum tolerance threshold; is the maximum tolerance threshold; is the minimum mutation probability; is the maximum mutation probability; is the current round number; is the maximum round number; Repeat steps 3.1 to 4.4 for a total of rounds to finally obtain the optimal horizontal well sectional point position, valve opening, and optimal injection-production system plan.

Images