Horizontal subdivision capacity optimization method based on adaptive entropy deep reinforcement learning

Abstract

The invention provides a horizontal subdivision productivity optimization method based on adaptive entropy deep reinforcement learning, and belongs to the field of oil and gas field development engineering, and the method comprises the following steps: constructing an adaptive entropy deep reinforcement learning strategy model, interacting with a horizontal well subdivision oil production reservoir shaft numerical simulator, storing horizontal well subdivision experience, and optimizing the horizontal well subdivision productivity. Training and updating the adaptive entropy deep reinforcement learning strategy model by using historical subdivision experience; the strategy model is applied, the oil reservoir state is input, and a subdivision optimization scheme at the current moment is rapidly output. Compared with an existing horizontal well subdivision oil extraction optimization method, the method has the advantages that a sequence subdivision optimization scheme can be quickly output, strategy coefficients are automatically adjusted, and the capacity of strategy exploration for global optimum is effectively improved. Meanwhile, the method is combined with an oil reservoir shaft coupling model, it is ensured that an optimization strategy is tightly attached to the actual physical process, the accuracy of yield evaluation is improved, and the purpose of rapid optimization of the horizontal well sequence subdivision scheme is achieved.

Description

Technical Field

[0001] The present invention relates to the field of oil and gas field development engineering, and in particular to a method for optimizing horizontal compartment production capacity based on adaptive entropy deep reinforcement learning. Background Art

[0002] In the field of modern oil production, horizontal well technology has become a key means to improve the recovery rate of low-permeability, thin-layer and complex-structure reservoirs. In particular, compartmented horizontal well design significantly increases the reservoir contact area by arranging multiple production horizontal sections in a single wellbore, realizing the effective development of complex reservoir structures. However, the optimization of horizontal well compartment design is a very challenging task, involving the precise configuration of the length and center point position of each production horizontal section to maximize the overall production of the horizontal well. In traditional methods, reservoir engineers use geological models, reservoir simulation software and empirical formulas to determine compartment parameters through trial and error or analytical methods based on physical laws. However, such methods are often limited by model accuracy, computing resources and insufficient characterization of complex geological nonlinear characteristics, making it difficult to achieve a global optimal solution. In addition, factors such as the uncertainty of reservoir dynamics, the variability of fluid properties and the interaction between the wellbore and the formation make it difficult for fixed-mode design strategies to adapt to changes in actual production conditions.

[0003] With the widespread application of artificial intelligence technology in the oil and gas industry, especially the excellent performance of reinforcement learning methods in complex decision-making problems, new solutions have been provided for the optimization of horizontal well compartments. Although preliminary attempts have demonstrated the potential of continuous reinforcement learning in reservoir development decisions, existing applications still have limitations. The processing efficiency of existing algorithms for large-scale state spaces needs to be improved, especially when facing large reservoirs with a large number of compartment possibilities, which may lead to slow convergence or fall into local optimality. At the same time, existing studies often ignore the spatiotemporal heterogeneity of reservoir characteristics, and do not fully consider the dynamic coupling relationship between the wellbore and the reservoir and the non-uniform characteristics of fluid flow, which may affect the universality and robustness of the strategy. Therefore, realizing the intelligent optimization of horizontal well compartment production is still an urgent problem to be solved. Summary of the invention

[0004] In view of the limitations of existing model-based and experience-based methods in dealing with complex nonlinear problems, as well as the problems of reinforcement learning in dealing with large-scale continuous decision-making and considering spatiotemporal heterogeneity and fluid inhomogeneity, the present invention proposes a horizontal compartment production capacity optimization method based on adaptive entropy deep reinforcement learning. By intelligently optimizing the length and center point position of horizontal well compartments, it dynamically adapts to reservoir changes, thereby effectively improving horizontal well production, providing an advanced and practical technical solution for the efficient development of complex reservoirs.

[0005] To achieve the above object, the technical solution of the present invention is implemented as follows: a method for optimizing horizontal subdivision capacity based on adaptive entropy deep reinforcement learning, comprising the following steps:

[0006] S1, determine the number of horizontal well compartments at each time step during the development of the target reservoir and the constraints on the reservoir boundary and production horizontal well compartments;

[0007] S2, initialize the adaptive entropy deep reinforcement learning strategy model;

[0008] S3, combining the constraints of production horizontal well compartmentation and the continuous action space of horizontal well compartmentation, the strategy model interacts with the numerical simulator, inputs the reservoir state, outputs the horizontal well compartmentation plan at the current moment, collects compartmentation experience and saves it to the historical experience library;

[0009] S4, using the historical compartment experience training in the historical experience library to update the adaptive entropy strategy model;

[0010] S5, repeat the above steps until the adaptive entropy strategy model converges and saves the adaptive entropy strategy model, applies the adaptive entropy strategy model, and quickly outputs the sequence compartment optimization plan given the reservoir state.

[0011] Further, step S1 includes the following steps:

[0012] S11, set the number of horizontal well compartments m and the minimum spacing L of horizontal well compartments at each time step during the development process Δ ;

[0013] S12, based on the length L of the target horizontal well in the x direction and the distance z between the boundary of the target reservoir in the z direction and the target horizontal well w , set different compartment constraints for reservoir boundary and horizontal well section, which are shown in formula (1) and (2) respectively:

[0014] z h ∈[-z w ,z w ](1)

[0015]

[0016] In the formula, x i and L i denote the center position and length of the ith compartment, z h Indicates the position of the horizontal well in the z direction.

[0017] Further, in step S2, the adaptive entropy deep reinforcement learning strategy model is composed of a compartment strategy network, a strategy value measurement network and a target strategy value measurement network; step S2 includes the following steps:

[0018] S21, the compartment strategy network is a neural network π parameterized by φ φ (a t |s t ) represents the reservoir state s t is input, output continuous action a t ; The reservoir state s t Contains the wellbore center pressure at the current time step of the reservoir model and the flow pressure at the center of the i-th horizontal wellbore As shown in formula (3), the continuous action a t Including the optimized horizontal well compartment center position x and length L at the current time step, as shown in formula (4):

[0019]

[0020]

[0021] In the formula, s t represents the state of the reservoir at the tth time step; and They represent the wellbore center pressure and the flow pressure at the center of the i-th horizontal wellbore at the t-th time step, respectively; a t represents the continuous action at the tth time step; and They represent the center position and length of the compartment of the i-th horizontal well at the t-th time step respectively; m represents the number of compartments in each time step;

[0022] S22, the policy value metric network is a neural network Q parameterized by θ θ (s t ) represents the reservoir state s t As input, it outputs the Q value of continuous actions; the target strategy value measurement network is composed of Parameterized Neural Networks Indicated by reservoir state s t As input, it outputs the target Q value of continuous actions. Q represents the expected reward or value of taking a specific action in a given state and is dimensionless.

[0023] S23, set the discount factor γ, the strategy entropy factor α, and the target entropy π φ (a t |s t ), Q θ (s t ) and the learning rate λ of α φ , θ and λ α , and the number of training steps N in a production cycle step, reward r t is the horizontal well production at the current time step, as shown in formula (5):

[0024]

[0025] In the formula, r t represents the reward at the tth time step; q i Indicates the first i The flow rate of the segment, q i It is obtained from the wellbore coupling model of horizontal well compartment production reservoir, as shown in formula (6):

[0026]

[0027] In the formula, and represents the potential of the center point of the horizontal wellbore and the center point of the wellbore, respectively. μ, k, represent the viscosity and permeability of the reservoir, respectively. p e and p wj represent the pressure at the drainage boundary and the flow pressure at the midpoint of the jth horizontal wellbore, G represents the start-up pressure gradient, r e and r w Indicates the oil drainage radius and the wellbore radius.

[0028] Further, step S3 includes the following steps:

[0029] S31, perform Gaussian distribution sampling on the continuous action space and output the continuous action a t , including the center position and length of the horizontal well compartment at the current time step;

[0030] S32, judge a t Whether the compartment constraint conditions set in step S1 are met, if not, repeat S31, if the constraint conditions are met, proceed to the next step S33;

[0031] S33, apply the obtained compartment center position and length to the reservoir model, and use the numerical simulator to calculate the reward r at time t t , and obtain the reservoir state s at the next time step t+1 , according to the development requirements, set linear constraints on the production capacity of different compartments of horizontal wells q i ∈[Q imin ,Q imax ], and set the pressure drop constraint ΔP for each compartment i <ΔP max , while considering nonlinear constraints including the water content or formation pressure of the reservoir, if the constraints are met, proceed to the next step S34, if not, repeat S31; q irepresents the flow rate from the oil layer into the i-th section of the horizontal wellbore, m 3 / d;Q imin represents the minimum flow constraint corresponding to the i-th section of the horizontal well, m 3 / d;Q imax represents the maximum flow constraint corresponding to the i-th section of the horizontal well, m 3 / d; △P i represents the pressure drop in the i-th section of the horizontal well, MPa; △P max Indicates the maximum pressure drop constraint, MPa;

[0032] S34, collect compartment experience (s t ,a t ,r t ,s t+1 ) and saved in the historical experience library.

[0033] Further, step S4 includes the following steps:

[0034] S41, based on the historical experience in the historical experience library, the parameters θ of the strategy value measurement network are trained and updated by minimizing the Q value loss function, as shown in formulas (7), (8), and (9):

[0035]

[0036]

[0037] θ←θ-λ θ ▽ θ J Q (θ)(9)

[0038] In the formula, represents the estimated value of the target state at time step t+1; J Q (θ) represents the Q-value loss function, which is used to measure the difference between the Q-value and the state estimate at the current time step t; and It means to find the expectation of the difference of Q values ​​calculated based on the samples in the historical experience library; MSE(·) means to take the mean square error; ▽ θ J Q (θ) represents the gradient of θ;

[0039] S42, based on the sample data in the historical experience library, the parameters φ of the compartment strategy network are trained and updated by minimizing the adaptive entropy strategy loss function, as shown in formulas (10) and (11):

[0040]

[0041] φ←φ-λ φ ▽ φ Jπ (φ)(11)

[0042] In the formula, J π (φ) represents the adaptive entropy strategy loss function, which is used to find the optimal adaptive entropy strategy;

[0043] ▽ φ J π (φ) means to find the gradient of φ;

[0044] S43, based on the sample data in the historical experience library, the policy entropy factor α is updated by minimizing the policy entropy factor loss function, as shown in formulas (12) and (13):

[0045]

[0046] α←α-λ α ▽ α J(α)(13) In the formula, J(α) represents the policy entropy factor loss function, which is used to update α to achieve automatic adjustment of entropy; Represents the target entropy, which is used to guide the update of α; ▽ α J(α) represents the gradient of α.

[0047] Further, step S5 includes repeating steps S3 and S4 until the adaptive entropy strategy model converges, saving the trained adaptive entropy strategy model, applying the adaptive entropy strategy model, given the reservoir state s t ′, quickly output the sequence compartment optimization plan a t '~π φ (s t ′).

[0048] Compared with the prior art, the horizontal compartment production capacity optimization method based on adaptive entropy deep reinforcement learning described in the present invention has the following advantages: Compared with the existing horizontal well oil production optimization method, the method provided by the present invention can learn from the historical experience of each round in the optimization process, and maximize the entropy of the strategy while maximizing the oil production by automatically adjusting the strategy entropy factor, so as to improve the strategy exploration ability. At the same time, the method proposed by the present invention is combined with the reservoir wellbore coupling model to ensure that the optimization strategy is closely aligned with the actual physical process, thereby improving the accuracy of production evaluation. The method proposed by the present invention can not only improve the ability of global optimization, but also improve the accuracy by combining the coupling relationship of the actual physical process, thereby achieving the purpose of rapid optimization of the horizontal well sequence compartment scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0050] Figure 1 It is a design flow chart of the horizontal subdivision capacity optimization method based on adaptive entropy deep reinforcement learning described in the present invention;

[0051] Figure 2 It is a schematic diagram of the framework of the horizontal subdivision capacity optimization method based on adaptive entropy deep reinforcement learning described in the present invention;

[0052] Figure 3 It is a comparison chart of oil field production curves of the horizontal compartment capacity optimization method based on adaptive entropy deep reinforcement learning and the optimization schemes of genetic algorithm (GA) and particle swarm algorithm (PSO) in the embodiment of the present invention;

[0053] Figure 4 It is a comparison result of oil production curves of different compartments in the horizontal compartment capacity optimization method based on adaptive entropy deep reinforcement learning in an embodiment of the present invention;

[0054] Figure 5 It is a curve showing the change of the strategy entropy factor with the number of numerical simulations in the process of optimizing the horizontal well compartment production based on adaptive entropy deep reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0056] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0057] A reservoir model was used as an example for testing and compared with the traditional optimization method. The reservoir model is a low-permeability closed reservoir with a model starting pressure gradient of 0.001MPa / m, a supply pressure of 30MPa, a bottom hole flow pressure of 15MPa, and a horizontal permeability of 0.01μm. 2 , vertical permeability is 0.005μm 2 , horizontal well length is 1000m, oil layer thickness is 20m, wellbore radius is 0.1m, supply radius is 1000m, density is 840kg / m 3 , the crude oil volume coefficient is 1.05.

[0058] like Figure 1-Figure 2 As shown, the present invention proposes a horizontal compartment capacity optimization method based on adaptive entropy deep reinforcement learning. Based on the data of the reservoir model in the above embodiment, the specific steps are as follows:

[0059] S1, determine the number of horizontal well compartments at each time step during the development of the target reservoir and the constraints on the reservoir boundary and production horizontal well compartments;

[0060] Specifically, step S1 includes the following steps:

[0061] S11, set the number of compartments m at each time step in the development process and the minimum spacing L of horizontal well compartments Δ ;

[0062] S12, based on the length L of the target horizontal well in the x direction and the distance z between the boundary of the target reservoir in the z direction and the target horizontal well w , set different compartment constraints for reservoir boundary and horizontal well section, which are shown in formula (1) and (2) respectively:

[0063] z h ∈[-z w ,z w ](1)

[0064]

[0065] In the formula, x i and L i represents the center position and length of the ith compartment, z h Indicates the position of the horizontal well in the z direction.

[0066] S2, initialize the adaptive entropy deep reinforcement learning strategy model;

[0067] Specifically, step S2 includes the following steps:

[0068] S21, the compartment strategy network is a neural network π parameterized by φ φ (a t |s t ) represents the reservoir state s t is input, output continuous action a t ; The reservoir state s t Contains the wellbore center pressure at the current time step of the reservoir model and the flow pressure at the center of the i-th horizontal wellbore As shown in formula (3), the continuous action a t Including the optimized horizontal well compartment center position x and length L at the current time step, as shown in formula (4):

[0069]

[0070]

[0071] In the formula, s t represents the state of the reservoir at the tth time step; and They represent the wellbore center pressure and the flow pressure at the center of the i-th horizontal wellbore at the t-th time step, respectively; at represents the continuous action at the tth time step; and They represent the center position and length of the compartment of the i-th horizontal well at the t-th time step respectively; m represents the number of compartments in each time step.

[0072] S22, the policy value metric network is a neural network Q parameterized by θ θ (s t ) represents the reservoir state s t As input, it outputs the Q value of continuous actions; the target strategy value measurement network is composed of Parameterized Neural Networks Indicated by reservoir state s t As input, it outputs the target Q value of continuous actions. Q represents the expected reward or value of taking a specific action in a given state and is dimensionless.

[0073] S23, set the discount factor γ, the strategy entropy factor α, and the target entropy π φ (a t |s t ), Q θ (s t ) and the learning rate λ of α φ , θ and λ α , and the number of training steps N in a production cycle step , reward r t is the horizontal well production at the current time step, as shown in formula (5):

[0074]

[0075] In the formula, r t represents the reward at the tth time step; q i Indicates the first i The flow rate of the segment, q i It is obtained from the wellbore coupling model of horizontal well compartment production reservoir, as shown in formula (6):

[0076]

[0077] In the formula, and Represents the center point of the i-th horizontal wellbore and the center point of the wellbore respectively. Value, μ, k, represent the viscosity and permeability of the reservoir, respectively, p e and p wj represent the pressure at the drainage boundary and the flow pressure at the midpoint of the jth horizontal wellbore, G represents the start-up pressure gradient, r e and r wIt represents the oil drainage radius and the wellbore radius. By combining it with the reservoir-wellbore coupling model, it can not only improve the global optimization capability, but also improve the accuracy of production evaluation by combining the coupling relationship of the actual physical process, thus achieving the purpose of rapid optimization of the horizontal well sequence compartment scheme.

[0078] S3, combined with the constraints of production horizontal well compartmentation and the continuous action space of horizontal well compartmentation, the strategy model interacts with the numerical simulator, inputs the reservoir state, outputs the horizontal well compartmentation plan at the current moment, collects compartmentation experience and saves it to the historical experience library.

[0079] Specifically, step S3 includes the following steps:

[0080] S31, perform Gaussian distribution sampling on the continuous action space and output the continuous action a t , including the center position and length of the horizontal well compartment at the current time step;

[0081] S32, judge a t Whether the compartment constraints set in S1 are met, if not,

[0082] Then repeat S31, and if the constraint condition is satisfied, proceed to the next step S33;

[0083] S33, the obtained sub-cabin center position and length (continuous action a t ) is applied to the reservoir model and the reward r at time t is calculated using a numerical simulator t , and obtain the reservoir state s at the next time step t+1 , according to the development requirements, set linear constraints on the production capacity of different compartments of horizontal wells i ∈[Q imin ,Q imax ], and set the pressure drop constraint ΔP for each compartment i <ΔP max , while considering nonlinear constraints including the water content or formation pressure of the reservoir, if the constraints are met, proceed to the next step S34, if not, repeat S31; q i represents the flow rate from the oil layer into the i-th section of the horizontal wellbore, m 3 / d;Q imin represents the minimum flow constraint corresponding to the i-th section of the horizontal well, m 3 / d;Q imax represents the maximum flow constraint corresponding to the i-th section of the horizontal well, m 3 / d; △P i represents the pressure drop in the i-th section of the horizontal well, MPa; △P max Indicates the maximum pressure drop constraint, MPa;

[0084] S34, collect compartment experience (s t ,a t ,r t ,s t+1 ) and saved in the historical experience library.

[0085] S4, using the historical compartment experience training in the historical experience library to update the adaptive entropy strategy model;

[0086] Specifically, step S4 includes the following steps:

[0087] S41, by learning the historical experience accumulated in the historical experience library in each round of the optimization process, and minimizing the Q-value loss function, the parameters θ of the strategy value measurement network are trained and updated, as shown in formulas (7), (8), and (9):

[0088]

[0089]

[0090] θ←θ-λ θ ▽ θ J Q (θ)(9)

[0091] In the formula, represents the estimated value of the target state at time step t+1; J Q (θ) represents the Q-value loss function, which is used to measure the difference between the Q-value and the state estimate at the current time step t; and It means to find the expectation of the difference of Q values ​​calculated based on the samples in the historical experience library; MSE(·) means to take the mean square error; ▽ θ J Q (θ) represents the gradient of θ;

[0092] S42, based on the sample data in the historical experience library, the parameters φ of the compartment strategy network are trained and updated by minimizing the adaptive entropy strategy loss function, as shown in formulas (10) and (11):

[0093]

[0094] φ←φ-λ φ ▽ φ J π (φ)(11)

[0095] In the formula, J π (φ) represents the adaptive entropy strategy loss function, which is used to find the optimal adaptive entropy strategy;

[0096] ▽ φ J π(φ) means to find the gradient of φ;

[0097] S43, through the sample data in the historical experience library, minimizes the policy entropy factor loss function and realizes the automatic adjustment of the policy entropy factor α, as shown in formulas (12) and (13):

[0098]

[0099] α←α-λ α ▽ α J(α)(13)

[0100] Where J(α) represents the policy entropy factor loss function, which is used to update α and realize automatic adjustment of entropy; H represents the target entropy, which is used to guide the update of α; α J(α) represents the gradient of α.

[0101] S5, repeat the above steps until the adaptive entropy strategy model converges and saves the adaptive entropy strategy model, so as to maximize the entropy of the strategy while maximizing the oil recovery, so as to improve the strategy exploration ability, and finally achieve the application of the adaptive entropy strategy model, given the reservoir state, and quickly output the sequence compartment optimization plan;

[0102] Specifically, step S5 is to repeat steps S3 and S4 until the adaptive entropy strategy model converges, that is, the oil production tends to be stable, save the trained adaptive entropy strategy model for later use, and apply the saved adaptive entropy strategy model. Given the current reservoir state s t ′, output the compartment optimization solution a for the current time step t '~π φ (s t ′).

[0103] The final optimization results are compared and analyzed as follows:

[0104] Figure 3 This is a comparison chart of the oil field production curves of the horizontal compartment capacity optimization method based on adaptive entropy deep reinforcement learning in the embodiment of the present invention and the genetic algorithm (GA) and particle swarm algorithm (PSO) optimization schemes. The horizontal axis is the supply boundary distance, the vertical axis is the horizontal well production, and the number of horizontal well compartments is 4. It can be seen that compared with the scheme optimized by the GA and PSO methods, the method proposed by the present invention optimizes the length and center point position of the horizontal well compartments intelligently, dynamically adapts to reservoir changes, and achieves the maximum oil production at each supply boundary distance, effectively realizing oil increase.

[0105] Figure 4This is a comparison result of different compartment oil production curves of the horizontal compartment capacity optimization method based on adaptive entropy deep reinforcement learning in the embodiment of the present invention. The horizontal axis is the number of compartments, the vertical axis is the horizontal well production and the horizontal well production increment, and the supply boundary distance is 10m. It can be seen that the horizontal well production increases with the increase in the number of compartments, but it can be clearly seen that the production increase rate (production increment rate) gradually slows down with the increase in the number of compartments. Therefore, considering that the number of compartments is related to the cost in the actual application process, the number of compartments should not be too many.

[0106] Figure 5 This is a curve showing the change of the strategy entropy factor with the number of numerical simulations in the optimization process of horizontal well compartmented oil production based on adaptive entropy deep reinforcement learning in an embodiment of the present invention. The horizontal axis is the number of numerical simulation interactions, and the vertical axis is the strategy entropy factor. It can be seen that the strategy entropy factor gradually increases in the initial stage of the entire optimization process to encourage the strategy to explore more new spaces. When the strategy has explored the new space, the optimal strategy has been determined, and it is necessary to reduce the strategy exploration capability. By automatically reducing the strategy entropy factor, the compartment strategy network is guided to output the optimal compartment solution, thereby achieving efficient exploration of the strategy.

[0107] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A horizontal subdivision capacity optimization method based on adaptive entropy deep reinforcement learning, characterized in that: The following steps are involved: S1, determine the number of horizontal well compartments at each time step during the development of the target reservoir and the constraints on the reservoir boundary and production horizontal well compartments; S2, initialize the adaptive entropy deep reinforcement learning strategy model; S3, combining the constraints of production horizontal well compartmentation and the continuous action space of horizontal well compartmentation, the strategy model interacts with the numerical simulator, inputs the reservoir state, outputs the horizontal well compartmentation plan at the current moment, collects compartmentation experience and saves it to the historical experience library; S4, using the historical compartment experience training in the historical experience library to update the adaptive entropy strategy model; S5, repeat the above steps until the adaptive entropy strategy model converges and saves the adaptive entropy strategy model, applies the adaptive entropy strategy model, and quickly outputs the sequence compartment optimization plan given the reservoir state.

2. According to claim 1, a method for optimizing horizontal subdivision capacity based on adaptive entropy deep reinforcement learning is characterized in that: Step S1 includes the following steps: S11, set the number of horizontal well compartments m and the minimum spacing L of horizontal well compartments at each time step during the development process Δ ; S12, based on the length L of the target horizontal well in the x direction and the distance z between the boundary of the target reservoir in the z direction and the target horizontal well w , set different compartment constraints for reservoir boundary and horizontal well section, which are shown in formula (1) and (2) respectively: With h ∈[-z w ,With w ](1) In the formula, x i and L i Respectively represent the center position and length of the ith compartment, m, z h Represents the position of the horizontal well in the z direction, m.

3. The method for optimizing horizontal subdivision capacity based on adaptive entropy deep reinforcement learning according to claim 2 is characterized in that: In step S2, the adaptive entropy deep reinforcement learning strategy model is composed of a compartment strategy network, a strategy value measurement network, and a target strategy value measurement network; step S2 includes the following steps: S21, the compartment strategy network is a neural network π parameterized by φ φ (a t |s t ) represents the reservoir state s t is input, output continuous action a t ; The reservoir state s t Contains the wellbore center pressure at the current time step of the reservoir model and the flow pressure at the center of the i-th horizontal wellbore As shown in formula (3), the continuous action a t Including the optimized horizontal well compartment center position x and length L at the current time step, as shown in formula (4): In the formula, s t represents the state of the reservoir at the tth time step; and They represent the wellbore center pressure at the tth time step and the flow pressure at the center of the i-th horizontal wellbore, respectively; a t represents the continuous action at the tth time step; and They represent the center position and length of the compartment of the i-th horizontal well at the t-th time step respectively; m represents the number of compartments in each time step; S22, the policy value metric network is a neural network Q parameterized by θ θ (s t ) represents the reservoir state s t As input, it outputs the Q value of continuous actions; the target strategy value measurement network is composed of Parameterized Neural Networks Indicated by reservoir state s t As input, it outputs the target Q value of continuous actions. Q represents the expected reward or value of taking a specific action in a given state and is dimensionless. S23, set the discount factor γ, the strategy entropy factor α, and the target entropy π φ (a t |s t ), Q θ (s t ) and the learning rate λ of α φ , θ and λ α , and the number of training steps N in a production cycle step , reward r t is the horizontal well production at the current time step, as shown in formula (5): In the formula, r t represents the reward at the tth time step; q i Indicates the first i The flow rate of the segment, q i It is obtained from the wellbore coupling model of horizontal well compartment production reservoir, as shown in formula (6): In the formula, and represents the potential of the center point of the horizontal wellbore and the center point of the wellbore, respectively. μ, k, represent the viscosity and permeability of the reservoir, respectively. p e and p wj represent the pressure at the drainage boundary and the flowing pressure at the midpoint of the jth horizontal wellbore, G represents the starting pressure gradient, r e and r w Indicates the oil drainage radius and the wellbore radius.

4. The method for optimizing horizontal subdivision capacity based on adaptive entropy deep reinforcement learning according to claim 3 is characterized in that: Step S3 includes the following steps: S31, perform Gaussian distribution sampling on the continuous action space and output the continuous action a t , including the center position and length of the horizontal well compartment at the current time step; S32, judge a t Whether the compartment constraint conditions set in step S1 are met, if not, repeat S31, if the constraint conditions are met, proceed to the next step S33; S33, apply the obtained compartment center position and length to the reservoir model, and use the numerical simulator to calculate the reward r at time t t , and obtain the reservoir state s at the next time step t+1 , according to the development requirements, set linear constraints on the production capacity of different compartments of horizontal wells q i ∈[Q imin ,Q imax ], and set the pressure drop constraint ΔP for each compartment i <ΔP max , while considering nonlinear constraints including the water content or formation pressure of the reservoir, if the constraints are met, proceed to the next step S34, if not, repeat S31; q i represents the flow rate from the oil layer into the i-th section of the horizontal wellbore, m 3 / d;Q imin represents the minimum flow constraint corresponding to the i-th section of the horizontal well, m 3 / d;Q imax represents the maximum flow constraint corresponding to the i-th section of the horizontal well, m 3 / d; △P i represents the pressure drop in the i-th section of the horizontal well, MPa; △P max Indicates the maximum pressure drop constraint, MPa; S34, collect compartment experience (s t ,a t ,r t ,s t+1 ) and saved in the historical experience library.

5. The method for optimizing horizontal subdivision capacity based on adaptive entropy deep reinforcement learning according to claim 4 is characterized in that: Step S4 includes the following steps: S41, based on the historical experience in the historical experience library, the parameters θ of the strategy value measurement network are trained and updated by minimizing the Q value loss function, as shown in formulas (7), (8), and (9): In the formula, represents the estimated value of the target state at time step t+1; J Q (θ) represents the Q-value loss function, which is used to measure the difference between the Q-value and the state estimate at the current time step t; and It means to find the expectation of the Q value difference calculated based on the samples in the historical experience library; MSE(·) means to take the mean square error; It means to find the gradient of θ; S42, based on the sample data in the historical experience library, the parameters φ of the compartment strategy network are trained and updated by minimizing the adaptive entropy strategy loss function, as shown in formulas (10) and (11): In the formula, J π (φ) represents the adaptive entropy strategy loss function, which is used to find the optimal adaptive entropy strategy; It means to find the gradient of φ; S43, based on the sample data in the historical experience library, the policy entropy factor α is updated by minimizing the policy entropy factor loss function, as shown in formulas (12) and (13): Where J(α) represents the policy entropy factor loss function, which is used to update α to achieve automatic adjustment of entropy; Represents the target entropy, which is used to guide the update of α; It means to find the gradient of α.

6. The method for optimizing horizontal subdivision capacity based on adaptive entropy deep reinforcement learning according to claim 1 is characterized by: Step S5 includes repeating steps S3 and S4 until the adaptive entropy strategy model converges, saving the trained adaptive entropy strategy model, applying the adaptive entropy strategy model, and given the reservoir state s t ′, quickly output the sequence compartment optimization solution a t '~π φ (s t ′).

Images