Downhole tool face dynamic control method and system based on reinforcement learning

Abstract

The embodiment of the application provides a kind of based on reinforcement learning's downhole tool face dynamic control method and system, belong to directional drilling technical field.The method comprises: the drilling state data of target well is collected, data preprocessing is executed to the drilling state data, corresponding state vector is constructed based on the drilling state data after preprocessing;Action decision model trained based on DDPG algorithm is called based on the state vector, and action instruction is obtained;Wherein, the network strategy parameter of the action decision model is obtained based on the deviation between current drilling state data and pre-constructed reward function after correction of last round network strategy parameter;The action instruction is executed to parse, and target physical action is determined;Based on target physical action, top drive wobble is executed, and execution result is recovered, until execution result indicates that tool face angle meets expectation.The stability and real-time control effect under complex well section and severe disturbance working condition are improved in the application scheme.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of directional drilling, in particular to a downhole tool face dynamic control method based on reinforcement learning and a downhole tool face dynamic control system based on reinforcement learning. BACKGROUND

[0002] In the field of directional drilling, controlling the tool face angle is one of the core links to achieve precise guidance in sliding drilling. The existing tool face control technology mainly relies on the top drive wiggling system to adjust the direction of the downhole tool face by adjusting the angular displacement or torque of the drill string. However, the current mainstream technology still faces two technical bottlenecks, which seriously restricts its control accuracy and response ability in complex well sections.

[0003] The existing control system generally sets the top drive wiggling command based on a static model or fixed control parameters. Most systems rely on the preset theoretical length of the well section buffer zone or the drill string counter-torque response curve to estimate the control parameters, ignoring the dynamic interaction process between the drill string and the formation. In actual drilling, due to the uncertainty of wellbore curvature, drill string structure and formation characteristics, the friction and counter-torque experienced by the drill string often show highly nonlinear fluctuations, causing the preset model to deviate significantly from the actual response. This type of static modeling method is particularly unstable in well sections with severe formation changes or long sliding sections, easily causing tool face angle drift, control lag or excessive correction, and thus affecting trajectory control quality and drilling efficiency.

[0004] Most current systems use discrete sampling data as feedback, often collecting downhole angle and attitude information through MWD (Measurement While Drilling) equipment. However, the MWD system has the limitations of low sampling rate and large data uplink delay, with a sampling period of more than 30 seconds and a transmission delay of several tens of seconds to several minutes. Under these conditions, the state information obtained by the ground control system is severely lagging, unable to reflect real-time changes in downhole conditions, resulting in control commands being made based on outdated data, affecting the effectiveness and real-time performance of control, and even causing severe tool face oscillation or drill string overload due to control-feedback loop disruption.

[0005] Therefore, the current tool face control technology has obvious deficiencies in dealing with dynamic downhole environments and handling state feedback lags, and technical breakthroughs are needed in tool face response modeling and real-time control strategies to meet the growing demand for precise control of complex structure wells and long horizontal wells. SUMMARY

[0006] The embodiment of the present application aims to provide a downhole tool face dynamic control method and system based on reinforcement learning, so as to at least solve the problem that the current tool face control technology has obvious deficiencies in coping with dynamic downhole environment and processing state feedback lag.

[0007] In order to achieve the above-mentioned purpose, the first aspect of the present application provides a downhole tool face dynamic control method based on reinforcement learning, the method comprising: collecting drilling state data of a target well, performing data preprocessing on the drilling state data, and constructing a corresponding state vector based on the preprocessed drilling state data; calling an action decision model trained based on a DDPG algorithm based on the state vector to obtain an action instruction; wherein the network strategy parameters of the action decision model are obtained by correcting the last round of network strategy parameters based on the deviation between the current drilling state data and the pre-constructed reward function; performing analysis on the action instruction to determine a target physical action; performing top drive wobble based on the target physical action, and recovering the execution result until the tool face angle meets the expectation.

[0008] Optionally, the data preprocessing on the drilling state data and the construction of the corresponding state vector based on the preprocessed drilling state data comprise: sequentially performing multi-stage domain filtering processing, wavelet denoising processing and interpolation synchronization processing based on a timestamp sequence on the drilling state data to obtain preprocessed data; performing feature extraction on the formation type, the rate of deviation angle change and the drill string structure parameters contained in the preprocessed drilling state data to obtain a feature extraction result; generating a well section working condition feature vector based on the feature extraction result, and matching the well section working condition feature vector with a preset state template to determine a target feature set for constructing the state vector and a normalization processing mode; and clipping and / or expanding the dimension and parameter structure of the state vector generated based on the preprocessed data according to the matched target feature set and the normalization processing mode to obtain the corresponding state vector.

[0009] Optionally, after constructing the corresponding state vector based on the preprocessed drilling state data, the method further comprises: constructing a state attention degree mapping structure based on the well section working condition feature vector, wherein the well section working condition feature vector comprises the rate of deviation angle change, the wellbore curvature grade and the drill string stiffness factor, and the well section working condition feature is used to represent the response characteristics of different well sections; in each control period, the state attention degree mapping structure is used to retrieve the attention degree weight set corresponding to the current well section from a pre-constructed state dimension-attention weight mapping table; the attention degree weight set is dimensionally corresponding to the state vector, and the item-by-item weighting operation is performed to form a weighted state vector.

[0010] Optionally, the action decision model comprises a policy network outputting a control action through forward inference and a value network for estimating a state-action value function; the policy network receives a state vector as input, sequentially performs standardization processing, feature encoding processing and continuous action mapping, and outputs an action instruction for controlling the target rig angle increment and torque compensation of the top drive wobbling; the value network is used to perform correction on the last round of network policy parameters based on the deviation between the current drilling state data and the reward function to obtain the current round of network policy parameters; wherein the rules for performing correction on the last round of network policy parameters based on the deviation between the current drilling state data and the reward function include: calculating the expected return of the current state-action pair based on the current drilling state data and the corresponding reward function output value; comparing the expected return with the historical evaluation value output by the last round of value network to obtain an action value error term; inputting the error term as a loss function, and performing parameter update on the policy network through gradient back propagation to obtain the current round of network policy parameters.

[0011] Optionally, the construction rules of the reward function include: extracting the tool face angle deviation, the current top drive angle and the current torque value in the current drilling state data as target indicators; performing normalization processing on each target indicator to obtain a set of indicators with unified dimensions; based on the type of the well section where the drilling stage is located, matching the corresponding weight parameters from the pre-configured weight configuration set to give each indicator a weighting coefficient; performing linear combination of the weighted indicator set to obtain the corresponding reward function.

[0012] Optionally, the action instruction is parsed to determine the target physical action, including: parsing the action instruction to extract the angle increment and torque compensation for the top drive wobbling; performing linear accumulation on the angle increment and the current top drive actual rotation angle, and performing amplitude limiting processing on the accumulated value to obtain a target top drive angle value; performing dynamic amplitude limiting adjustment on the torque compensation to determine the maximum allowed torque output at the corresponding level according to the downhole vibration level feedback to obtain a target torque; taking the target top drive angle value and the target torque as the target physical action.

[0013] Optionally, during the top drive wobbling based on the target physical action, the method further comprises: continuously monitoring the tool face angle deviation and the downhole vibration level; when the tool face angle deviation exceeds a first preset threshold or the downhole vibration level reaches a second preset level, triggering an emergency back-in operation; wherein the emergency back-in operation comprises: reducing the current target top drive angle value to a first target proportion value of the original value to form a transition angle target; performing angle adjustment based on the transition angle target until the top drive angle returns to a pre-set safe reference angle; simultaneously reducing the target torque to a second target proportion value of the pre-set safe reference torque and maintaining this state until the tool face angle deviation is within the first stable threshold range and the downhole vibration level is lower than the second stable threshold for a period of time exceeding a pre-set recovery period.

[0014] Optionally, after the execution result indicates that the tool face angle meets the expectation, the method further comprises: collecting feedback information of the top drive device, including a current top drive actual angle and a current torque output value; synchronously receiving a latest tool face angle value and a vibration level in the downhole measurement data, to form a feedback data set after action execution based on the current top drive actual angle and the current torque output value, the latest tool face angle value and the vibration level; performing time alignment and interpolation completion operations on the feedback data set to construct a state feedback vector; merging the state feedback vector with the original state vector to replace the historical values in the corresponding dimensions to form a new state vector; taking the new state vector as input data of the action decision model in the next cycle.

[0015] The second aspect of the present application provides a downhole tool face dynamic control system based on reinforcement learning, comprising: a collection unit configured to collect drilling state data of a target well, perform data preprocessing on the drilling state data, and construct a corresponding state vector based on the preprocessed drilling state data; a decision unit configured to call an action decision model trained based on a DDPG algorithm based on the state vector to obtain an action instruction; wherein the network policy parameters of the action decision model are obtained by correcting the current drilling state data and the preconstructed reward function based on the previous network policy parameters; an analysis unit configured to analyze the action instruction to determine a target physical action; and an execution unit configured to execute top drive wobble based on the target physical action and recover the execution result until the execution result indicates that the tool face angle meets the expectation.

[0016] In another aspect, the present application provides a computer-readable storage medium having instructions stored thereon, which, when executed on a computer, cause the computer to perform the above-mentioned downhole tool face dynamic control method based on reinforcement learning.

[0017] Through the above technical solution, the present application scheme can construct a state vector reflecting the current well section working condition by collecting drilling state data of a target well, and generate a control action by introducing an action decision model trained based on a DDPG algorithm, thereby realizing adaptive optimization of the tool face angle control process. Compared with the traditional method based on fixed control parameters, this method can dynamically adjust the strategy parameters under different formation conditions and changes in the drill string structure, and continuously correct the model using the deviation between the real-time state data and the preset reward function, thereby improving the response capability and precision of the tool face control. In addition, by combining action instruction analysis and target physical action execution feedback, a closed-loop control process is constructed, which effectively alleviates the decision failure problem caused by the lag of MWD data, and improves the stability and real-time control effect of the system under complex well sections and severe disturbance working conditions.

[0018] Other features and advantages of the present application will be illustrated in detail in the following detailed description part. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings are included to provide a further understanding of the present application and are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and serve to explain the principles of the present application, but are not intended to limit the present application. In the drawings:

[0020] Figure 1 is a method flow chart of the downhole tool face dynamic control method based on reinforcement learning provided by an embodiment of the present application. As shown in

[0021] Figure 2 is a system structure diagram of the downhole tool face dynamic control system based on reinforcement learning provided by an embodiment of the present application. DETAILED DESCRIPTION

[0022] The specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present application, and are not intended to limit the present application.

[0023] Figure 1 is a method flow chart of the downhole tool face dynamic control method based on reinforcement learning provided by an embodiment of the present application. As shown in Figure 1 The present application provides a downhole tool face dynamic control method based on reinforcement learning, which comprises the following steps:

[0024] Step S10: Collecting drilling state data of a target well, performing data preprocessing on the drilling state data, and constructing a corresponding state vector based on the preprocessed drilling state data.

[0025] Specifically, the drilling state data is sequentially subjected to multi-stage domain filtering processing, wavelet denoising processing and interpolation synchronization processing based on a timestamp sequence to obtain preprocessed data; feature extraction is performed on the formation type, inclination rate and drill string structure parameters contained in the preprocessed drilling state data to obtain a feature extraction result; a well section working condition feature vector is generated based on the feature extraction result, and the well section working condition feature vector is matched with a preset state template to determine a target feature set and a normalization processing mode for constructing the state vector; the dimension and parameter structure of the state vector generated based on the preprocessed data are cropped and / or expanded according to the matched target feature set and normalization processing mode to obtain a corresponding state vector.

[0026] In the embodiments of the present application, the original drilling state data formed in the drilling process of the target well is collected, which includes formation identification information (such as logging curve, formation label), well trajectory change (such as inclination angle, azimuth angle), drill string structure parameters (such as drilling assembly number, drill string stiffness parameters), top drive real-time signals (such as rotation angle, torque), and time stamp of surface and downhole measurement data, etc. For such multi-channel, asynchronous sampling of original data, three types of preprocessing operations are performed in turn:

[0027] 1) Multi-stage domain filtering processing is adopted, different filter kernel functions are introduced in high-frequency disturbance interval, transition interval and low-frequency trend interval, differential sliding filtering is performed on drill string vibration signal and downhole torque feedback, so as to separate formation excitation response and equipment noise.

[0028] 2) Wavelet threshold denoising method is used to perform multi-scale wavelet decomposition and reconstruction processing on the data channel containing nonlinear disturbance characteristics (such as inclination angle change, formation resistivity), effectively eliminating intermittent strong interference components and retaining main response characteristics.

[0029] 3) For the time asynchronization problem between the surface top drive data and the downhole MWD data, an interpolation synchronization strategy based on time stamp sequence is constructed, and cubic spline interpolation or sequence alignment method based on sliding window is adopted to ensure the consistency of multi-source data in time dimension.

[0030] After the above preprocessing is completed, the key feature variables for describing the current well section working condition are extracted from the processed data, including: formation type (represented by lithology number or logging response parameter), inclination angle change rate (calculated by the derivative of inclination angle to time or footage), drill string structure parameters (extracted from the stiffness, length, connection mode, etc. indexes according to the drilling assembly code). Based on these feature variables, a well section working condition feature vector is generated, and the working condition vector is matched with a group of preset state construction templates.

[0031] Further, the state template is a group of state variable configuration schemes pre-constructed for different well section working conditions, covering typical well type categories such as vertical section, curved section and long horizontal section. Each template defines several fields in a structured manner, including: the dimension number of the required feature set, the physical meaning of each feature item, the corresponding unit dimension range, the normalization method (linear normalization, logarithmic compression, Z-score standardization, etc.), and whether the feature is optional.

[0032] For example, for the vertical section template, the main variables related to axial advancement stability are included, such as the rate of change of the WOB, the amount of change in the top drive speed, and the torque stability index, and the number of dimensions is controlled to be within 12 dimensions; while the long horizontal section template focuses on describing the hole friction effect and trajectory control accuracy, and adds variables such as tool face angle variance, hole inclination first derivative, vibration level index, and the total dimension can be expanded to more than 20 dimensions.

[0033] The matching of the well section and the template is completed according to the key indicators in the well section working condition feature vector. Specifically, the matching rules can be set as follows: if the rate of change of the hole inclination is less than 5° / 30m, the change of the well depth is linear, and the length of the drill string is shorter than a set threshold, then the vertical section template is matched; if the hole inclination derivative fluctuates within a certain range and the drill string flexibility parameter is higher, then the curved section template is matched; if the rate of change of the hole inclination is stable at a high level and the length of the sliding drilling exceeds a critical value, then the long horizontal section template is matched.

[0034] After the matching is completed, a structure adjustment operation is performed on the processed state vector, including two dimensions: one is feature clipping: removing features that have little effect on the current well section response, such as removing variables such as the first derivative of the hole inclination and the tool face angle variance that have weak correlation with trajectory control in the vertical section; the second is feature expansion: in the long horizontal section or high friction section, high-order dynamic features derived from existing variables are introduced, such as second-order difference of torque to extract load mutation information, or sliding window fluctuation rate of hole inclination sequence to reveal the trend of small trajectory deviation. In addition, for each retained or newly added feature, the normalization strategy pre-set in the template is processed to ensure that the numerical range is aligned to the model training input range, preventing some variables from having an uneven gradient effect in training or reasoning due to different scales.

[0035] Through the state variable structure regulation mechanism driven by the template, the state representation is more sensitive to the working condition and has better feature recognition ability, which not only improves the expression efficiency of the reinforcement learning model for the response characteristics of the current well section, but also significantly enhances the generalization adaptability and control accuracy of the strategy under multiple well type conditions.

[0036] Preferably, after constructing the corresponding state vector based on the pre-processed drilling state data, the method further comprises: constructing a state attention mapping structure based on a well section working condition feature vector, wherein the well section working condition feature vector includes a rate of change of hole inclination, a hole curvature level, and a drill string stiffness factor, and the well section working condition feature is used to represent the response characteristics of different well sections; in each control period, the state attention mapping structure is used to retrieve the attention weight set corresponding to the current well section from the pre-constructed state dimension-attention weight mapping table; the attention weight set is dimensionally corresponding to the state vector, and a weighted state vector is formed by performing item-by-item weighting operation.

[0037] In the embodiments of the present application, after constructing the corresponding state vector based on the pretreated drilling state data, in order to further improve the working condition adaptability of the state expression, attention weighting operation is performed on the state vector before the control strategy is called, so as to enhance the response sensitivity of key feature dimensions. Specifically, first, a state attention mapping structure is constructed according to a well section working condition feature vector. The well section working condition feature vector is used to describe the dynamic response characteristics of the current well section, and at least includes the following three types of parameters:

[0038] 1) Rate of change of inclination: by performing first-order derivative calculation on the inclination along the time or footage sequence, the degree of trajectory change is reflected;

[0039] 2) Wellbore curvature level: based on the well trajectory model, the spatial bending degree of the wellbore in the unit well depth range is calculated as the basis for identifying the curved well or horizontal well section;

[0040] 3) Drilling string stiffness factor: derived from the structural parameters of the drilling assembly (such as pipe diameter, connection method, thickness distribution), used to reflect the overall deformation ability and mechanical response delay characteristics of the drilling string.

[0041] After constructing the well section working condition feature vector according to the above parameters in a predetermined order, matching is performed in combination with a pre-constructed state attention mapping structure. The mapping structure is essentially a multi-dimensional mapping table or query matrix, which lists the attention weight to be allocated to each state feature dimension under different working conditions. The table is obtained by historical data driven modeling or simulation result calibration. For example, in a long horizontal well section, the weight values of the tool face angle related features and the inclination derivative features are significantly higher than those of the torque related features; while in the vertical section, the top drive load stability may be more sensitive.

[0042] In actual execution, in each control period, the current well section working condition feature vector is matched with the mapping structure, and the attention weight group most similar to the current working condition is retrieved. After the weight group and the constructed original state vector are one-to-one corresponding in dimension, element-level item-by-item weighting operation is performed, that is, each component in the state vector is multiplied by its corresponding attention factor to form a new weighted state vector. This weighting method substantially adjusts the influence intensity of each state variable on the control model without changing the dimension structure of the original state vector, which is equivalent to introducing an "input-level attention mechanism". This processing strategy not only improves the expression pertinence of the state representation under different well sections, but also enhances the action decision robustness of the model under variable geological conditions, which helps the control strategy to respond more accurately to the small changes of key variables, and improves the overall control precision and stability.

[0043] Step S20: calling an action decision model trained based on the DDPG algorithm based on the state vector to obtain an action instruction.

[0044] Specifically, the action decision model comprises a policy network outputting a control action through forward inference and a value network for estimating a state-action value function; the policy network receives a state vector as input, sequentially performs standardization processing, feature encoding processing and continuous action mapping, and outputs an action instruction for controlling a target top drive angle increment and a torque compensation amount of the top drive wobbling; the value network is used to perform correction on the last round of network policy parameters based on the deviation between the current drilling state data and the reward function, to obtain the network policy parameters of the current round; wherein the rules for performing correction on the last round of network policy parameters based on the deviation between the current drilling state data and the reward function comprise: calculating the expected return of the current state-action pair based on the current drilling state data and the corresponding reward function output value; comparing the expected return with the historical evaluation value output by the last round of value network to obtain an action value error term; inputting the error term as a loss function, and performing parameter update on the policy network through gradient back propagation to obtain the network policy parameters of the current round.

[0045] Further, the construction rules of the reward function comprise: extracting a tool face angle deviation, a current top drive angle and a current torque value in the current drilling state data as target indicators; performing normalization processing on each target indicator to obtain a set of indicators with unified dimensions; based on the type of the well section where the drilling stage is located, matching the corresponding weight parameters from the preset weight configuration set, and respectively assigning weight coefficients to each indicator; performing linear combination of the weighted indicator set to obtain the corresponding reward function.

[0046] In the embodiment of the present application, after the construction of the state vector is completed, in order to realize intelligent decision control of the top drive wobbling behavior, the action decision model trained in advance can be called based on the state vector to generate the action instruction in the current control period. The action decision model is modeled and trained by using the Deep Deterministic Policy Gradient (DDPG) algorithm in deep reinforcement learning, has the ability of high-precision policy optimization in continuous action space, and is suitable for the scene requirements of continuous angle and torque control in actual drilling process.

[0047] The action decision model mainly consists of two functional networks, namely a policy network (i.e. an “Actor” network) and a value network (i.e. a “Critic” network). The policy network is used to perform forward inference according to the current state vector, and output an action instruction for controlling the top drive wobbling behavior; the value network is used to evaluate the value performance of the current state-action pair in the long-term control process, and provide an evaluation signal for the update of the policy parameters.

[0048] Specifically, the policy network receives the constructed state vector as input, which has been dimensionally pruned, structurally expanded, and feature-weighted according to the well section working condition and state attention mechanism. To ensure the stability and numerical consistency of subsequent network training and reasoning, the input state vector is first standardized, that is, all feature dimensions are uniformly mapped to the normalized interval of [–1, 1] or [0, 1], commonly using Z-score standardization or Min-Max normalization strategy.

[0049] After completing the standardization process, the feature encoding stage is entered. In this stage, the state vector is encoded into a high-dimensional policy feature representation through the stacking of multiple nonlinear activation functions (such as ReLU, Tanh) and fully connected layer mapping. The encoded result is input into the action mapping layer, which consists of one or more continuous value output nodes corresponding to the angle increment output node and the torque compensation output node in the top drive control target. The output values constitute the action instruction in the current control period. To ensure the controllable range of action output, a hyperbolic tangent (tanh) or Sigmoid activation function is usually added at the end of the action mapping layer to compress the output result and make it fall within the preset physical safety control interval.

[0050] The action instruction contains two elements: one is the target top drive angle increment, which reflects the model's correction trend and amplitude for the current tool face offset; the other is the torque compensation, which is expressed as an incremental correction value on the current base torque signal, used to suppress oscillation and improve following stability.

[0051] After the action generation is completed, periodic policy optimization of the action decision model is required to ensure its continuous adaptability under changing formation conditions and drill string response conditions. This optimization process mainly relies on the evaluation mechanism of the value network and the feedback signal of the reward function.

[0052] In each control period, based on the paired information of the current drilling state data and the executed action result, the current state-action pair is constructed and input into the value network for evaluation. The value network is generally a multi-layer perceptron structure, with the input being the concatenation of the state vector and the action vector, and the output being the cumulative expected reward value of the state-action pair under the expected control target, denoted as Q(s, a). To quantify the accuracy and correction direction of the evaluation value, the actual reward function value obtained in the same state-action pair in the previous control period is subtracted to form the action value error term ΔQ.

[0053] The error term AQ participates in the subsequent policy parameter optimization process as the basic indicator of the loss function. The loss function generally adopts the mean square error (MSE) structure, that is, AQ 2 is taken as the optimization target. Then, the backpropagation algorithm is used to perform gradient calculation on the weight parameters in the policy network, and the optimizer (such as Adam) is used for parameter update. The whole optimization process can combine the target network soft update mechanism and the experience replay mechanism to further improve the stability and generalization ability of model training. Among them, the target network soft update mechanism sets a set of "delayed update copies" for the policy network and the value network respectively, and periodically replicates the main network weight with a small step (such as τ = 0.005), avoiding violent fluctuations in the training process; while the experience replay mechanism establishes a four-tuple cache queue of state-action-reward-next state, randomly samples from it at each update, breaks the time correlation between samples, and improves the convergence speed and policy generalization ability.

[0054] The core of the above policy optimization process depends on the design quality of the reward function. In order to ensure that the reward function has good mapping to the key control targets in the drilling process, the construction process of the reward function needs to be modeled for the control indicators in the actual drilling conditions.

[0055] Specifically, in each control period, three types of variables closely related to control accuracy are extracted from the current drilling state data, namely: tool face angle deviation (i.e. the difference between the actual tool face angle and the expected angle), current top drive angle (to reflect the aggressiveness of the control behavior), and current torque value (to reflect the mechanical load during execution). These three variables represent the accuracy target, control stability target and energy balance target, which are commonly used multi-objective optimization indicators in reinforcement learning control. In order to facilitate quantitative processing and model processing consistency, the above variables need to be normalized, which can be done in the following way:

[0056] 1) Tool face angle deviation: linear compression with ±180° as the interval;

[0057] 2) Top drive angle: Min-Max standardization based on historical angle response range;

[0058] 3) Torque value: percentage scaling with the set maximum allowable torque as the reference.

[0059] The normalized indicator set constitutes the reward function input variable set. Next, according to the drilling stage and structural characteristics of the current well section, a matching weight parameter group is retrieved from the preset weight configuration set, and a weighted coefficient corresponding to the three target indicators is respectively assigned. For example, in the early stage of long horizontal section drilling, in order to ensure the trajectory guidance accuracy, the tool face angle deviation weight can be set to 0.6, the top drive angle weight to 0.3, and the torque value weight to 0.1; while in the late stage of the well section, the control action tends to be stable, the torque weight can be appropriately increased, and the angle jump behavior can be reduced. Finally, the weighted normalized indicators are linearly combined to construct the reward function value R(t) in the current control period.

[0060] The reward function R(t) is used as the evaluation target of the value network in the next control cycle, and is compared and optimized with the behavior output by the policy network. Through the above method, a closed-loop control link is formed. The scheme introduces a DDPG-based deep reinforcement learning policy network structure, so that the control process of the top drive wobble in the drilling process can be decoupled from the static rule base constraint, and instead, the policy adaptive update is realized in a data-driven and feedback-dominated manner. Especially in complex working conditions such as multi-well section switching, high nonlinearity of friction, tool face angle response hysteresis, etc., it has good dynamic adaptability and control accuracy, significantly improves the tracking ability and response sensitivity of the control system to the tool face angle, and enhances the stability and intelligent level of the overall drilling trajectory control.

[0061] In one possible implementation, the state vector is represented as:

[0062]

[0063] Wherein, the tool face deviation ΔTF reflects the target tracking error; the top drive torque T surface characterizes the stress state of the drill string; the swing angle θ swing and its rate of change θ' swing describe the current action characteristics; the vibration level v level evaluates the downhole safety state. The control instruction output is a two-dimensional action vector:

[0064]

[0065] The former controls the angle span (range ±5°) of a single swing, and the latter adjusts the torque compensation amount (range ±3kN·m). The policy network realizes action optimization by minimizing the value function

[0066]

[0067] wherein is the target critic network, ​For the target actor network, the discount factor γ = 0.99 balances the current reward with the long-term effect. The reward function is designed as:

[0068]

[0069] This formula allocates 50% weight to tool face tracking accuracy, 30% to suppress torque fluctuation, and 20% to constrain the action amplitude, guiding the system to strike a balance between precision and safety. The weight parameters are determined through sensitivity analysis and combined with field experience and domain knowledge.

[0070] The algorithm realizes stable learning through an actor-critic dual network architecture. Both the actor network and the critic network are designed as three-layer fully connected neural networks, taking into account both feature extraction efficiency and real-time control requirements. The actor network input layer receives the standardized state vector (including tool face deviation, top drive torque, etc.), and after passing through the first hidden layer (512 nodes) and the second hidden layer (256 nodes), it performs feature abstraction. The ReLU activation function is used to enhance the non-linear expression ability, and the output layer is constrained by the Tanh function (Tanh) to limit the action range and linearly map to the actual physical quantity, i.e., the swing angle and torque. The critic network input layer fuses the state vector and action instruction, and the hidden layer uses the LeakyReLU activation function to alleviate the gradient sparsity problem. The output layer directly generates the action value score. The network weight initialization uses the He initialization method suitable for adaptive nonlinear units, and the input data is preprocessed by online mean variance normalization. Layer normalization modules are added after the hidden layer to improve training stability.

[0071] To improve training efficiency and avoid strong correlation between data, the experience replay mechanism is used, which stores the past state, action, reward, and next state (st, at, rt, st+1) in a buffer. During training, a batch of data is randomly selected to update the network, which can reuse old data and reduce the bias caused by consecutive samples. In addition, to avoid drastic fluctuations in value estimation during training, the algorithm introduces a target network whose parameters are not immediately updated but slowly synchronized with the online network parameters in a soft update manner. The update formula is:

[0072]

[0073] where is the target network parameter, which provides a stable training target for the law, is the online network parameter, which is updated in real time through interaction with the environment, is the soft update coefficient, The value of is determined to be 0.005 through multiple tests, and a smaller The value can ensure that the target network parameter is updated slowly, maintain the stability of training, and ensure that the change of the target network is smooth enough.

[0074] The action exploration aspect adopts decaying Gaussian noise, and the standard deviation of the noise decays exponentially with the training step number:

[0075]

[0076] Random noise is added to the action output by the actor network, so that the noise is larger at the beginning of training (when t is smaller), encouraging the attempt of different actions to fully explore the environment, and the noise gradually decreases as the training step number increases, eventually making the strategy stable. This design enables the algorithm to extensively explore in the early stage and accurately optimize in the later stage, thereby achieving efficient and stable learning in complex tasks (such as continuous action space problems).

[0077] Step S30: parsing the action instruction to determine the target physical action.

[0078] Specifically, the action instruction is parsed to extract an angle increment for the top drive wobble and a torque compensation amount; the angle increment is linearly added to a current top drive actual rotation angle, and the added value is subjected to amplitude limiting processing to obtain a target top drive angle value; the torque compensation amount is subjected to dynamic amplitude limiting adjustment to determine a maximum allowable torque output at a corresponding level according to downhole vibration level feedback to obtain a target torque; and the target top drive angle value and the target torque are taken as the target physical action.

[0079] In the embodiment of the present application, after generating the action instruction based on the reinforcement learning model, in order to ensure that the output action can accurately, stably and in accordance with the actual physical execution conditions drive the top drive wobble behavior, it is necessary to structure the action instruction and convert it into explicit physical control parameters, i.e. target top drive angle value and target torque output value. This process not only involves numerical decoding of the action instruction itself, but also requires amplitude limiting processing and dynamic adjustment in combination with the current device state and downhole working conditions to ensure the safety and rationality of the execution process.

[0080] Specifically, the action instruction generally consists of two components: one is an angle control component, usually represented as a target angle increment value relative to the current top drive rotation angle position; the other is a torque control component, represented as a torque compensation amount on the basis of the load. First, the two components are extracted from the action instruction. For the angle increment value, linear accumulation can be used to superimpose the current top drive real-time actual rotation angle to form a target top drive angle value for the next cycle of control execution. The current top drive actual angle can be provided by a ground angle encoder or a rotation sensor in a time-synchronized manner.

[0081] To prevent large jumps in the angle control command, especially under high friction conditions, which may cause "anti-torque oscillation" or "posture overshoot" of the drill string, the target angle value needs to be subjected to amplitude limiting processing. The amplitude limiting processing is based on the set physical safety range for boundary constraint, for example, the top drive angle control range is limited to ±180°, and when the target angle exceeds this range, a saturation value clipping operation is performed. The torque compensation amount in the action command is subjected to dynamic amplitude limiting processing. Unlike the fixed boundary strategy of angle control, torque output has stronger state dependence, so the maximum allowable torque threshold needs to be dynamically calculated according to the downhole vibration level feedback information. The vibration level can be obtained by processing the axial / radial acceleration signals fed back by the measurement while drilling (MWD) tool, and is divided into multiple levels, such as V0~V3, wherein the higher the level, the stronger the downhole vibration. For each vibration level, a set of maximum allowable torque coefficients is preset, which is significantly reduced at high vibration levels to protect the drill string and top drive device. Finally, the basic torque value is added to the command compensation amount to obtain the target torque value to be executed, and a dynamic amplitude limiting operation is performed.

[0082] Based on the scheme of the application, the target physical action is composed of two parts: the top drive target angle value after amplitude limiting and the target torque output value after amplitude limiting. These two control quantities jointly determine the actual execution state of the top drive device in the current control period. By introducing the state-dependent dynamic amplitude limiting mechanism, the execution safety and response accuracy of the control action are improved, and the control oscillation or overload risk caused by strategy output fluctuation is avoided, further enhancing the adaptability and engineering usability of the reinforcement learning control strategy in complex drilling environments.

[0083] In a possible implementation, in the instruction analysis stage, first, the action vector is subjected to physical quantity conversion. The swing angle increment ΔA swing is linearly superimposed on the current top drive actual angle A current to generate the target angle A target =A current +ΔA swing In this way, the angle adjustment process can be more stable, and sudden large changes can be avoided to prevent equipment shaking or loss of control. Subsequently, the three-ring PID controller is used to convert it into the speed command of the driving motor, and through simulation testing and combined with real drilling data, the proportional coefficient Kp is set to 0.8, which is responsible for quickly responding to angle deviation, the integral coefficient Ki is set to 0.1, which is responsible for eliminating long-term accumulated small errors, and the differential coefficient Kd is set to 0.3, which is responsible for suppressing rapid changes to prevent system jitter. The period of this control process is strictly limited to within 50 milliseconds, which ensures that the system can quickly respond to angle changes, accurately stop at the target position while maintaining adjustment speed, and neither sluggish nor oscillate due to excessive adjustment.

[0084] To ensure the safety of the mechanical system, three layers of physical constraint mechanisms are built in. At the angle control level, the accumulated swing angle is strictly limited within the range of ±180°. When the target angle calculation value exceeds this range, the system automatically performs hard limiting truncation. For example, if A target >180°, then A final =180° is forcibly set, and vice versa, when A target <−180°, A final =−180° is set. At the torque control level, in addition to filtering processing, when the downhole vibration level vlevel≥4 or the tool face deviation ΔTF>45°, an emergency return is triggered immediately: within 0.5 seconds, the swing angle is reduced to 50% of the current value, and the swing is returned to the reference position (Abase=0°) at a constant angular velocity of 2° / s, while the torque output is reset to 70% of the reference value until the state monitoring confirms that the downhole working condition has returned to a safe state.

[0085] During execution, the actual angle A actual and torque sensor data T feedback are continuously received from the top drive feedback, and the data are transmitted to the state detection for subsequent calculation, providing key data support for real-time optimization of reinforcement learning.

[0086] In another possible implementation, during drilling of a part of a long horizontal well section, due to the large length of the drill string and the strong flexibility of the configuration, the top drive swing angle applied on the ground will not be immediately transmitted to the downhole sliding drill bit, and there is a significant angle response lag phenomenon, which is manifested as that the tool face does not immediately respond although the control angle has been adjusted, thereby causing trajectory deviation and target instability problems. To overcome the influence of such “delayed response”, the embodiment introduces an angle prediction correction mechanism based on a drill string axial compression wave propagation model in the action instruction analysis stage, which is used to compensate the target angle setting value in advance.

[0087] Specifically, after obtaining the angle increment value contained in the action instruction, the current drill string length L, the drill string elastic modulus E, the cross-sectional area A, and the unit mass ρ are calculated, and the drill string propagation velocity model is constructed in combination with the ground-downhole torque feedback difference:

[0088]

[0089] According to the current length L of the drill string and the propagation velocity v, the estimated delay time t of the ground control signal transmitted to the drill bit end can be calculated. Further based on the angle change trend of the top drive in the past time window, a first-order extrapolation model is used to predict the potential lag amount of the downhole angle response , and the current angle increment instruction is compensated:

[0090]

[0091] The compensated angle is taken as a target top drive angle value to participate in the amplitude limiting judgment and subsequent physical action construction. The mechanism can effectively improve the angle control feedforward property under the long drill string structure, reduce the tool face angle delay and overshoot, and improve the consistency between the control instruction and the downhole response. Through the embodiment, not only the drill string axial dynamics characteristics are integrated, but also the control model application effect under the "long delay chain structure" condition is enhanced by introducing the dynamic prediction compensation logic, which is suitable for complex well section scenarios with a well depth of more than 2500 meters and a horizontal section length of more than 1000 meters. In the existing control strategy, it is still a cold design, and has strong technical recognition and creative value.

[0092] Step S40: based on the target physical action, the top drive wobble is executed, and the execution result is recovered until the execution result indicates that the tool face angle meets the expectation.

[0093] Specifically, during the execution of the top drive wobble based on the target physical action, the method further comprises: continuously monitoring the tool face angle deviation and the downhole vibration level; when the tool face angle deviation exceeds a first preset threshold or the downhole vibration level reaches a second preset level, triggering an emergency return operation; wherein the emergency return operation comprises: reducing the current target top drive angle value to a first target proportion value of the original value to form a transition angle target; executing angle adjustment based on the transition angle target until the top drive angle returns to a preset safe reference angle; synchronously reducing the target torque to a second target proportion value of the preset safe reference torque, and maintaining the state until the tool face angle deviation is within the first stable threshold range and the downhole vibration level is lower than the second stable threshold for more than a preset recovery period.

[0094] In the implementation of the present application, in the process of executing the target physical action, the set angle adjustment and torque output are completed by driving the top drive device to realize the control of the downhole tool face angle. The process continues until the feedback result returned from the while-drilling measurement tool indicates that the current downhole tool face angle has basically reached the preset target, and the control action is considered as an effective closure.

[0095] In actual drilling conditions, due to the complexity of well section types and significant differences in formation physical properties, the tool face control process may encounter problems such as angle drift and vibration superposition, resulting in unstable state of the top drive wobble behavior. In order to ensure the control accuracy and the safety of the drill string structure, a set of safety monitoring and return protection logic needs to be implemented simultaneously during the execution of the control action.

[0096] Specifically, during the control instruction execution, two key state quantities are continuously monitored in real time. One is the tool face angle deviation, which is the difference between the actual tool face angle downhole and the control target. The other is the downhole vibration level, which is divided into several levels by analyzing the vibration sensing data returned by the MWD, and is used to assess the dynamic load strength borne by the current drill string.

[0097] When the tool face angle deviation exceeds the set safety threshold, or the vibration level reaches a high-risk level (such as frequent severe axial vibration or lateral jumping), an emergency back-off operation is triggered immediately. The so-called back-off operation is to actively interrupt the execution process of the current control angle and torque, and instead gradually guide the top drive state back to a safe and stable initial position. During this process, the target angle value originally planned to be executed is first reduced, which is usually reduced by a certain percentage, for example, half of the original value, and used as a temporary transition angle target. Then, instead of returning to the origin immediately, the top drive angle is gradually brought back to the default "reference angle" through a smooth and controlled angle adjustment process. This reference value is usually zero degrees or other factory-set neutral angle.

[0098] At the same time, the current torque output also needs to be reduced synchronously. The torque control is not simply disconnected, but reduced to a certain percentage of the safety reference torque, for example, 70%. This is done to reduce the elastic rebound of the drill string or the impact of the top drive unloading caused by sudden load reduction.

[0099] After the emergency back-off operation is completed, the tool face angle deviation and vibration level are continuously monitored. Only when both indicators are within the safe range and maintain a complete recovery period of time (such as 5 seconds continuously stable), it is considered that the downhole state has recovered, and the back-off protection can be removed, and the normal control action execution phase is re-entered. Through this dynamic and safe back-off strategy, the equipment overload risk can be effectively reduced under complex well sections, severe disturbances or abnormal response conditions, and the robustness and operation recoverability of the top drive control are improved. It is an important guarantee measure to ensure the long-term stable operation of the entire control scheme in actual engineering scenarios.

[0100] Preferably, after the execution result indicates that the tool face angle meets the expectation, the method further comprises: collecting feedback information of the top drive device including a current top drive actual angle and a current torque output value; synchronously receiving a latest tool face angle value and a vibration level in the downhole measurement data to form a feedback data set after action execution based on the current top drive actual angle and the current torque output value, the latest tool face angle value and the vibration level; performing time alignment and interpolation completion operations on the feedback data set to construct a state feedback vector; merging the state feedback vector with the original state vector to replace historical values in corresponding dimensions to form a new state vector; and taking the new state vector as input data of the action decision model in the next cycle.

[0101] In the embodiment of the application, after the action execution of the current control cycle is completed and it is judged that the tool face angle has been basically adjusted to the expected target, in order to ensure that the subsequent control strategy continues to have real-time and adaptability, it is necessary to effectively recycle and update various state data after the current action execution, so as to complete the iterative construction of the state vector. This stage is not simply data collection, but a closed-loop processing process including data fusion, time alignment, dimension replacement and the like, and the ultimate goal is to provide a complete structure and accurate state input basis for the next control cycle.

[0102] Specifically, first, the execution feedback information of the top drive device at the end of the current control cycle is collected, and the collected content at least includes the actual angle value of the current top drive and the current torque output value. The former can be provided by the top drive encoder in real time, and the latter is obtained by the torque sensor, representing the control execution state of the ground end.

[0103] At the same time, the latest data uploaded by the measurement while drilling tool (MWD) in the wellbore need to be synchronously received, mainly including the tool face angle measurement value of the current well section and the vibration level of the drill string. Among them, the tool face angle reflects the actual response of the lower drilling tool group, and the vibration level is used to judge the stress and disturbance of the drill string in the formation, which is often calculated by a multi-axis acceleration sensor and divided into different levels to represent the risk degree.

[0104] The above four data, the angle, the torque value, the tool face angle and the vibration level, together constitute the feedback data set after the action of the current control cycle is completed. Since these data come from the ground equipment and the downhole equipment, their collection frequency, upload delay and data time sequence are often inconsistent, so it is necessary to first perform a time alignment operation on the feedback data set. The rule of time alignment is usually based on time stamp window sliding matching: selecting the ground equipment collection time as the reference, finding the nearest point of the time stamp in the downhole data, and pairing; if the data of a certain time is missing, the numerical completion is performed by an interpolation algorithm, and the commonly used methods include linear interpolation, spline interpolation or sliding window average.

[0105] After time alignment and interpolation completion, a complete structure state feedback vector is constructed. The state feedback vector is combined with the previous round of original state vector. The so-called combination is not full coverage, but selectively updating part of the dimensions in the state vector - usually those that change significantly with the execution feedback, such as angle, torque, vibration level, etc. Other long-term static features such as drill string structure code, well section category, etc. are kept unchanged.

[0106] After the combination is completed, the current latest state vector is formed as the basis for inputting the reinforcement learning action decision model in the next control period. Since the state vector integrates surface execution information and downhole response information, it has good timeliness and working condition reflecting ability, and can provide more accurate environment representation for model policy output.

[0107] Through the feedback construction and state updating mechanism, not only the problem of different synchronization between surface and downhole data and large time difference is effectively solved, but also the closed-loop response ability of the model in the control process is enhanced, ensuring that the state information based on the policy updating process is real, accurate and traceable, thereby improving the stability and practicality of the entire reinforcement learning control process. Especially in sliding drilling, long horizontal well section or well inclination section, this mechanism can significantly improve the sensitivity and adaptability of policy response, and avoid the accumulation of control deviation caused by state perception lag.

[0108] Figure 2 is a system structure diagram of a downhole tool face dynamic control system based on reinforcement learning provided by an embodiment of the present application. As shown in Figure 2 The embodiment of the present application provides a downhole tool face dynamic control system based on reinforcement learning, which comprises: a collection unit for collecting drilling state data of a target well, performing data preprocessing on the drilling state data, and constructing a corresponding state vector based on the preprocessed drilling state data; a decision unit for calling an action decision model trained based on a DDPG algorithm based on the state vector, and obtaining an action instruction; wherein the network policy parameters of the action decision model are obtained by correcting the network policy parameters of the previous round based on the deviation between the current drilling state data and a preconstructed reward function; an analysis unit for performing analysis on the action instruction to determine a target physical action; and an execution unit for executing top drive wobble based on the target physical action, and recovering the execution result until the tool face angle meets the expectation.

[0109] The embodiment of the present application further provides a computer readable storage medium, which stores instructions, and when the instructions are run on a computer, the computer executes the above-mentioned downhole tool face dynamic control method based on reinforcement learning.

[0110] Those skilled in the art can understand that all or part of the steps of the method for implementing the above-mentioned embodiments can be completed by programs instructing relevant hardware, the programs are stored in a storage medium, and the programs include a plurality of instructions for enabling a single-chip microcomputer, a chip or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage media capable of storing program codes.

[0111] The optional embodiments of the present application are described in detail above in combination with the drawings, but the embodiments of the present application are not limited to the specific details in the above-described embodiments. Within the technical concept range of the embodiments of the present application, various simple modifications can be made to the technical solutions of the embodiments of the present application, and these simple modifications all belong to the protection scope of the embodiments of the present application. In addition, it should be noted that each specific technical feature described in the above-described specific embodiments can be combined in any appropriate manner without contradiction. In order to avoid unnecessary repetition, the embodiments of the present application will not be described again for various possible combinations.

[0112] In addition, various different embodiments of the present application can also be combined in any manner, as long as it does not deviate from the idea of the embodiments of the present application, and it should also be considered as disclosed by the embodiments of the present application.

Claims

1. A dynamic control method for downhole tool faces based on reinforcement learning, characterized in that, The method includes: Drilling status data of the target well is collected, and data preprocessing is performed on the drilling status data. A corresponding state vector is then constructed based on the preprocessed drilling status data. Data preprocessing is performed on the drilling status data, and a corresponding state vector is constructed based on the preprocessed drilling status data. This includes: sequentially performing multi-stage domain filtering, wavelet denoising, and timestamp-based interpolation synchronization on the drilling status data to obtain preprocessed data; performing feature extraction on the formation type, well inclination angle change rate, and drill string structure parameters contained in the preprocessed drilling status data to obtain feature extraction results; generating a well section operating condition feature vector based on the feature extraction results, and matching the well section operating condition feature vector with a preset state template to determine the target feature set and normalization processing method for constructing the state vector; and pruning and / or expanding the dimension and parameter structure of the state vector generated based on the preprocessed data according to the matched target feature set and normalization processing method to obtain the corresponding state vector. Based on the state vector, the action decision model trained using the DDPG algorithm is invoked to obtain action instructions; wherein... The network strategy parameters of the action decision model are obtained by correcting the network strategy parameters of the previous round based on the deviation between the current drilling status data and the pre-constructed reward function; The action command is parsed to determine the target physical action; The top drive is torn based on the target physical action, and the execution result is retrieved until the execution result indicates that the tool face angle meets the expectation.

2. The method according to claim 1, characterized in that, After constructing the corresponding state vector based on the preprocessed drilling state data, the method further includes: A state attention mapping structure is constructed based on the well section operating condition feature vector, wherein the well section operating condition feature vector includes the well inclination angle change rate, wellbore curvature level and drill string stiffness factor, and the well section operating condition features are used to characterize the response characteristics of different well sections. Within each control cycle, the attention weight reorganization corresponding to the current well section is retrieved from the pre-built state dimension-attention weight mapping table using the state attention mapping structure; The attention weights are recombined and mapped to the dimensions of the state vector, and a weighted operation is performed on each item to form a weighted state vector.

3. The method according to claim 1, characterized in that, The action decision model includes a policy network that outputs control actions through forward inference and a value network for estimating the state-action value function; The strategy network receives the state vector as input, performs standardization processing, feature encoding processing and continuous action mapping in sequence, and outputs action commands for controlling the target top drive angle increment and torque compensation amount of the top drive yaw. The value network is used to adjust the network strategy parameters of the previous round based on the deviation between the current drilling status data and the reward function, in order to obtain the network strategy parameters for the current round; wherein... The rules for adjusting the network strategy parameters based on the deviation between the current drilling status data and the reward function include: Calculate the expected reward of the current state-action pair based on the current drilling status data and the corresponding reward function output value; By comparing the expected return with the historical evaluation value of the previous value network output, the action value error term is obtained; The error term is used as input to the loss function, and the parameters of the policy network are updated through gradient backpropagation to obtain the network policy parameters for this round.

4. The method according to claim 1, characterized in that, The rules for constructing the reward function include: Extract the tool face angle deviation, current top drive angle, and current torque value, which characterize control performance, from the current drilling status data, and use them as target indicators; Normalization is performed on each target indicator to obtain a set of indicators with uniform dimensions; Based on the well section type in the drilling stage, corresponding weight parameters are matched from the preset weight configuration set, and weighting coefficients are assigned to each indicator respectively. The corresponding reward function is obtained by performing a linear combination of the weighted set of indicators.

5. The method according to claim 1, characterized in that, The action command is parsed to determine the target physical action, including: The action command is analyzed to extract the angle increment and torque compensation amount used for the top drive yaw. The angle increment is linearly accumulated with the current actual top drive angle, and the accumulated value is subjected to amplitude limiting processing to obtain the target top drive angle value; The torque compensation amount is dynamically limited and adjusted, and the maximum allowable torque output at the corresponding level is determined based on the downhole vibration level feedback to obtain the target torque; The target top drive angle and target torque are taken as the target physical motion.

6. The method according to claim 1, characterized in that, During the execution of the top drive torsional oscillation based on the target physical motion, the method further includes: Continuously monitor tool face angle deviation and downhole vibration level; When the tool face angle deviation exceeds the first preset threshold, or the downhole vibration level reaches the second preset level, an emergency centering operation is triggered; wherein, The emergency return-to-center operation includes: The current target top drive angle value is reduced to the first target ratio value of the original value to form a transition angle target; The angle is adjusted based on the transition angle target until the top drive angle returns to the preset safety reference angle. The target torque is simultaneously reduced to a second target ratio value of the preset safe reference torque, and this state is maintained until the tool face angle deviation is within the first stable threshold range and the downhole vibration level is below the second stable threshold for more than the preset recovery period.

7. The method according to claim 1, characterized in that, After the execution result indicates that the facet angle of the tool meets expectations, the method further includes: The top drive equipment collects feedback information including the current actual top drive angle and the current torque output value. The system synchronously receives the latest tool face angle value and vibration level from downhole measurement data to form a set of feedback data after the action is executed, based on the current actual top drive angle and current torque output value, the latest tool face angle value and vibration level. Time alignment and interpolation completion operations are performed on the feedback data set to construct a state feedback vector; The state feedback vector is merged with the original state vector, and the historical values ​​of the corresponding dimensions are replaced to form a new state vector; The new state vector is used as the input data for the action decision model in the next cycle.

8. A dynamic control system for downhole tool faces based on reinforcement learning, characterized in that, The system is applied to the reinforcement learning-based downhole toolface dynamic control method according to any one of claims 1-7, and the system comprises: The acquisition unit is used to acquire drilling status data of the target well, perform data preprocessing on the drilling status data, and construct a corresponding status vector based on the preprocessed drilling status data. The decision unit is used to invoke the action decision model trained based on the DDPG algorithm based on the state vector to obtain action instructions; wherein, The network strategy parameters of the action decision model are obtained by correcting the network strategy parameters of the previous round based on the deviation between the current drilling status data and the pre-constructed reward function; The parsing unit is used to parse the action command and determine the target physical action; The execution unit is used to perform top drive torsion based on the target physical action and collect the execution results until the execution result indicates that the tool face angle meets expectations.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the reinforcement learning-based dynamic control method for downhole tool faces as described in any one of claims 1-7.

Images