Underground tool face dynamic control method and system based on reinforcement learning

Abstract

The embodiment of the invention provides an underground tool face dynamic control method and system based on reinforcement learning, and belongs to the technical field of directional drilling. The method comprises the steps of collecting drilling state data of a target well, performing data preprocessing on the drilling state data, and constructing a corresponding state vector based on the preprocessed drilling state data; calling an action decision model obtained by training based on a DDPG algorithm based on the state vector to obtain an action instruction; wherein the network strategy parameter of the action decision model is obtained by correcting the previous network strategy parameter based on the deviation between the current drilling state data and a pre-constructed reward function; analyzing the action instruction to determine a target physical action; and executing top drive torsional pendulum based on the target physical action, and recycling an execution result until the execution result indicates that the tool face angle meets the expectation. According to the scheme, the stability and the real-time control effect under the working conditions of complex well sections and severe disturbance are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of directional drilling, and in particular to a downhole tool face dynamic control method based on reinforcement learning and a downhole tool face dynamic control system based on reinforcement learning. Background Art

[0002] In directional drilling, controlling the tool face angle is a key component in achieving precise guidance during sliding drilling. Existing tool face control technologies primarily rely on top drive yaw systems, which adjust the downhole tool face orientation by adjusting the drill string's angular displacement or torque. However, current mainstream technologies face two technical bottlenecks, severely limiting their control accuracy and responsiveness in complex well sections.

[0003] Existing control systems generally use methods based on static models or fixed control parameters to set top drive torsion and swing instructions. Most systems rely on the preset theoretical length of the well section buffer or the drill string anti-torque response curve to estimate the control parameters, ignoring the dynamic interaction between the drill string and the formation. During the actual drilling process, due to the uncertainty of the wellbore curvature, drill string structure and formation characteristics, the friction and anti-torque experienced by the drill string often show highly nonlinear fluctuations, causing the preset model to deviate significantly from the actual response. This type of static modeling method is particularly unstable in well sections with drastic formation changes or long sliding sections, and is prone to problems such as tool face angle drift, control lag or over-correction, which in turn affects the trajectory control quality and drilling efficiency.

[0004] Most current systems rely on discretely sampled data as their feedback basis, often using MWD (Measurement While Drilling) equipment to collect downhole angle and attitude information. However, MWD systems are inherently limited by low sampling rates and significant data uplink latency. Sampling cycles typically exceed 30 seconds, and transmission delays can range from tens of seconds to several minutes. Under these conditions, the status information received by the ground control system is severely delayed, failing to reflect real-time downhole operating conditions. This results in control commands being based on outdated data, impacting the effectiveness and real-time nature of control. This can even lead to severe toolface oscillation or drillstring overload due to a breakdown in the control-feedback loop.

[0005] Therefore, current tool face control technology has obvious deficiencies in coping with dynamic downhole environments and handling state feedback lag. There is an urgent need for technological breakthroughs in tool face response modeling and real-time control strategies to adapt to the growing demand for fine control in complex structure wells and long horizontal wells. Summary of the Invention

[0006] The purpose of the embodiments of the present invention is to provide a downhole tool face dynamic control method and system based on reinforcement learning, so as to at least solve the problem that the current tool face control technology has obvious deficiencies in coping with dynamic downhole environments and processing state feedback lag.

[0007] In order to achieve the above-mentioned objectives, the first aspect of the present invention provides a downhole tool face dynamic control method based on reinforcement learning, the method comprising: collecting drilling status data of a target well, performing data preprocessing on the drilling status data, and constructing a corresponding state vector based on the preprocessed drilling status data; calling an action decision model trained based on a DDPG algorithm based on the state vector to obtain an action instruction; wherein, the network strategy parameters of the action decision model are obtained after correction of the deviation between the previous round of network strategy parameters based on the current drilling status data and a pre-constructed reward function; parsing the action instruction to determine the target physical action; executing a top drive torsion swing based on the target physical action, and recovering the execution result until the execution result indicates that the tool face angle meets the expectation.

[0008] Optionally, data preprocessing is performed on the drilling status data, and a corresponding state vector is constructed based on the preprocessed drilling status data, including: performing multi-stage domain filtering processing, wavelet denoising processing and interpolation synchronization processing based on a timestamp sequence on the drilling status data in sequence to obtain preprocessed data; performing feature extraction on the formation type, well inclination change rate and drill string structure parameters contained in the preprocessed drilling status data to obtain feature extraction results; generating a well section working condition feature vector based on the feature extraction results, and matching the well section working condition feature vector with a preset state template to determine a target feature set and normalization processing method for constructing the state vector; and cropping and / or expanding the dimension and parameter structure of the state vector generated based on the preprocessed data according to the matched target feature set and normalization processing method to obtain a corresponding state vector.

[0009] Optionally, after constructing the corresponding state vector based on the preprocessed drilling status data, the method further includes: constructing a state attention mapping structure based on the well section operating condition feature vector, wherein the well section operating condition feature vector includes the well inclination angle change rate, the wellbore curvature grade and the drill string stiffness factor, and the well section operating condition characteristics are used to characterize the response characteristics of different well sections; in each control cycle, the state attention mapping structure is used to retrieve the attention weight group corresponding to the current well section from the pre-constructed state dimension-attention weight mapping table; the attention weight group is dimensionally matched with the state vector, and an item-by-item weighted operation is performed to form a weighted state vector.

[0010] Optionally, the action decision model includes a policy network that outputs control actions through forward reasoning and a value network for estimating state-action value functions; the policy network receives a state vector as input, performs normalization processing, feature encoding processing and continuous action mapping in sequence, and outputs action instructions for the target top drive angle increment and torque compensation amount for controlling top drive torsion; the value network is used to correct the policy parameters of the previous round of network based on the deviation between the current drilling status data and the reward function to obtain the network policy parameters of this round; wherein, the rules for correcting the policy parameters of the previous round based on the deviation between the current drilling status data and the reward function include: calculating the expected return of the current state-action pair based on the current drilling status data and the corresponding reward function output value; comparing the expected return with the historical evaluation value output by the previous round of value network to obtain the action value error term; using the error term as the loss function input, performing parameter update on the policy network through gradient back propagation to obtain the network policy parameters of this round.

[0011] Optionally, the construction rules of the reward function include: extracting the tool face angle deviation, current top drive angle and current torque value that characterize the control performance from the current drilling status data as target indicators; performing normalization processing on each target indicator to obtain a set of indicators with a unified dimension; based on the type of well section in the drilling stage, matching corresponding weight parameters from a preset weight configuration set, and assigning weighted coefficients to each indicator; performing linear combination of the weighted indicator set to obtain the corresponding reward function.

[0012] Optionally, the action instruction is parsed to determine the target physical action, including: parsing the action instruction to extract the angle increment and torque compensation for top drive torsion; linearly adding the angle increment and the current actual rotation angle of the top drive, and performing limiting processing on the accumulated value to obtain the target top drive angle value; performing dynamic limiting adjustment on the torque compensation amount, determining the maximum allowable torque output at the corresponding level based on the downhole vibration level feedback, and obtaining the target torque; and using the target top drive angle value and target torque as the target physical action.

[0013] Optionally, during the execution of top drive torsion based on the target physical action, the method further includes: continuously monitoring the tool face angle deviation and the downhole vibration level; when the tool face angle deviation exceeds a first preset threshold, or the downhole vibration level reaches a second preset level, triggering an emergency return operation; wherein, the emergency return operation includes: reducing the current target top drive angle value to a first target proportional value of the original value to form a transition angle target; performing angle adjustment based on the transition angle target until the top drive angle returns to a preset safety reference angle; synchronously reducing the target torque to a second target proportional value of the preset safety reference torque, and maintaining this state until the tool face angle deviation is within the first stable threshold range and the downhole vibration level is lower than the second stable threshold for more than a preset recovery period.

[0014] Optionally, after the execution result indicates that the tool face angle meets expectations, the method further includes: collecting feedback information of the top drive equipment including the current top drive actual angle and the current torque output value; synchronously receiving the latest tool face angle value and vibration level in the downhole measurement data to form a feedback data set after the action is executed based on the current top drive actual angle and the current torque output value, the latest tool face angle value and the vibration level; performing time alignment and interpolation completion operations on the feedback data set to construct a state feedback vector; merging the state feedback vector with the original state vector, replacing the historical values of the corresponding dimensions therein to form a new state vector; and using the new state vector as input data for the action decision model in the next cycle.

[0015] A second aspect of the present invention provides a downhole tool face dynamic control system based on reinforcement learning, the system comprising: an acquisition unit for acquiring drilling status data of a target well, performing data preprocessing on the drilling status data, and constructing a corresponding state vector based on the preprocessed drilling status data; a decision unit for calling an action decision model trained based on a DDPG algorithm based on the state vector to obtain an action instruction; wherein the network strategy parameters of the action decision model are obtained after correction of the deviation between the previous round of network strategy parameters based on the current drilling status data and a pre-constructed reward function; a parsing unit for parsing the action instruction to determine a target physical action; and an execution unit for executing a top drive torsion swing based on the target physical action, and recovering the execution result until the execution result indicates that the tool face angle meets expectations.

[0016] On the other hand, the present invention provides a computer-readable storage medium having instructions stored thereon, which, when executed on a computer, enables the computer to execute the above-mentioned reinforcement learning-based downhole tool face dynamic control method.

[0017] Through the above technical solution, the solution of the present invention collects the drilling status data of the target well, constructs a state vector reflecting the current well section working condition, and introduces an action decision model trained based on the DDPG algorithm to generate control actions, which can achieve adaptive optimization of the tool face angle control process. Compared with the traditional method based on fixed control parameters, this method can dynamically adjust the strategy parameters under different formation conditions and drill string structure changes, and continuously use the deviation between the real-time status data and the preset reward function to correct the model, thereby improving the responsiveness and accuracy of the tool face control. In addition, by combining the action instruction analysis and the target physical action execution feedback, a closed-loop control process is constructed, which effectively alleviates the decision failure problem caused by the lag of MWD data and improves the stability and real-time control effect of the system in complex well sections and severe disturbance conditions.

[0018] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following detailed description, they are used to explain the embodiments of the present invention, but do not constitute a limitation of the embodiments of the present invention. In the accompanying drawings: Figure 1 A downhole tool face dynamic control method based on reinforcement learning is provided in one embodiment of the present invention; Figure 2 It is a system structure diagram of a downhole tool face dynamic control system based on reinforcement learning provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0021] Figure 1 This is a flow chart of a downhole tool face dynamic control method based on reinforcement learning provided by an embodiment of the present invention. Figure 1 As shown, an embodiment of the present invention provides a downhole tool face dynamic control method based on reinforcement learning, the method comprising: Step S10: collecting drilling status data of the target well, performing data preprocessing on the drilling status data, and constructing a corresponding state vector based on the preprocessed drilling status data.

[0022] Specifically, the drilling status data is sequentially subjected to multi-stage domain filtering processing, wavelet denoising processing, and interpolation synchronization processing based on a timestamp sequence to obtain preprocessed data; feature extraction is performed on the formation type, well inclination change rate, and drill string structure parameters contained in the preprocessed drilling status data to obtain feature extraction results; a well section operating condition feature vector is generated based on the feature extraction results, and the well section operating condition feature vector is matched with a preset state template to determine a target feature set and a normalization processing method for constructing a state vector; the dimension and parameter structure of the state vector generated based on the preprocessed data are cropped and / or expanded according to the matched target feature set and normalization processing method to obtain a corresponding state vector.

[0023] In this embodiment of the present invention, raw drilling status data generated during the drilling process of the target well is collected. This data includes formation identification information (such as well logging curves and formation labels), well trajectory changes (such as well inclination and azimuth), drill string structural parameters (such as drill tool assembly number and drill string stiffness parameters), top drive real-time signals (such as rotation angle and torque), and time stamps of surface and downhole measurement data. For this type of multi-channel, asynchronously sampled raw data, three types of preprocessing operations are performed in sequence: 1) A multi-stage domain filtering process is adopted. Different filter kernel functions are introduced in the high-frequency disturbance interval, transition interval, and low-frequency trend interval respectively. The drill string vibration signal and downhole torque feedback are subjected to differential sliding filtering, thereby separating the formation excitation response and equipment noise.

[0024] 2) The wavelet threshold denoising method is used to perform multi-scale wavelet decomposition and reconstruction on data channels containing nonlinear disturbance characteristics (such as well inclination angle changes and formation resistivity), effectively eliminating intermittent strong interference components and retaining the main response characteristics.

[0025] 3) To address the time asynchrony problem between surface top drive data and downhole MWD data, an interpolation synchronization strategy based on timestamp sequences is constructed, using cubic spline interpolation or sliding window-based sequence alignment methods to ensure the consistency of multi-source data in the time dimension.

[0026] After completing the aforementioned preprocessing, key characteristic variables describing the current well section operating conditions are extracted from the processed data. These include: formation type (characterized by lithology code or logging response parameters), inclination change rate (calculated using the derivative of inclination with respect to time or footage), and drill string structural parameters (according to the drill tool assembly code, such as stiffness, length, and connection method). Based on these characteristic variables, a well section operating condition feature vector is generated and matched against a set of pre-set state construction templates.

[0027] Furthermore, the state templates are pre-built state variable configurations for different well sections, covering typical well types such as vertical sections, curved sections, and long horizontal sections. Each template defines several fields in a structured manner, including the number of dimensions in the required feature set, the physical meaning of each feature item, the corresponding unit dimension range, the normalization method (linear normalization, logarithmic compression, Z-score normalization, etc.), and whether the feature is optional.

[0028] For example, the vertical section template mainly includes variables related to axial propulsion stability, such as drilling pressure fluctuation rate, top drive speed change and torque stability index, and the number of dimensions is controlled within 12 dimensions; while the long horizontal section template focuses on describing the wellbore friction effect and trajectory control accuracy, and adds variables such as tool face angle variance, first-order derivative of well inclination angle, vibration level index, etc., and the total dimension can be expanded to more than 20 dimensions.

[0029] Well sections are matched to templates based on key indicators in the well condition feature vector. Specifically, the matching rules can be set as follows: if the well inclination rate of change is less than 5° / 30m, the well depth changes linearly, and the drill string length is shorter than a set threshold, the vertical section template is matched; if the well inclination derivative fluctuates within a specific range and the drill string flexibility parameter is high, the curved section template is matched; if the well inclination rate of change is stable at a high level and the sliding drilling length exceeds a critical value, the long horizontal section template is matched.

[0030] After matching is complete, a structural adjustment operation is performed on the processed state vector, which includes two dimensions: first, feature clipping: removing features that have minimal impact on the response of the current well section. For example, in vertical sections, variables with weak correlation with trajectory control, such as the first-order derivative of the wellbore angle and the variance of the tool face angle, are eliminated. Second, feature expansion: In long horizontal sections or high-friction sections, higher-order dynamic features derived from existing variables are introduced. For example, second-order differences are performed on torque to extract load mutation information, or sliding window volatility is calculated on the wellbore angle series to reveal subtle trajectory deviation trends. In addition, each retained or newly added feature is processed according to the normalization strategy preset in the template to ensure that its numerical range is aligned with the model training input range, preventing a variable with different scales from generating uneven gradient effects during training or inference.

[0031] Through the template-driven state variable structure control mechanism, the state representation is made more sensitive to working conditions and has more feature recognition capabilities. This not only improves the efficiency of the reinforcement learning model in expressing the response characteristics of the current well section, but also significantly enhances the generalization adaptability and control accuracy of the strategy under multiple well types.

[0032] Preferably, after constructing the corresponding state vector based on the preprocessed drilling status data, the method further includes: constructing a state attention mapping structure based on the well section working condition feature vector, wherein the well section working condition feature vector includes the well inclination angle change rate, the wellbore curvature grade and the drill string stiffness factor, and the well section working condition characteristics are used to characterize the response characteristics of different well sections; in each control cycle, the state attention mapping structure is used to retrieve the attention weight group corresponding to the current well section from the pre-constructed state dimension-attention weight mapping table; the attention weight group is dimensionally matched with the state vector, and an item-by-item weighted operation is performed to form a weighted state vector.

[0033] In an embodiment of the present invention, after constructing a corresponding state vector based on preprocessed drilling status data, to further improve the adaptability of the state representation to working conditions, an attention weighting operation is performed on the state vector before the control strategy is called to enhance its response sensitivity to key feature dimensions. Specifically, a state attention mapping structure is first constructed based on the well section working condition feature vector. The well section working condition feature vector is used to characterize the dynamic response characteristics of the current well section and includes at least the following three types of parameters: 1) Well inclination change rate: This method reflects the severity of trajectory changes by calculating the first-order derivative of the well inclination along the time or footage sequence. 2) Wellbore curvature grade: Calculate the spatial curvature of the wellbore within a unit depth range based on the well trajectory model, which serves as the basis for identifying curved wells or horizontal well sections; 3) Drill string stiffness factor: derived from the structural parameters of the drill tool assembly (such as pipe diameter, connection method, and thickness distribution), it is used to reflect the overall deformation capacity and mechanical response delay characteristics of the drill string.

[0034] After constructing the well section operating condition feature vector using the above parameters in a preset order, it is matched against a pre-built state attention mapping structure. This mapping structure is essentially a multidimensional mapping table or query matrix, which lists the attention weights to be assigned to each state feature dimension under different operating conditions. This table is calibrated through historical data-driven modeling or simulation results. For example, in long horizontal well sections, tool face angle-related features and well inclination angle derivative features have significantly higher weights than torque-related features. In vertical sections, however, top drive load stability may be more sensitive.

[0035] During the actual execution process, within each control cycle, the current well section operating condition feature vector is matched with the mapping structure, and the attention weight group closest to the current operating condition is retrieved from it. After the weight group is matched one-to-one with the constructed original state vector according to the dimensions, an element-by-element weighting operation is performed, that is, each component in the state vector is multiplied by its corresponding attention factor to form a new weighted state vector. This weighting method substantially adjusts the influence of each state variable on the control model without changing the dimensional structure of the original state vector, which is equivalent to introducing an "input-level attention mechanism." This processing strategy not only improves the targeted expression of state representation in different well sections, but also enhances the robustness of the model's action decision-making under variable geological conditions, helping the control strategy to more accurately respond to small changes in key variables and improve overall control accuracy and stability.

[0036] Step S20: Based on the state vector, the action decision model trained based on the DDPG algorithm is called to obtain an action instruction.

[0037] Specifically, the action decision model includes a policy network that outputs control actions through forward reasoning and a value network for estimating state-action value functions; the policy network receives a state vector as input, performs normalization processing, feature encoding processing and continuous action mapping in sequence, and outputs action instructions for the target top drive angle increment and torque compensation amount for controlling the top drive torsion; the value network is used to correct the policy parameters of the previous round of network based on the deviation between the current drilling state data and the reward function to obtain the network policy parameters of this round; wherein, the rules for correcting the policy parameters of the previous round based on the deviation between the current drilling state data and the reward function include: calculating the expected return of the current state-action pair based on the current drilling state data and the corresponding reward function output value; comparing the expected return with the historical evaluation value output by the previous round of value network to obtain the action value error term; using the error term as the loss function input, performing parameter update on the policy network through gradient back propagation to obtain the network policy parameters of this round.

[0038] Furthermore, the construction rules of the reward function include: extracting the tool face angle deviation, current top drive angle and current torque value that characterize the control performance from the current drilling status data as target indicators; performing normalization processing on each target indicator to obtain a set of indicators with a unified dimension; based on the type of well section in the drilling stage, matching corresponding weight parameters from a preset weight configuration set, and assigning weighted coefficients to each indicator; performing linear combination of the weighted indicator set to obtain the corresponding reward function.

[0039] In this embodiment of the present invention, after constructing the state vector, a pre-trained action decision model is invoked based on the state vector to generate action commands for the current control cycle in order to achieve intelligent decision-making control of the top drive's torsional swing behavior. This action decision model is modeled and trained using the Deep Deterministic Policy Gradient (DDPG) algorithm from deep reinforcement learning. This model is capable of high-precision policy optimization in a continuous action space, making it suitable for the continuous angle and torque control requirements of actual drilling processes.

[0040] The action decision model primarily consists of two functional networks: a policy network (the "Actor" network) and a value network (the "Critic" network). The policy network performs forward reasoning based on the current state vector and outputs action commands for controlling the top drive's torsional swing behavior. The value network evaluates the value of the current state-action pair over the long-term control process, providing an evaluation signal for updating the policy parameters.

[0041] Specifically, the policy network receives as input a constructed state vector, which has been dimensionally pruned, structurally expanded, and feature-weighted based on the well section conditions and the state attention mechanism. To ensure stability and numerical consistency during subsequent network training and inference, the input state vector is first normalized. This involves mapping all feature dimensions to a uniform range of [–1, 1] or [0, 1]. Z-score normalization or Min-Max normalization is often used.

[0042] After the standardization process is completed, the feature encoding stage begins. In this stage, the state vector is encoded into a high-dimensional strategy feature representation by superimposing multiple layers of nonlinear activation functions (such as ReLU, Tanh) and mapping with a fully connected layer. The encoded result is input into the action mapping layer, which consists of one or more continuous value output nodes, corresponding to the angle increment output node and the torque compensation output node in the top drive control target. The output value constitutes the action instruction under the current control cycle. In order to ensure the controllable range of the action output, an activation function such as hyperbolic tangent (tanh) or sigmoid is usually added at the end of the action mapping layer to compress the output result so that it falls within the preset physical safety control range.

[0043] The action command contains two elements: one is the target top drive angle increment, which reflects the correction trend and amplitude of the model for the current tool face offset; the other is the torque compensation amount, which is expressed as an incremental correction value on top of the current basic torque signal, used to suppress oscillation and improve tracking stability.

[0044] After action generation is complete, the action decision model must undergo periodic strategy optimization to ensure its continued adaptability to changing formation conditions and drillstring response. This optimization process primarily relies on the evaluation mechanism of the value network and the feedback signal of the reward function.

[0045] During each control cycle, a state-action pair is constructed based on the pairing of current drilling status data and the results of executed actions. This pair is then fed into a value network for evaluation. This network is typically a multilayer perceptron structure, with the input consisting of the concatenation of the state vector and the action vector. Its output is the cumulative expected reward for this state-action pair under the desired control objective, denoted as Q(s, a). To quantify the accuracy of this evaluation and determine the direction of correction, the difference between this evaluation and the actual reward function value obtained for the same state-action pair in the previous control cycle is calculated to form the action value error term ΔQ.

[0046] This error term, ΔQ, serves as the basis for the loss function and participates in the subsequent policy parameter optimization process. The loss function typically adopts a mean squared error (MSE) structure, with ΔQ² serving as the optimization objective. Next, backpropagation is used to calculate gradients for the policy network's weight parameters, and an optimizer (such as Adam) is used to update the parameters. This optimization process can be combined with a target network soft update mechanism and an experience replay mechanism to further enhance model training stability and generalization. The target network soft update mechanism implements a set of "delayed update replicas" for each policy network and value network, periodically and slowly replicating the main network weights with a small step size (e.g., τ = 0.005) to avoid drastic fluctuations during training. The experience replay mechanism, on the other hand, establishes a cache queue of historical state-action-reward-next-state four-tuples and randomly samples from them during each update round, breaking temporal correlation between samples and improving convergence speed and policy generalization.

[0047] The core of this strategy optimization process relies on the quality of the reward function design. To ensure that the reward function maps well to the key control objectives during drilling, the reward function construction process needs to be modeled based on the control indicators in actual drilling conditions.

[0048] Specifically, in each control cycle, three types of variables closely related to control accuracy are extracted from the current drilling status data: tool face angle deviation (i.e., the difference between the actual tool face angle and the expected angle), the current top drive angle (used to reflect the aggressiveness of the control behavior), and the current torque value (used to reflect the mechanical load during execution). These three variables represent the accuracy target, control stability target, and energy balance target, respectively, and are commonly used multi-objective optimization indicators in reinforcement learning control. In order to facilitate consistency in quantitative processing and model processing, the above variables need to be normalized, and the following method is often used: 1) Tool face angle deviation: linear compression is performed in the range of ±180°; 2) Top drive angle: Min-Max normalization based on historical angle response range; 3) Torque value: Percentage scaling is performed based on the set maximum allowable torque.

[0049] The normalized set of indicators uniformly constitutes the reward function input variable set. Next, based on the drilling stage and structural characteristics of the current well section, the matching weight parameter group is retrieved from the preset weight configuration set, and the weighted coefficients corresponding to the above three target indicators are assigned respectively. For example, in the early stage of drilling a long horizontal section, in order to ensure the trajectory guidance accuracy, the tool face angle deviation weight can be set to 0.6, the top drive angle weight to 0.3, and the torque value weight to 0.1; when the control action tends to be stable at the end of the well section, the torque weight can be appropriately increased to reduce the angle jump behavior. Finally, the weighted normalized indicators are linearly weighted combined to construct the reward function value R(t) within the current control cycle. This reward function R(t) serves as the evaluation target for the value network in the next control cycle, and is compared and optimized with the behavior output by the policy network. This approach forms a closed-loop control link. By introducing a DDPG-based deep reinforcement learning policy network structure, this solution allows the top drive torsion swing control process during drilling to break away from the constraints of a static rule base and instead implement adaptive policy updates in a data-driven and feedback-driven manner. This approach exhibits excellent dynamic adaptability and control accuracy, particularly under complex operating conditions such as multi-section switching, high nonlinear friction, and tool face angle response hysteresis. This significantly improves the control system's tracking capability and response sensitivity to the tool face angle, enhancing the stability and intelligence of the overall drilling trajectory control.

[0050] In one possible implementation, the state vector is represented as:

[0051] Among them, the tool face deviation ΔTF reflects the target tracking error; the top drive torque T surface Characterizes the stress state of the drill string; swing angle θ swing and its rate of change θ′ swing Describe the current action characteristics; vibration level v level Evaluate the safety status of the mine. The control command output is a two-dimensional action vector:

[0052] The former controls the angular span of a single swing (range ±5°), and the latter adjusts the torque compensation (range ±3kN·m). The policy network minimizes the value function To achieve action optimization:

[0053] in For the target critic network, For the target actor network, the discount factor γ=0.99 balances the current benefits and long-term effects. The reward function is designed as:

[0054] This formula allocates 50% weight to toolface tracking accuracy, 30% to suppressing torque fluctuations, and 20% to constraining motion amplitude, guiding the system to achieve a balance between precision and safety. These weightings were determined through sensitivity analysis, combined with field experience and domain knowledge.

[0055] The algorithm achieves stable learning through a dual actor-critic network architecture. Both the actor and critic networks utilize a three-layer, fully connected neural network design, balancing feature extraction efficiency with real-time control requirements. The actor network's input layer receives a normalized state vector (including parameters such as toolface deviation and top drive torque). Feature abstraction is performed through the first hidden layer (512 nodes) and the second hidden layer (256 nodes). The Reluctant Unit (ReLU) activation function is used to enhance nonlinear representation. The output layer constrains the action range using the hyperbolic tangent (Tanh) function and linearly maps it to actual physical quantities, namely yaw angle and torque. The critic network's input layer fuses the state vector with the action command. The hidden layer uses the LeakyReLU activation function to mitigate gradient sparsity issues. The output layer directly generates an action value score. Network weights are initialized using the He initialization method adapted to nonlinear units. Input data is preprocessed using online mean-variance normalization. A layer normalization module is added after the hidden layer to improve training stability.

[0056] To improve training efficiency and avoid strong correlations between data, an experience replay mechanism is employed. This mechanism stores past states, actions, rewards, and the next state (st, at, rt, st+1) in a buffer. During training, a random batch of data is sampled to update the network. This allows for reuse of old data while reducing bias introduced by consecutive samples. Furthermore, to avoid drastic fluctuations in value estimates during training, the algorithm introduces a target network. Its parameters are not updated immediately, but rather are slowly synchronized with the parameters of the online network through a soft update. The update formula is:

[0057] in Refers to the target network parameters, which are used to provide the training target of the stabilization method. Refers to online network parameters, which are updated in real time through interaction with the environment. is the soft update coefficient, The value of is determined to be 0.005 through multiple experiments, and the smaller The value can ensure that the target network parameters are updated slowly, maintain the stability of training, and ensure that the changes of the target network are smooth enough.

[0058] For motion exploration, attenuated Gaussian noise is used, and its standard deviation decays exponentially with the number of training steps:

[0059] Random noise is added to the actions output by the actor network. This results in higher noise in the early stages of training (when t is small), encouraging experimentation with different actions to fully explore the environment. However, as the number of training steps increases, the noise gradually decreases, ultimately stabilizing the policy. This design enables extensive exploration early on and precise optimization later, enabling efficient and stable learning in complex tasks such as continuous action space problems.

[0060] Step S30: parsing the action instruction to determine the target physical action.

[0061] Specifically, the action instruction is parsed to extract the angle increment and torque compensation for the top drive torsion; the angle increment and the current actual rotation angle of the top drive are linearly accumulated, and the accumulated value is limited to obtain a target top drive angle value; dynamic limit adjustment is performed on the torque compensation amount, and the maximum allowable torque output at the corresponding level is determined according to the downhole vibration level feedback to obtain the target torque; the target top drive angle value and target torque are used as the target physical action.

[0062] In this embodiment of the present invention, after generating action instructions based on a reinforcement learning model, to ensure that the outputted actions can accurately and stably drive the top drive torsional pendulum behavior in accordance with actual physical execution conditions, the action instructions need to be structured and parsed, and converted into clear physical control parameters, namely the target top drive angle value and target torque output value. This process not only involves numerical decoding of the action instructions themselves, but also requires limiting and dynamic adjustment based on the current equipment status and downhole operating conditions to ensure the safety of the execution process and the rationality of the response.

[0063] Specifically, an action command generally consists of two components: an angle control component, typically expressed as a target angle increment relative to the current top drive angular position; and a torque control component, expressed as a torque compensation applied to the base load. First, these two components are extracted separately from the action command. The angle increment is linearly added to the current top drive's actual real-time angle to form the target top drive angle for the next control cycle. The current top drive's actual angle can be obtained in a time-synchronized manner, provided by a surface angle encoder or rotation sensor.

[0064] To prevent excessive jumps in angle control commands, particularly under high friction conditions in horizontal sections, which could cause drillstring "reverse torque oscillation" or "attitude overshoot," the target angle value must be clipped. This clipping is based on a set physical safety range, for example, limiting the top drive angle control range to ±180°. When the target angle exceeds this range, saturation clipping is performed. Dynamic clipping is performed on the torque compensation in the action command. Unlike fixed-boundary strategies for angle control, torque output is more state-dependent, necessitating a dynamic calculation of the maximum allowable torque threshold based on downhole vibration level feedback. This vibration level is derived from axial and radial acceleration signals fed back by the measurement while drilling (MWD) tool and is classified into multiple levels, such as V0 to V3, where higher levels indicate greater downhole vibration. For each vibration level, a set of maximum allowable torque coefficients is preset, which are significantly reduced at higher vibration levels to protect the drillstring and top drive. Finally, the base torque value is added to the command compensation to determine the target torque value to be executed for dynamic clipping.

[0065] Based on the present invention's solution, the target physical action consists of two components: the top drive target angle value after limiting and the target torque output value after limiting. These two control quantities jointly determine the actual execution state of the top drive within the current control cycle. By introducing a state-dependent dynamic limiting mechanism, the execution safety and response accuracy of the control action are improved while also avoiding the risk of control oscillation or overload caused by fluctuations in the strategy output. This further enhances the adaptability and engineering usability of the reinforcement learning control strategy in complex drilling environments.

[0066] In a possible implementation, during the instruction parsing phase, the motion vector is first converted into a physical quantity. swing The actual angle A of the current top drive current Linear superposition to generate target angle A target =A current +ΔA swing This method can make the angle adjustment process smoother and avoid sudden, large changes that could cause equipment shaking or loss of control. This is then converted into a speed command for the drive motor via a three-loop PID controller. Through simulation tests and combined with actual drilling data, the proportional coefficient Kp is set to 0.8 to ensure rapid response to angle deviations, the integral coefficient Ki is set to 0.1 to eliminate small errors accumulated over time, and the differential coefficient Kd is set to 0.3 to suppress excessive changes and prevent system jitter. The cycle of this control process is strictly limited to less than 50 milliseconds. This time requirement ensures that the system can quickly respond to angle changes, maintain adjustment speed, and accurately stop at the target position, without being sluggish or oscillating due to over-adjustment.

[0067] To ensure the safety of the mechanical system, a three-layer physical constraint mechanism is built in. At the angle control level, the cumulative swing angle is strictly limited to the range of ±180°. When the target angle calculation value exceeds this range, the system automatically performs hard limit truncation. For example: if A target >180° then force setting A final =180°, on the contrary when A target When <−180°, set A final = −180°. At the torque control level, in addition to filtering, emergency recentering is immediately triggered when the downhole vibration level vlevel ≥ 4 or the tool face deviation ΔTF > 45°: the swing angle is reduced to 50% of its current value within 0.5 seconds and then swung back to the reference position (Abase = 0°) at a constant angular velocity of 2° / s. The torque output is simultaneously reset to 70% of the reference value until condition monitoring confirms that downhole conditions have returned to a safe state.

[0068] During the execution, the actual angle A is continuously received from the top drive feedback. actual With torque sensor data T feedback The data is then transmitted to the state detection for subsequent calculations, providing key data support for the real-time optimization of reinforcement learning.

[0069] In another possible implementation, during the drilling of some long horizontal well sections, due to the long drillstring and its highly flexible configuration, the surface-applied top drive yaw angle is not immediately transmitted to the downhole sliding drill bit. This results in a significant angular response lag, manifesting as the toolface not immediately responding despite the control angle being adjusted. This can lead to trajectory deviation and target instability. To overcome this "delayed response," this embodiment introduces an angle prediction and correction mechanism based on a drillstring axial compression wave propagation model during the motion command parsing phase to pre-compensate for the target angle setting.

[0070] Specifically, after obtaining the angle increment value contained in the action command, the current drill string length 𝐿, drill string elastic modulus 𝐸, cross-sectional area 𝐴, and unit mass 𝜌 are calculated first. Then, combined with the difference in surface-downhole torque feedback, the drill string propagation velocity model is constructed:

[0071] According to the current length of the drill string and propagation speed , which can calculate the estimated delay time for the ground control signal to be transmitted to the drill bit end . Further based on the top drive in the past Angle trend within the time window, using a first-order extrapolation model to predict the potential hysteresis of downhole angle response , and compensate the current angle increment instruction:

[0072] The compensated angle is used as the target top drive angle value to participate in the limit judgment and subsequent physical action construction. This mechanism can effectively improve the angle control feedforward performance under long drill string structures, reduce tool face angle delay and overshoot, and improve the consistency between control instructions and downhole responses. Through this embodiment, not only the axial dynamic characteristics of the drill string are integrated, but also the application effect of the control model under the condition of "structural long delay chain" is enhanced by introducing dynamic prediction compensation logic. It is suitable for complex well section scenarios with a well depth of more than 2,500 meters and a horizontal section length of more than 1,000 meters. It is still an unpopular design in existing control strategies and has strong technical recognition and creative value.

[0073] Step S40: executing the top drive torsion swing based on the target physical action, and collecting the execution result until the execution result indicates that the tool face angle meets the expectation.

[0074] Specifically, during the execution of top drive torsion based on the target physical action, the method also includes: continuously monitoring the tool face angle deviation and the downhole vibration level; when the tool face angle deviation exceeds a first preset threshold, or the downhole vibration level reaches a second preset level, triggering an emergency return to centering operation; wherein, the emergency return to centering operation includes: reducing the current target top drive angle value to a first target proportional value of the original value to form a transition angle target; performing angle adjustment based on the transition angle target until the top drive angle returns to a preset safety reference angle; synchronously reducing the target torque to a second target proportional value of the preset safety reference torque, and maintaining this state until the tool face angle deviation is within the first stable threshold range and the downhole vibration level is lower than the second stable threshold for more than a preset recovery period.

[0075] In this implementation, the top drive is driven to achieve the desired angle adjustment and torque output during the execution of the target physical action, thereby controlling the downhole tool face angle. This process continues until feedback from the measurement while drilling tool indicates that the current downhole tool face angle has essentially reached the preset target, and the control action is considered to be effectively closed.

[0076] In actual drilling conditions, due to complex well sections and significant variations in formation properties, toolface control can encounter issues such as angle drift and vibration overlay, leading to unstable top drive torsional behavior. To ensure control accuracy and drillstring structural safety, a set of safety monitoring and return-to-center protection logic must be implemented simultaneously during control operations.

[0077] Specifically, during the execution of control instructions, two key state quantities are continuously monitored in real time. One is the tool face angle deviation, that is, the difference between the actual tool face angle downhole and the control target; the other is the downhole vibration level. By analyzing the vibration sensor data sent back by the MWD, it is divided into several levels to evaluate the dynamic load intensity currently borne by the drill string.

[0078] When the tool face angle deviation exceeds the set safety threshold, or the vibration level reaches a high-risk level (such as severe axial vibration or frequent lateral jumps), an emergency return to center operation is immediately triggered. The so-called return to center operation refers to actively interrupting the execution process of the current control angle and torque, and gradually guiding the top drive state back to a safe and stable initial position. In this process, the target angle value originally planned to be executed is first reduced. Usually, the angle is reduced proportionally, for example, set to half of the original value, as a temporary transition angle target. Then, instead of returning to the origin immediately, the top drive angle is gradually brought back to the default "reference angle" through a smooth and controlled angle adjustment process. This reference value is usually zero degrees or the neutral angle set by other equipment at the factory.

[0079] At the same time, the current torque output must be reduced. Torque control doesn't simply disconnect the drive, but rather reduces output to a percentage of a pre-set safety reference torque, such as 70%. This is done to reduce elastic rebound of the drill string or top drive unloading shock caused by a sudden load reduction.

[0080] After the emergency return to center operation is completed, continue to monitor the changing trends of the tool face angle deviation and vibration level. Only when both indicators remain within the safe range and maintain a complete recovery cycle (for example, 5 consecutive seconds of stability) is the downhole state considered to have recovered, the return to center protection can be released, and the normal control action execution phase can be re-entered. This dynamic and safe return to center strategy can effectively reduce the risk of equipment overload in complex well sections, severe disturbances, or abnormal response conditions, improve the robustness and operational recoverability of top drive control, and is an important guarantee for ensuring the long-term stable operation of the entire control solution in actual engineering scenarios.

[0081] Preferably, after the execution result indicates that the tool face angle meets expectations, the method further includes: collecting feedback information of the top drive equipment including the current top drive actual angle and the current torque output value; synchronously receiving the latest tool face angle value and vibration level in the downhole measurement data to form a feedback data set after the action is executed based on the current top drive actual angle and the current torque output value, the latest tool face angle value and the vibration level; performing time alignment and interpolation completion operations on the feedback data set to construct a state feedback vector; merging the state feedback vector with the original state vector, replacing the historical values of the corresponding dimensions therein to form a new state vector; and using the new state vector as input data for the action decision model in the next cycle.

[0082] In this embodiment of the present invention, after completing the actions of the current control cycle and determining that the toolface angle has been essentially adjusted to the desired target, to ensure the continued real-time and adaptability of subsequent control strategies, it is necessary to effectively recover and update the various state data following this round of actions, thereby completing the iterative construction of the state vector. This stage is not simply data collection, but a closed-loop processing process involving data fusion, time alignment, and dimension replacement. Its ultimate goal is to provide a complete and accurate input foundation for the next control cycle.

[0083] Specifically, the top drive's execution feedback at the end of the current control cycle is collected. This information includes at least the top drive's actual angle and torque output. The former is provided in real time by the top drive encoder, while the latter is obtained from the torque sensor and represents the control execution status on the ground.

[0084] At the same time, the latest data uploaded by the downhole measurement while drilling (MWD) tool must be received simultaneously. This primarily includes the tool face angle measurement for the current well section and the drill string vibration level. The tool face angle reflects the actual response of the lower drill string, while the vibration level is used to determine the forces and disturbances on the drill string in the formation. This level is often calculated using a multi-axis accelerometer and is categorized into different levels to represent the degree of risk.

[0085] The above four data items—drive angle, torque value, tool face angle, and vibration level—together constitute the feedback data set after the current control cycle is completed. Since these data originate from surface and downhole equipment, their acquisition frequencies, upload delays, and data timings are often inconsistent, so time alignment must first be performed on the feedback data set. The rules for time alignment are typically based on sliding matching of timestamp windows: The time of acquisition by the surface equipment is used as a reference, and the nearest timestamp point in the downhole data is searched forward or backward for matching. If data at a certain moment is missing, the data is supplemented numerically using an interpolation algorithm. Common methods include linear interpolation, spline interpolation, or sliding window averaging.

[0086] After time alignment and interpolation, a complete state feedback vector is constructed. This state feedback vector is merged with the original state vector from the previous round. This merging process does not cover all aspects of the state vector, but rather selectively updates some dimensions of the state vector—typically those that vary significantly with execution feedback, such as angle, torque, and vibration level. Other long-term static features, such as the drill string structure code and well section classification, remain unchanged.

[0087] After the merger is complete, the latest state vector is formed, which serves as the basis for input into the reinforcement learning action decision model for the next control cycle. Because this state vector integrates surface execution information and downhole response information, it has good timeliness and reflectiveness of working conditions, providing a more accurate environmental representation for the model strategy output.

[0088] This feedback construction and state update mechanism not only effectively addresses the issues of asynchronous and time-sensitive data between surface and downhole data, but also enhances the model's closed-loop responsiveness within the control process, ensuring that the state information underlying the strategy update process is authentic, accurate, and traceable, thereby improving the stability and practicality of the entire reinforcement learning control process. This mechanism significantly enhances the sensitivity and adaptability of strategy responses, particularly during sliding drilling, long horizontal sections, or areas with drastic changes in well inclination, avoiding the accumulation of control deviations caused by delayed state perception.

[0089] Figure 2 This is a system structure diagram of a downhole tool face dynamic control system based on reinforcement learning provided by an embodiment of the present invention. Figure 2 As shown, an embodiment of the present invention provides a downhole tool face dynamic control system based on reinforcement learning, the system comprising: an acquisition unit, for acquiring drilling status data of a target well, performing data preprocessing on the drilling status data, and constructing a corresponding state vector based on the preprocessed drilling status data; a decision unit, for calling an action decision model trained based on a DDPG algorithm based on the state vector to obtain an action instruction; wherein the network strategy parameters of the action decision model are obtained after correction of the deviation between the previous round of network strategy parameters based on the current drilling status data and a pre-constructed reward function; a parsing unit, for parsing the action instruction to determine a target physical action; and an execution unit, for executing a top drive torsion swing based on the target physical action, and recovering the execution result until the execution result indicates that the tool face angle meets the expectation. An embodiment of the present invention further provides a computer-readable storage medium having instructions stored thereon, which, when executed on a computer, enables the computer to execute the above-mentioned reinforcement learning-based downhole tool face dynamic control method.

[0090] Those skilled in the art will appreciate that all or part of the steps in the methods described in the aforementioned embodiments can be performed by instructing the relevant hardware through a program. The program, stored in a storage medium, includes instructions for causing a microcontroller, chip, or processor to execute all or part of the steps in the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0091] The above describes in detail the optional embodiments of the present invention in conjunction with the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above embodiments. Within the technical concept of the embodiments of the present invention, a variety of simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all fall within the scope of protection of the embodiments of the present invention. It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner unless there is any contradiction. In order to avoid unnecessary repetition, the embodiments of the present invention will no longer describe the various possible combinations separately.

[0092] In addition, the various embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the embodiments of the present invention, they should also be regarded as the contents disclosed in the embodiments of the present invention.

Claims

1. A downhole tool face dynamic control method based on reinforcement learning, characterized in that: The method comprises: collecting drilling status data of a target well, performing data preprocessing on the drilling status data, and constructing a corresponding state vector based on the preprocessed drilling status data; Based on the state vector, the action decision model trained based on the DDPG algorithm is called to obtain the action instruction; wherein, The network strategy parameters of the action decision model are obtained by correcting the deviation between the previous round of network strategy parameters and the pre-built reward function based on the current drilling status data; Parsing the action instruction to determine a target physical action; The top drive oscillation is executed based on the target physical action, and the execution results are collected until the execution results indicate that the tool face angle meets the expectation.

2. The method according to claim 1, characterized in that Performing data preprocessing on the drilling status data and constructing a corresponding state vector based on the preprocessed drilling status data includes: The drilling status data is sequentially subjected to multi-stage domain filtering processing, wavelet denoising processing, and interpolation synchronization processing based on a timestamp sequence to obtain pre-processed data; Perform feature extraction on the formation type, well inclination angle change rate and drill string structure parameters contained in the pre-processed drilling status data to obtain feature extraction results; Generate a well section operating condition feature vector based on the feature extraction result, and match the well section operating condition feature vector with a preset state template to determine a target feature set and a normalization processing method for constructing a state vector; The dimension and parameter structure of the state vector generated based on the preprocessed data are cropped and / or expanded according to the matched target feature set and normalization processing method to obtain the corresponding state vector.

3. The method according to claim 1, characterized in that After constructing the corresponding state vector based on the preprocessed drilling state data, the method further includes: A state attention mapping structure is constructed based on a well section operating condition feature vector, wherein the well section operating condition feature vector includes a well inclination angle change rate, a wellbore curvature grade, and a drill string stiffness factor. The well section operating condition feature is used to characterize the response characteristics of different well sections. In each control cycle, the state attention mapping structure is used to retrieve the attention weight group corresponding to the current well section from the pre-built state dimension-attention weight mapping table; The attention weight group is dimensionally matched with the state vector, and a weighted operation is performed item by item to form a weighted state vector.

4. The method according to claim 1, wherein The action decision model includes a policy network for outputting control actions through forward reasoning and a value network for estimating state-action value functions; The strategy network receives the state vector as input, performs normalization processing, feature encoding processing and continuous action mapping in sequence, and outputs action instructions for the target top drive angle increment and torque compensation amount for controlling the top drive torsion. The value network is used to modify the network strategy parameters of the previous round based on the deviation between the current drilling status data and the reward function to obtain the network strategy parameters of this round; wherein, The rules for correcting the previous round of network policy parameters based on the deviation between the current drilling status data and the reward function include: Calculate the expected return of the current state-action pair based on the current drilling state data and the corresponding reward function output value; Compare the expected return with the historical evaluation value output by the previous round of value network to obtain the action value error term; The error term is used as the input of the loss function, and the policy network parameters are updated through gradient back propagation to obtain the network policy parameters of this round.

5. The method according to claim 1, characterized in that The construction rules of the reward function include: Extracting the tool face angle deviation, current top drive angle, and current torque value from the current drilling status data, which represent the control performance, as target indicators; Perform normalization processing on each target indicator to obtain an indicator set with a unified dimension; Based on the well section type during the drilling phase, the corresponding weight parameters are matched from the preset weight configuration set, and weighted coefficients are assigned to each indicator; Perform a linear combination of the weighted indicator set to obtain the corresponding reward function.

6. The method according to claim 1, wherein Parsing the action instruction to determine the target physical action includes: parsing the motion command to extract the angle increment and torque compensation for the top drive oscillation; Linearly accumulating the angle increment and the current actual top drive rotation angle, and performing a clipping process on the accumulated value to obtain a target top drive angle value; Performing dynamic limit adjustment on the torque compensation amount, determining the maximum allowable torque output at the corresponding level based on downhole vibration level feedback, and obtaining the target torque; The target top drive angle value and the target torque are used as target physical actions.

7. The method according to claim 1, characterized in that During execution of the top drive oscillation based on the target physical action, the method further includes: Continuously monitor tool face angle deviation and downhole vibration levels; When the tool face angle deviation exceeds the first preset threshold, or the downhole vibration level reaches the second preset level, an emergency return operation is triggered; wherein, The emergency return operation includes: Reducing the current target top drive angle value to a first target proportional value of the original value to form a transition angle target; Execute angle adjustment based on the transition angle target until the top drive angle returns to the preset safety reference angle; The target torque is simultaneously reduced to a second target proportional value of the preset safety reference torque, and this state is maintained until the tool face angle deviation is within the first stable threshold range and the downhole vibration level is lower than the second stable threshold for more than a preset recovery period.

8. The method according to claim 1, characterized in that After the execution result indicates that the tool face angle meets expectations, the method further includes: Collect feedback information from the top drive device, including the current top drive actual angle and current torque output value; Synchronously receiving the latest tool face angle value and vibration level in the downhole measurement data to form a feedback data set after the action is executed based on the current top drive actual angle and current torque output value, the latest tool face angle value and vibration level; Performing time alignment and interpolation completion operations on the feedback data set to construct a state feedback vector; Merge the state feedback vector with the original state vector, replacing the historical values of the corresponding dimensions to form a new state vector; The new state vector is used as the input data of the action decision model in the next cycle.

9. A downhole tool face dynamic control system based on reinforcement learning, characterized in that: The system comprises: an acquisition unit, configured to acquire drilling status data of a target well, perform data preprocessing on the drilling status data, and construct a corresponding state vector based on the preprocessed drilling status data; A decision unit is used to call the action decision model trained based on the DDPG algorithm based on the state vector to obtain an action instruction; wherein, The network strategy parameters of the action decision model are obtained by correcting the deviation between the previous round of network strategy parameters and the pre-built reward function based on the current drilling status data; A parsing unit, configured to parse the action instruction and determine a target physical action; The execution unit is used to execute the top drive torsion swing based on the target physical action and collect the execution result until the execution result indicates that the tool face angle meets the expectation.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the downhole tool face dynamic control method based on reinforcement learning as described in any one of claims 1 to 8.

Images