Early identification method of drilling overflow based on deep reinforcement learning

Abstract

This invention is applicable to the field of oil drilling engineering technology and provides a method for early identification of drilling overflows based on deep reinforcement learning. The method includes: first, data collection and preprocessing; then, building and optimizing a drilling overflow early identification model based on the proximal policy optimization (PPO) algorithm; and finally, real-time data collected during drilling operations is preprocessed and input into the model. The PPO algorithm analyzes changes in characteristic quantities, and determines whether there are signs of overflow according to the overflow drilling process. If so, an early warning is issued based on a reward function strategy. This method combines traditional threshold methods with machine learning, utilizing a deep reinforcement learning algorithm to optimize drilling parameter thresholds. It monitors characteristic quantities in real time to capture changes before overflows occur, and self-learns historical data to identify overflow patterns, reducing reliance on on-site operators. Furthermore, the method emphasizes demonstrating the model's decision logic to enhance engineers' understanding and trust, providing more reliable overflow monitoring for drilling operations, reducing accident risks, and ensuring wellbore safety.

Description

Technical Field

[0001] The present invention belongs to the technical field of petroleum drilling engineering, and in particular relates to a method for early identification of drilling overflows based on deep reinforcement learning. Background Art

[0002] In the oil drilling industry, spills are a critical issue. Once a drilling spill occurs, it poses multiple risks. Downhole, this signifies a loss of downhole pressure management, potentially damaging the reservoir directly. The intrusion of pollutants can impact the reservoir's long-term production efficiency and, in severe cases, even cause permanent damage, leading to energy resource loss. Economically, addressing spills requires significant human, material, and financial resources, placing a heavy burden on drilling companies. More seriously, if a spill is not properly addressed in its early stages, it can rapidly develop into a kick or even a blowout. A blowout, an extreme form of well control failure, can lead to serious consequences such as toxic gas leaks, fires, and explosions. These events not only directly threaten the lives of drilling platform workers but also have long-term negative impacts on the surrounding ecosystem. Therefore, early identification and timely handling of spills are crucial to ensuring personnel safety, minimizing economic losses, and preventing more serious accidents.

[0003] Currently, drilling overflow monitoring technologies primarily include traditional threshold methods and machine learning approaches. Traditional threshold methods employ a series of predetermined safety thresholds to monitor key drilling parameters in real time, such as inlet and outlet flow differentials, vertical pressure, methane, ethane, pump stroke, and hook height. When a parameter fluctuates beyond a threshold, the monitoring system issues an alarm. Setting thresholds typically relies on historical data analysis, geological condition assessments, and engineer experience. While this method provides a basic approach for drilling overflow monitoring, it presents numerous challenges. It relies heavily on individual engineer experience, and different engineers may interpret the same situation differently. Rapidly changing drilling environments can lead to delayed responses to real-time data, hindering timely control of overflows. Furthermore, geological conditions are complex and variable, and a single threshold approach can easily lead to false positives or false negatives, making it difficult to adapt to all drilling environments. This problem is particularly prominent in the exploration and development of unconventional oil and gas resources.

[0004] In recent years, the application of machine learning methods in drilling flooding monitoring has steadily increased. Commonly used techniques include support vector machines, decision trees, random forests, and deep learning networks. These methods are capable of analyzing large amounts of drilling data and identifying flooding signs promptly and accurately. For example, Shi Xiaoyan et al. proposed a solution based on the random forest algorithm, improving the accuracy and real-time performance of flooding detection; Sun Baojiang et al. proposed an integrated pattern recognition model to successfully diagnose gas surge faults; and Li Yufei et al. proposed an intelligent early flooding identification method based on support vector machines (SVMs) and DS evidence theory, improving monitoring reliability. However, most machine learning methods operate independently of traditional threshold methods. For frontline drillers and field engineers, these methods are often viewed as "black box" operations, lacking transparency into their internal working mechanisms and decision-making logic. This makes it difficult for frontline drillers to understand the basis for warnings, undermining their trust in the system and the timeliness of warning responses. When the algorithm's predictions do not align with workers' experience, this can lead to hesitation in decision-making or even disregard of the algorithm's warnings, posing safety risks in emergency situations.

[0005] In order to solve the above problems, the present invention proposes an early identification method for drilling overflow based on deep reinforcement learning. Summary of the Invention

[0006] The purpose of the present invention is to provide a method for early identification of drilling overflow based on deep reinforcement learning, aiming to solve the problems raised in the above background technology.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] The early identification method of drilling overflow based on deep reinforcement learning includes the following steps:

[0009] Step 1: Data collection and preprocessing;

[0010] The collected data was cleaned using real historical drilling data. A time window method was used to construct samples. Six characteristic quantities were selected: the difference between the outlet and inlet flow rates, vertical pressure, methane, ethane, pump stroke, and hook height. The mean, slope, and variance of the characteristic quantities within the time window were calculated. By determining whether the change in the statistical quantity of adjacent windows exceeded the threshold, it was determined whether it was an overflow and the label was recorded.

[0011] Step 2: Construction and optimization of the drilling overflow early identification model based on the proximal strategy optimization algorithm, including parameter initialization, main iteration loop, strategy parameter update, value function learning, and iteration loop steps;

[0012] Step 3: Early identification of drilling overflow;

[0013] After preprocessing, real-time data is input into an early detection model for drilling overflows based on a proximal strategy optimization algorithm. The model uses parameters optimized by the proximal strategy optimization algorithm to analyze changes in inlet and outlet flow differences, vertical pressure, methane, ethane, pump stroke, and hook height. It then identifies signs of overflow by identifying the overflow drilling process. If overflow signs are detected, an early warning is issued based on the strategy determined by the reward function.

[0014] Furthermore, in step 1, invalid data points are replaced with zeros and features with incomplete information are eliminated during data cleaning; when constructing samples using the time window method, a time series matrix is ​​constructed in units of 380 time slices, with a sliding step size of 1.

[0015] Furthermore, the specific process of step 2 is as follows:

[0016] Step 21: Parameter initialization;

[0017] Setting hyperparameter cutoff factors , strategy update sub-iteration number M The sum value function updates the number of sub-iterations B ; Initialize strategy parameters θ and the initial value function parameters ;

[0018] Step 22: Main iteration loop;

[0019] Starting from the first iteration, , each iteration performs operations in the following order:

[0020] Trajectory Collection and Reward Calculation: Executing the Current Policy in a Drilling Environment π θk , collect trajectory dataset D k ={ τ i}, and calculate the cumulative reward for each time step G t ,in τ i is the collected trajectory data;

[0021] Advantage function estimation: based on the current value function , the advantage estimation method is used to calculate the advantage value of each state-action pair ;

[0022] Step 23: Update strategy parameters;

[0023] Entering the strategy optimization phase, the strategy parameters are updated through the following sub-iteration process:

[0024] right Execute in sequence:

[0025] Calculate the probability ratio of the new and old strategies:

[0026] ;

[0027] Where, is the probability ratio of the new and old strategies; Strategies before and after the update; This is the strategy before the update; θ old are the strategy parameters before updating; is the updated strategy parameter; is the advantage value of the state-action pair; is the time step t The state of the environment;

[0028] The Adam optimizer is used to maximize the truncated objective function of PPO through stochastic gradient ascent:

[0029] ;

[0030] Where, For the k +Strategy network parameters after 1 strategy optimization; is the number of iterations; is the trajectory point; is the trajectory set; is the time step; is the advantage value of the state-action pair; is the probability ratio of the new and old strategies; is the clipping function; is the hyperparameter cutoff factor;

[0031] Step 24: Value function learning;

[0032] After completing the policy parameter update, the value function optimization phase begins, where the parameters are updated through the following sub-iterations:

[0033] right Execute in sequence:

[0034] Use gradient descent to minimize the mean square error of the value function:

[0035] ;

[0036] Where, For the first k +Value network parameters after 1 round of strategy optimization; For the value network state valuation; is the cumulative reward for each time step;

[0037] Step 25: Iterate the loop;

[0038] Repeat steps 22 to 24 until the algorithm converges or reaches the preset termination condition to obtain the optimized model.

[0039] Furthermore, step 2 also includes a Reward function construction step, and the specific process is as follows:

[0040] Set a "sweet spot" and give the highest score to the marked points in the sweet spot. Give negative scores to the marked points outside the sweet spot but in normal state before and after the overflow occurs. Give positive scores to the points where the overflow actually occurs, which are much lower than the scores of the marked points in the sweet spot.

[0041] Define the structured action space:

[0042] ;

[0043] Where, is the action space, that is, the set of all possible actions; is the action vector; For each dimension of the action vector; is the set of real numbers, is an 18-dimensional real number set;

[0044] The scoring function formula is as follows:

[0045] ;

[0046] ;

[0047] Where, is the scoring function; Represents the number of marking points in the sweet spot; The number of marked points outside the sweet zone that are real overflow points; Represents the number of non-overflow points marked outside the sweet zone; Indicates that the conditions in square brackets are met number; is the coordinate of the real overflow point; The model predicts i The position of an overflow point in the time series.

[0048] Furthermore, in step 3, the process flow for determining overflow drilling is as follows:

[0049] First, check whether the inlet and outlet flow difference or vertical pressure increases. If not, the current flow change may be a normal fluctuation. If it increases, check whether the methane or ethane content increases. If the methane or ethane content increases, it is judged that there are signs of overflow. If not, check whether the pump stroke increases. If the pump stroke increases, the current flow change may be a normal fluctuation. If not, check whether the hook height decreases. If the hook height decreases, the current flow change may be a normal fluctuation. If not, it is judged that there are signs of overflow.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. High Accuracy and Detection Rate: The algorithm proposed in this paper has a 100% detection rate in the time window preceding an overflow and an overall accuracy rate of 79%. Compared with traditional machine learning algorithms such as support vector machines (SVM), random forests (RF), and back propagation neural networks (BP-NN), it has a higher correct recognition rate and a lower false alarm rate. It can provide drilling engineers with timely and accurate warnings, helping them take preventive measures to ensure wellbore safety and reduce economic losses, and has outstanding practical application value.

[0052] 2. Fast Response and Precise Positioning: The algorithm proposed in this paper surpasses competing algorithms in processing speed, resulting in a more rapid response to overflow events in practical applications. With a minimum distance (ND) of only 5 and an average distance (AD) of only 193 from the initial overflow point, it can accurately locate the overflow starting point, buying valuable time to prevent blowouts. This significantly improves the response speed and positioning accuracy to overflows, facilitating timely control of overflow hazards.

[0053] 3. Effectively reduce false alarm rates: This invention constructs a reasonable reward function to distinguish between normal operation and overflow events, assigning scores to marker points in different areas to reduce the possibility of false alarms. During drilling operations, this prevents false alarms from disrupting production order, increasing costs and risks, making the monitoring system more reliable and helping engineers make accurate decisions based on early warning information.

[0054] 4. Ensure operational safety and reduce economic losses: The algorithm proposed in this invention monitors overflows in advance with a low false alarm rate, providing engineers with timely warnings so that they can adjust drilling parameters and implement well-killing operations, thereby avoiding the deterioration of overflows, protecting personnel lives, reducing the human, material, and financial resources required to handle overflow accidents, and lowering financial costs, playing a key role in improving drilling operation safety and economic benefits.

[0055] In summary, this invention combines traditional thresholding methods with machine learning to develop an automated model that doesn't rely on individual experience. It utilizes a deep reinforcement learning algorithm to optimize drilling parameter thresholds, monitors multiple characteristics such as inlet and outlet flow differentials, vertical pressure, and methane in real time to capture pre-overflow changes, and can also self-learn from historical data to identify overflow patterns, reducing reliance on on-site operators. Furthermore, this invention emphasizes demonstrating the decision-making process and logical foundations, enhancing engineers' understanding and trust in the model. This provides more reliable overflow monitoring for drilling operations, further reducing the risk of overflow accidents and ensuring wellbore safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Flow chart of the method of the present invention.

[0057] Figure 2 Flowchart for scoring the objective function for the overflow initiation center region.

[0058] Figure 3 This is the mean change curve of hook height, vertical pressure and pump stroke in the data set.

[0059] Figure 4 is the mean change curve of methane, ethane and inlet and outlet flow difference in the data set.

[0060] Figure 5 The flow chart for identifying overflow drilling process.

[0061] Figure 6 Overflow monitoring for dataset 2 with a window size of 380, a step size of 380, and a scoring function.

[0062] Figure 7 Overflow monitoring for dataset 7 with a window size of 380, a step size of 380, and a scoring function. DETAILED DESCRIPTION

[0063] In order to have a clearer understanding of the technical features, objectives and beneficial effects of the present invention, the technical solution of the present invention is now described in detail below, but it should not be understood as limiting the scope of implementation of the present invention.

[0064] The present invention provides a method for early identification of drilling overflow based on deep reinforcement learning, the method comprising the following steps (eg Figure 1 shown):

[0065] Step 1: Data collection and preprocessing;

[0066] Using historical drilling data from a well, the collected data was cleaned, invalid data points were replaced with zero, and incomplete features were removed. A time window method was used to construct samples, with a time series matrix constructed using 380 time slices and a sliding step size of 1. Six features were selected: the difference between inlet and outlet flow rates, standing pressure (standpipe pressure), methane, ethane, pump stroke, and hook height. The mean, slope, and variance of these features within the time window were calculated. The change in the statistical value within adjacent windows was determined to exceed a threshold to determine whether an overflow occurred and record the label. Figure 3 and Figure 4 The mean change curves of feature quantities of some data sets are shown.

[0067] Step 2: Construction and optimization of the drilling overflow early identification model based on the proximal policy optimization (PPO) algorithm;

[0068] The core of this model lies in accurately identifying sudden changes in feature quantities during the drilling process. Change points are identified by comparing the mean, slope, and variance of the feature quantities within the preceding and following time windows. Due to the presence of vibration noise in field-collected drilling data, an appropriate threshold must be set to identify whether a feature quantity change represents a change point. The threshold setting must balance capturing early warning signs with avoiding false positives. The choice of time window and step size also influence algorithm performance. A large window can lead to delayed response, while a small window can be sensitive to noise. The step size determines the sampling density, affecting the refinement of feature extraction. To optimize these parameters (threshold, time window size, and step size), a parameter optimization algorithm (PPO) is employed. The PPO algorithm incorporates a clipping probability ratio strategy update method, which reduces performance instability during training and improves learning efficiency. This method determines the optimal threshold, time window size, and step size, enabling efficient identification of change points in drilling feature quantities and enhancing the accuracy and reliability of overflow monitoring.

[0069] The specific steps are as follows:

[0070] Step 21: Parameter initialization;

[0071] Setting hyperparameter cutoff factors , strategy update sub-iteration number M The sum value function updates the number of sub-iterations B ; Initialize strategy parameters θ and the initial value function parameters .

[0072] Step 22: Main iteration loop;

[0073] Starting from the first iteration (i.e. ), each iteration performs operations in the following order:

[0074] Trajectory Collection and Reward Calculation: Executing the Current Policy in a Drilling Environment πθk , collect trajectory dataset D k ={ τ i}, and calculate the cumulative reward for each time step G t ,in τ i The collected trajectory data.

[0075] Advantage function estimation: based on the current value function , use advantage estimation methods (such as GAE) to calculate the advantage value of each state-action pair .

[0076] Step 23: Update strategy parameters;

[0077] Entering the strategy optimization phase, the strategy parameters are updated through the following sub-iteration process:

[0078] right Execute in sequence:

[0079] Calculate the probability ratio of the new and old strategies:

[0080] ;

[0081] Where, is the probability ratio of the new and old strategies; Strategies before and after the update; This is the strategy before the update; θ old are the strategy parameters before updating; is the updated strategy parameter; is the advantage value of the state-action pair; is the time step t The state of the environment.

[0082] The Adam optimizer is used to maximize the truncated objective function of PPO through stochastic gradient ascent:

[0083] ;

[0084] Where, For the k +Strategy network parameters after 1 strategy optimization; is the number of iterations; is the trajectory point; is the trajectory set; is the time step; is the advantage value of the state-action pair; is the probability ratio of the new and old strategies; is the clipping function; is the hyperparameter cutoff factor;

[0085] Step 24: Value function learning;

[0086] After completing the policy parameter update, the value function optimization phase begins, where the parameters are updated through the following sub-iterations:

[0087] right Execute in sequence:

[0088] Use gradient descent to minimize the mean square error of the value function:

[0089] ;

[0090] Where, For the first k +Value network parameters after 1 round of strategy optimization; For the value network state valuation; is the cumulative reward for each time step;

[0091] Through continuous updates, the value function is made to fit the cumulative reward more accurately.

[0092] Step 25: Iterate the loop;

[0093] Repeat steps 22 to 24 until the algorithm converges or reaches the preset termination condition, thereby obtaining the optimized model.

[0094] Reward function construction: Construct a scoring function for accurately controlling the position of the identification point, and use it as the reward function for the PPO algorithm to determine the center area of ​​the overflow starting point. Set a "sweet spot" (the predicted overflow point is within the time window before and after the actual overflow point). Based on the needs of early identification and warning of overflow, give the highest score (such as adding 1000 points) to the marked points in the sweet spot so that they receive priority attention; give negative numbers (such as deducting 50 points) to the marked points outside the sweet spot but in normal state before and after the overflow occurs (marked points outside the sweet spot are non-overflow points) to reduce false alarms; give positive but relatively low scores (such as adding 100 points) to the marked points where overflow actually occurs (marked points outside the sweet spot are real overflow points) to encourage the algorithm to concentrate the identification points near the overflow starting point. Define the structured action space:

[0095] ;

[0096] Where, is the action space, that is, the set of all possible actions; is the action vector, which is an 18-dimensional real vector; For each dimension of the action vector; is the set of real numbers, is an 18-dimensional real number set. Each dimension of the action vector a is limited to a very small range, ensuring that subtle changes in the action can accurately control the position of the recognition point, allowing the PPO algorithm to optimize the action selection strategy. The scoring function formula is as follows:

[0097] ;

[0098] ;

[0099] Where, is the scoring function; Represents the number of marking points in the sweet spot; The number of marked points outside the sweet zone that are real overflow points; Represents the number of non-overflow points marked outside the sweet zone; Indicates that the conditions in square brackets are met number; is the coordinate of the real overflow point; The model predicts i The position of an overflow point in the time series.

[0100] The score is determined by calculating the number of marked points in the sweet zone, the number of marked points outside the sweet zone that are real overflow points, and the number of marked points outside the sweet zone that are non-overflow points. Figure 2 The working mechanism of this function is intuitively demonstrated.

[0101] Step 3: Early identification of drilling overflow;

[0102] During drilling operations, drilling data is collected in real time. After preprocessing, the data is input into a drilling flood early detection model based on the Proximal Policy Optimization (PPO) algorithm. Using parameters optimized by the PPO algorithm, the model analyzes changes in characteristic quantities such as inlet and outlet flow differentials, vertical pressure, methane, ethane, pump stroke, and hook height. By identifying the flooding drilling process, the model determines whether there are signs of flooding. If signs of flooding are detected, an early warning is issued based on the strategy determined by the reward function, prompting personnel to take appropriate measures to prevent blowouts and ensure drilling safety.

[0103] The process flow for identifying overflow drilling is as follows: Figure 5 As shown, during data monitoring, first check whether the inlet and outlet flow difference or vertical pressure increases. If not, the current flow change may be a normal fluctuation. If it increases, check whether the methane or ethane content increases. If the methane or ethane content increases, it is judged that there are signs of overflow. If not, check whether the pump stroke increases. If the pump stroke increases, the current flow change may be a normal fluctuation. If not, check whether the hook height decreases. If the hook height decreases, the current flow change may be a normal fluctuation. If not, it is judged that there are signs of overflow.

[0104] The specific implementation of the present invention is described in detail below with reference to specific embodiments.

[0105] Example 1: Overflow monitoring of data set 2;

[0106] Dataset 2, a collection of historical drilling data from a well from October 10 to November 10, 2015, was used. This data includes data from the 60 minutes before and during the overflow, with a sampling rate slightly below 1 Hz (50 samples per minute). A time windowing approach was used to construct a time series matrix with 380 time slices and a sliding step size of 380. Six characteristic parameters were selected: the difference between inlet and outlet flow rates, standpipe pressure, methane, ethane, pump stroke, and hook height. A drilling overflow early detection model based on the Proximal Policy Optimization (PPO) algorithm was employed, combined with a reward function for monitoring and analysis. Based on the parameters optimized by the PPO algorithm, the model analyzed the changes in the six characteristic parameters and determined whether there were signs of overflow by identifying the overflow drilling process. If there were signs of overflow, an early warning was issued based on the strategy determined by the reward function.

[0107] from Figure 6 As can be seen in the figure, the vertical dashed line represents the actual initial overflow point, the blue line shows the dynamic change in the inlet and outlet flow difference over time, and the red dot represents the overflow point predicted by the model. The model accurately predicts the time of overflow events, and the red predicted point is close to the vertical dashed line, demonstrating the high accuracy of the model's predictions. Furthermore, the changes in the blue line demonstrate the model's sensitivity to flow changes and its rapid response. Furthermore, the red dot shows that the model has the ability to provide early warning before an actual overflow occurs, with a low false alarm rate, strongly demonstrating the model's effectiveness in overflow monitoring.

[0108] Example 2: Overflow monitoring of data set 7;

[0109] This embodiment uses data set 7 of real drilling history data in the same time period as that of embodiment 1. The data processing method, sample construction method, and monitoring model and function used are consistent with those of embodiment 1.

[0110] like Figure 7 As shown in the figure, the model performs well under the optimal time window size, step size, and scoring function. The actual initial overflow point is close to the red predicted point, indicating that the model can accurately predict the time of overflow. The changes in the inlet and outlet flow difference curves reflect the model's keen monitoring of flow changes. The predicted points not only accurately indicate the overflow time but also provide early warning with a low false alarm rate, further verifying the reliability and effectiveness of the model in practical applications.

[0111] Example 3: Performance evaluation;

[0112] This algorithm was compared with a support vector machine (SVM) algorithm, a random forest (RF) algorithm using a decision tree ensemble, and a traditional back-propagation neural network (BP-NN) algorithm. The experiments used the same dataset and applied the same preprocessing steps to all algorithms. Algorithm performance was comprehensively evaluated using metrics such as accuracy, the closest distance to the initial overflow point, precision, recall, average distance to the initial overflow point, and the number of overflow points monitored within the time window preceding the initial overflow point. These metrics are closely aligned with actual industrial needs and can provide guidance for algorithm improvement and practical application.

[0113] (1) Accuracy (ACC) measures the ability of the algorithm to correctly identify overflow events, including the ratio of the total number of correctly identified positive and negative examples to the total number of test samples. The formula is as follows:

[0114] ;

[0115] in, (True Positives) is the number of samples correctly predicted to be positive, (True Negatives) is the number of samples correctly predicted to be negative, FP (False Positives) is the number of samples that are incorrectly predicted to be positive, FN (FalseNegatives) is the number of samples that are incorrectly predicted as negative.

[0116] (2) The nearest distance to the initial overflow point (ND) focuses on the distance between the overflow point identified by the algorithm and the actual overflow starting point. The closer the distance, the higher the positioning accuracy of the algorithm. The formula is as follows:

[0117] ;

[0118] in, It is i The location of the predicted overflow point, is the actual overflow starting point.

[0119] (3) Precision (P) refers to the proportion of samples predicted to be positive that are actually positive, reflecting the accuracy of the algorithm. The formula is as follows:

[0120] ;

[0121] (4) Recall (R), also known as the true positive rate, refers to the proportion of samples that are correctly predicted to be positive among all samples that are actually positive. It reflects the completeness of the algorithm. The formula is as follows:

[0122] ;

[0123] (5) Average Distance (AD) from the initial overflow point provides a statistical perspective. It evaluates the overall positioning ability of the algorithm by calculating the average value of the distance between all predicted overflow points and the actual overflow starting point. The formula is as follows:

[0124] ;

[0125] in, N is the total number of predicted overflow points, It is i The location of the predicted overflow point, is the actual overflow starting point.

[0126] (6) Number of overflow points monitored in the time window before the initial overflow point (Counts): This criterion evaluates the number of overflow sign points that the algorithm can detect within the set time window before the overflow actually occurs. This indicator helps measure the algorithm's ability to warn of impending overflow events. The formula is as follows:

[0127] ;

[0128] in, is the actual overflow starting point, W is the length of the time window, For the i The locations of the predicted overflow points, N is the number of all overflow points predicted by the model.

[0129] The comparison results are shown in Table 1:

[0130] Table 1 Comparison of this algorithm with other algorithms

[0131]

[0132] As shown in Table 1, this algorithm demonstrates superiority across multiple evaluation metrics on the test set. Its accuracy reached 0.79, demonstrating a higher correct identification rate and lower false alarm rate compared to other algorithms. Its processing speed was faster, enabling a more swift response to overflow events. It also performed better in metrics such as the closest distance to the initial overflow point (only 5) and the average distance to the initial overflow point (only 193), indicating higher positioning accuracy. Taking all these metrics into consideration, this algorithm demonstrates superior performance in overflow monitoring tasks, with increased reliability and effectiveness, providing a superior solution for drilling overflow monitoring.

[0133] The above are only preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, several variations and improvements can be made without departing from the concept of the present invention. These should also be regarded as the scope of protection of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A method for early identification of drilling overflow based on deep reinforcement learning, characterized by: The following steps are involved: Step 1: Data collection and preprocessing; The collected data was cleaned using real historical drilling data. A time window method was used to construct samples. Six characteristic quantities were selected: the difference between the outlet and inlet flow rates, vertical pressure, methane, ethane, pump stroke, and hook height. The mean, slope, and variance of the characteristic quantities within the time window were calculated. By determining whether the change in the statistical quantity of adjacent windows exceeded the threshold, it was determined whether it was an overflow and the label was recorded. Step 2: Construction and optimization of the drilling overflow early identification model based on the proximal strategy optimization algorithm, including parameter initialization, main iteration loop, strategy parameter update, value function learning, and iteration loop steps; Step 3: Early identification of drilling overflow; After preprocessing, real-time data is fed into a drilling overflow early detection model based on a proximal strategy optimization algorithm. The model uses parameters optimized by the proximal strategy optimization algorithm to analyze changes in inlet and outlet flow differentials, vertical pressure, methane, ethane, pump stroke, and hook height. It then identifies overflow drilling process flow and determines whether there are signs of overflow. If overflow signs are detected, an early warning is issued based on the strategy determined by the reward function. The specific process of step 2 is as follows: Step 21: Parameter initialization; Setting hyperparameter cutoff factors , strategy update sub-iteration number M and value function update sub-iteration number B; initialize strategy parameters θ and initial value function parameters ; Step 22: Main iteration loop; Starting from the first iteration, , each iteration performs operations in the following order: Trajectory collection and reward calculation: Execute the current policy π in the drilling environment θk , collect trajectory dataset D k ={τ i }, and calculate the cumulative reward G for each time step t , where τ i is the collected trajectory data; Advantage function estimation: based on the current value function V ϕk , use the advantage estimation method to calculate the advantage value A of each state-action pair t ; Step 23: Update strategy parameters; Entering the strategy optimization phase, the strategy parameters are updated through the following sub-iteration process: right Execute in sequence: Calculate the probability ratio of the new and old strategies: ； Where, is the probability ratio of the new and old strategies; For the updated strategy; is the strategy before updating; θ old are the strategy parameters before updating; is the updated strategy parameter; is the advantage value of the state-action pair; is the environmental state at time step t; The Adam optimizer is used to maximize the truncated objective function of PPO through stochastic gradient ascent: ； Where, The policy network parameters after the k+1th policy optimization; is the number of iterations; is the trajectory point; is the trajectory set; is the time step; is the advantage value of the state-action pair; is the probability ratio of the new and old strategies; is the clipping function; is the hyperparameter cutoff factor; Step 24: Value function learning; After completing the policy parameter update, the value function optimization phase begins, where the parameters are updated through the following sub-iterations: right Execute in sequence: Use gradient descent to minimize the mean square error of the value function: ； Where, is the value network parameter after the k+1th strategy optimization; For the value network state valuation; is the cumulative reward for each time step; Step 25: Iterate the loop; Repeat steps 22 to 24 until the algorithm converges or reaches the preset termination condition, and obtain the optimized model; Step 2 also includes the Reward function construction step, the specific process is as follows: Set a "sweet spot" and give the highest score to the points within the sweet spot. Give negative scores to the points outside the sweet spot but in a normal state before and after the overflow occurs. Give positive scores to points where the overflow actually occurs, which are much lower than the scores of the points within the sweet spot. Define the structured action space: ； Where, is the action space, that is, the set of all possible actions; is the action vector; For each dimension of the action vector; is the set of real numbers, is an 18-dimensional real number set; The scoring function formula is as follows: ； ； Where, is the scoring function; Represents the number of marking points in the sweet spot; The number of marked points outside the sweet zone that are real overflow points; Represents the number of non-overflow points marked outside the sweet zone; Indicates that the conditions in square brackets are met number; is the coordinate of the real overflow point; is the position of the i-th overflow point in the time series predicted by the model; In step 3, the specific process of identifying overflow drilling is as follows: First, check whether the inlet and outlet flow difference or vertical pressure increases. If not, the current flow change may be a normal fluctuation. If it increases, check whether the methane or ethane content increases. If the methane or ethane content increases, it is judged that there are signs of overflow. If not, check whether the pump stroke increases. If the pump stroke increases, the current flow change may be a normal fluctuation. If not, check whether the hook height decreases. If the hook height decreases, the current flow change may be a normal fluctuation. If not, it is judged that there are signs of overflow.

2. The method for early identification of drilling overflow based on deep reinforcement learning according to claim 1, characterized in that: In step 1, during data cleaning, invalid data points are replaced with zero and features with incomplete information are removed; When the time window method is used to construct samples, the time series matrix is ​​constructed in units of 380 time slices, and the sliding step size is 1.

Images