Drilling process leakage position prediction method and system based on deep reinforcement learning

Abstract

The invention discloses a drilling process leakage position prediction method and system based on deep reinforcement learning, and the method comprises the steps: carrying out the data preprocessing of drilled data, and obtaining the preprocessing data; constructing a drilling process simulation environment based on the preprocessed data; training the drilling process leakage position prediction model based on the drilling process simulation environment to obtain an optimal drilling process leakage position prediction model; in the training process, continuous interaction is carried out through an intelligent agent and a drilling process simulation environment, a Q value output by a depth Q network is obtained, the Q value is updated through a time difference method, an action strategy is selected based on the updated Q value, and the action strategy is optimized through a reward function; in the drilling process, the intelligent body of the optimal drilling process leakage position prediction model continuously interacts with the environment of the drilling process, and a corresponding action strategy is executed, so that the problem that the drilling process leakage position cannot be intelligently and accurately predicted in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for predicting leakage positions in a drilling process based on deep reinforcement learning. Background Art

[0002] At present, in the field of oil drilling engineering, the problem of well leakage during drilling has always been a major challenge that plagues the development of the industry. Traditional drilling leakage location prediction mainly relies on experience judgment and simple data statistical analysis. Experience judgment is highly subjective and is limited by personal knowledge and past experience, making it difficult to accurately respond to complex and changeable formation conditions. Simple data statistical methods can only process surface data associations, and are insufficient in mining deep, nonlinear drilling data relationships.

[0003] With the increase in drilling depth and complex formation exploration, the defects of traditional methods such as low prediction accuracy and high false alarm rate have become increasingly prominent. This has led to frequent well leakage accidents, which not only caused a large amount of drilling fluid waste and greatly increased drilling costs, but also caused drilling delays due to the treatment of well leakage. At the same time, well leakage may cause formation pressure imbalance, which in turn may induce a series of serious underground accidents such as formation collapse and drill bit sticking, seriously threatening the safety and efficiency of drilling operations, and causing huge economic losses and technical difficulties for oil production companies.

[0004] Therefore, there is an urgent need for an innovative, high-precision method for predicting the location of leakage during drilling that can adapt to complex working conditions to fill the gap in existing technology. Summary of the invention

[0005] In view of this, the present invention proposes a method and system for predicting the location of leakage in the drilling process based on deep reinforcement learning, which can provide accurate leakage location prediction results before drilling construction, so as to successfully predict the location of leakage or successfully avoid leakage. The present invention provides the following technical solutions: A method for predicting the location of leakage in a drilling process based on deep reinforcement learning, comprising: Collecting drilled data, and performing data preprocessing on the drilled data to obtain preprocessed data; Based on deep reinforcement learning, a drilling process leakage location prediction model with a deep Q network as an intelligent agent is constructed, and a drilling process simulation environment is constructed based on the preprocessed data; Training the drilling process leakage position prediction model based on the drilling process simulation environment to obtain an optimal drilling process leakage position prediction model; During the training process, the intelligent agent continuously interacts with the drilling process simulation environment to obtain the Q value output by the deep Q network, the Q value is updated by the time difference method, an action strategy is selected based on the updated Q value, and the action strategy is optimized by the reward function; During the drilling process, the intelligent agent of the optimal drilling process leakage location prediction model continuously interacts with the drilling process environment and executes the corresponding action strategy to successfully predict the leakage location or successfully avoid leakage.

[0006] On the basis of the above technical solution, the present invention can also be improved as follows: Furthermore, the data preprocessing includes data cleaning, filling missing values, removing outliers and data normalization.

[0007] Further, the updating of the Q value by a time difference method includes: The Q value is calculated by formula (1); Formula (1);

[0008] In the formula, The state at time t is Take action when The Q value is the expected cumulative reward for taking a specific action in a specific state. is the learning rate, is the immediate reward obtained at time t, is the discount factor, is the state at time t+1, For the status The Q value corresponding to the action with the largest Q value among all possible actions is taken.

[0009] Further, the updating of the Q value by a time difference method includes: Calculate the target value at time step t using formula (2); Formula (2); In the formula, is the target value at time step t, is the immediate reward obtained at time t, is the discount factor, The maximum Q value is taken for all possible actions in the state of time step t+1.

[0010] Further, the updating of the Q value by the time difference method further includes: The loss function is calculated by formula (3); Formula (3); In the formula, is the loss function, For mathematical expectations, is the target value at time step t, is the Q function, which is used to estimate the state Take action Expected cumulative rewards; By minimizing the predicted Q value and the target value The difference between them is used to update the parameters of the deep Q network.

[0011] Further, optimizing the action strategy through a reward function includes: It is determined whether the intelligent agent successfully predicts the location of the well leakage or successfully avoids the well leakage. If the intelligent agent successfully predicts the location of the well leakage or successfully avoids the well leakage, a positive reward is given; if the intelligent agent fails to successfully predict the location of the well leakage or a well leakage occurs, a negative reward is given.

[0012] Furthermore, the method for predicting the location of leakage in the drilling process based on deep reinforcement learning also includes: The multi-stage data of the drilling process is stored through the experience replay mechanism. During training, a small batch of data is randomly sampled from the experience replay mechanism for training the leakage position prediction model of the drilling process, thereby breaking the temporal correlation between the training samples.

[0013] A drilling process leakage position prediction system based on deep reinforcement learning, comprising: A preprocessing module, used for collecting the drilled data, performing data preprocessing on the drilled data, and obtaining preprocessed data; A construction module is used to construct a drilling process leakage location prediction model with a deep Q network as an intelligent agent based on deep reinforcement learning, and to construct a drilling process simulation environment based on the preprocessed data; A training module, used for training the drilling process leakage position prediction model based on the drilling process simulation environment to obtain an optimal drilling process leakage position prediction model; During the training process, the intelligent agent continuously interacts with the drilling process simulation environment to obtain the Q value output by the deep Q network, the Q value is updated by the time difference method, an action strategy is selected based on the updated Q value, and the action strategy is optimized by the reward function; The prediction module is used to continuously interact with the environment of the drilling process through the intelligent agent of the optimal drilling process leakage position prediction model during the drilling process, and execute the corresponding action strategy to successfully predict the leakage position or successfully avoid leakage.

[0014] An electronic device comprises a memory, a processor and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the method when executing the computer program.

[0015] A non-transitory computer-readable storage medium stores a computer program, which implements the steps of the method when executed by a processor.

[0016] The present invention has the following advantages: According to the technical solution of the present invention, during the drilling process, the intelligent agent interacts with the environment in real time and adjusts drilling parameters such as drilling fluid density, pump pressure, etc. according to the action strategy. This enables timely response to changes in working conditions that may cause well leakage, and kills the risk of well leakage in the bud, avoiding a series of serious consequences caused by well leakage, such as drilling fluid loss, formation collapse, and drill bit sticking, etc., ensuring the smooth progress of drilling operations, reducing non-productive time, and improving drilling efficiency.

[0017] This method for predicting the location of leakage during drilling based on deep reinforcement learning provides scientific and reasonable decision-making suggestions for drilling engineers based on the updated Q value and action strategy optimized by the reward function. Instead of relying solely on empirical judgment, the best drilling operation plan is selected based on the intelligent analysis of the model. For example, when faced with changes in formation pressure, the model can quickly analyze and determine the appropriate adjustment range of the drilling fluid density, making decisions more accurate and timely, and improving the overall intelligence level of drilling operations. By optimizing the action strategy and rationally allocating resources, unnecessary waste of resources can be avoided, such as increased energy consumption and time costs caused by overly conservative drilling parameter settings, thereby reducing the total drilling cost and improving economic benefits.

[0018] This method for predicting the location of leakage during drilling based on deep reinforcement learning builds a simulation environment for training based on rich drilled data, and can learn the laws of leakage under different formation characteristics, so as to adapt to various complex formation structures and geological conditions in actual drilling. Whether it is soft formation, hard formation or fractured formation, it can effectively play the role of predicting and preventing leakage, enhancing the adaptability and versatility of drilling technology in different geological regions.

[0019] In summary, this method of predicting leakage locations during drilling based on deep reinforcement learning has brought significant technical improvements and economic benefits to oil drilling projects in terms of accurate prediction, intelligent decision-making, adaptability to complex environments, and ensuring operational stability. It is expected to promote the entire industry to develop in a more efficient, safe, and intelligent direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] For the purpose of illustration and not limitation, the present invention is now described in conjunction with the embodiments of the present invention and the accompanying drawings, in which: Figure 1 It is a flow chart of a method for predicting the location of leakage in a drilling process based on deep reinforcement learning in an embodiment of the present invention; Figure 2 It is a schematic diagram of the main components of the drilling process leakage position prediction system based on deep reinforcement learning in an embodiment of the present invention; Figure 3 The probability distribution of the deep Q network prediction results of the present invention; Figure 4 It is the drilling fluid density curve of the logging while drilling of the present invention; Figure 5 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solution in the implementation of the present invention will be clearly and completely described below in conjunction with the drawings in the implementation of the present invention. Obviously, the described implementation is only a part of the implementation of the present invention, not all of the implementations. Based on the implementation of the present invention, all other implementations obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first", "second", etc. in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present invention described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0024] Figure 1 FIG. 1 is a flow chart of a method for predicting a leakage position in a drilling process based on deep reinforcement learning in an embodiment of the present invention. Figure 1 As shown, a method for predicting leakage positions in a drilling process based on deep reinforcement learning provided by an embodiment of the present invention includes the following steps S101 to S105.

[0025] S101, collecting drilled data, and performing data preprocessing on the drilled data to obtain preprocessed data.

[0026] Specifically: the drilled data includes well location information, wellbore trajectory, logging while drilling data, depth measurement corresponding to leakage during drilling, leakage rate and leakage amount, leakage layer, drilling fluid density and viscosity, total amount of mud pool and other data.

[0027] The data preprocessing includes data cleaning, filling missing values, removing outliers and data normalization to obtain a standard data set.

[0028] S102, based on deep reinforcement learning, a drilling process leakage location prediction model with a deep Q network as an intelligent agent is constructed, and a drilling process simulation environment is constructed based on preprocessed data.

[0029] Specifically, the overall architecture of deep reinforcement learning (DRL) covers three categories: basic elements, main elements and core elements. Among them, the basic elements include players (Agent), environment (Environment) and goals (Goal); the main elements include state (State), action (Action) and reward (Reward); the core elements include strategy and value.

[0030] Agent: Control strategy neural network. The present invention adopts deep Q network (DQN) to represent the decision-making subject. Through the interaction between the intelligent agent and the drilling process environment, it learns how to predict the leakage location.

[0031] Environment: drilling system or geological environment; the environment of the present invention includes physical conditions, geological structures, drilling parameters (such as drilling pressure, rotation speed, drilling fluid flow, etc.) downhole, as well as historical drilling data and downhole sensor data; the environment responds to the actions of the intelligent agent (such as adjusting drilling parameters) and provides feedback (such as leakage risk).

[0032] Goal: Accurately predict the location of lost circulation; let the agent learn to predict the exact location of lost circulation during drilling, and avoid or reduce the impact of lost circulation in actual operation, so as to improve drilling efficiency and safety.

[0033] Action: Adjust drilling operation parameters; adjust various parameters in drilling operations such as drilling fluid density, drilling pressure, rotation speed, wellbore trajectory, etc.; these actions help the agent make predictions by affecting the pressure and geological interactions during the drilling process.

[0034] Reward: The accuracy of the prediction. The agent’s performance is measured through a reward function. If the agent successfully predicts the location of a leak (or successfully avoids a leak), a positive reward is given. If the prediction is inaccurate or a leak occurs, a negative reward is given. The reward function is defined based on time cost.

[0035] S103, training a drilling process leakage position prediction model based on a drilling process simulation environment to obtain an optimal drilling process leakage position prediction model.

[0036] Specifically, in a drilling process simulation environment, the preprocessed data is input into an initialized drilling process leakage location prediction model; the intelligent agent in the drilling process leakage location prediction model selects an action according to the current input drilling state (represented by various parameters); this action may be a hypothetical adjustment to the current drilling parameters (for example, adjusting the drilling fluid density or pump pressure), or a preliminary prediction of the possibility of a well leakage location.

[0037] The corresponding reward value is calculated based on the agent's actions and the preset physical rules (based on physical phenomena and experience in the actual drilling process); the design of the reward value should take into account multiple factors. For example, if the agent's actions can accurately predict the location of the well leakage or effectively avoid the deterioration of the well leakage situation, a higher positive reward will be given; if the prediction is wrong or leads to an increased risk of well leakage, a negative reward will be given.

[0038] The Q-value output by the deep Q-network is updated through a temporal difference method (such as TD-learning); specifically, the Q-value of the current state-action pair is adjusted based on the current reward obtained and the Q-value estimate of the next state; this enables the model to optimize the action strategy based on the long-term accumulated rewards.

[0039] Repeat the above process and let the agent perform a large number of interactive operations in the simulated environment, constantly select actions, obtain rewards and update Q values; in each iteration, randomly sample a batch of data from the experience replay library (storing the previous state-action-reward-next state quadruple) for training to break the temporal correlation of the data and improve the generalization ability of the model.

[0040] During the training process, the model is evaluated regularly. Using cross-validation and other techniques, the training data is divided into a training set and a validation set. The model's prediction accuracy, recall rate, F1-score and other evaluation indicators are calculated on the validation set to measure the performance of the drilling process leakage location prediction model in predicting the well leakage location.

[0041] The drilling process leakage location prediction model is optimized based on the evaluation results. If the drilling process leakage location prediction model is found to be overfitting (for example, it performs well on the training set but poorly on the validation set), measures can be taken such as increasing the training data and using regularization techniques to adjust the complexity of the model; if the model's prediction accuracy is low, methods such as adjusting the network architecture (such as increasing the number of hidden layers or neurons), optimizing the learning rate, or adjusting the reward function can be tried to improve the model performance.

[0042] Continue to train and optimize the drilling process leakage position prediction model until the performance of the drilling process leakage position prediction model on the validation set is stable and meets the preset requirements. When the evaluation index of the drilling process leakage position prediction model on the validation set is no longer significantly improved, the drilling process leakage position prediction model is considered to have converged, and the drilling process leakage position prediction model obtained at this time is the optimal drilling process leakage position prediction model.

[0043] The optimal drilling process leakage position prediction model is saved for subsequent leakage position prediction and leakage prevention operation guidance in the actual drilling process.

[0044] S104, during the training process, the intelligent agent continuously interacts with the drilling process simulation environment to obtain the Q value output by the deep Q network, the Q value is updated by the time difference method, the action strategy is selected based on the updated Q value, and the action strategy is optimized by the reward function.

[0045] Specifically, the agent learns the expected rewards of different state-action pairs by updating the Q-values.

[0046] The Q value is calculated by formula (1); Formula (1);

[0047] In the formula, The state at time t is Take action when The Q value is the expected cumulative reward for taking a specific action in a specific state. is the learning rate, is the immediate reward obtained at time t, is the discount factor, is the state at time t+1, For the status The Q value corresponding to the action with the largest Q value among all possible actions is taken.

[0048] The training goal of the deep Q network is to minimize the temporal difference (TD) error of the Q value.

[0049] The target value at time step t is calculated using formula (2).

[0050] Formula (2).

[0051] In the formula, is the target value at time step t, is the immediate reward obtained at time t, is the discount factor, is the state at time step t+1 The maximum Q value is taken for all possible actions.

[0052] The loss function is calculated using formula (3).

[0053] Formula (3).

[0054] In the formula, is the loss function, For mathematical expectations, is the target value at time step t, is the Q function, which is used to estimate the state Take action Expected cumulative reward.

[0055] By minimizing the predicted Q value and the target value The difference between them is used to update the parameters of the deep Q network.

[0056] Optimizing the action strategy by using a reward function includes: It is determined whether the agent successfully predicts the location of the well leakage or successfully avoids the well leakage. If the agent successfully predicts the location of the well leakage or successfully avoids the well leakage, a positive reward is given. If the agent fails to successfully predict the location of the well leakage or the well leakage occurs, a negative reward is given.

[0057] S105, during the drilling process, the intelligent agent of the optimal drilling process leakage position prediction model continuously interacts with the drilling process environment and executes the corresponding action strategy to successfully predict the leakage position or successfully avoid the leakage.

[0058] The method for predicting the location of leakage in the drilling process based on deep reinforcement learning also includes: The multi-stage data of the drilling process is stored through the experience replay mechanism. During training, a small batch of data is randomly sampled from the experience replay mechanism for training the leakage position prediction model of the drilling process, thereby breaking the temporal correlation between the training samples.

[0059] The core formula of experience replay is:

[0060] In the formula, is the state at time step t, is the action taken at time step t, is the reward obtained at time step t, is the state at time step t+1; In a deep Q-network, the agent interacts with the environment, and each interaction produces a quadruple. is stored in the experience replay buffer (i.e., set D); data is then randomly sampled from this buffer to train the neural network. By repeatedly sampling data from the set for training, the agent can learn better strategies to maximize the long-term cumulative reward.

[0061] The input data of the method for predicting the leakage position in the drilling process based on deep reinforcement learning can be subjected to correlation analysis, and the main factors affecting the prediction of the leakage position can be obtained based on the correlation analysis, thereby reducing the computational cost.

[0062] An embodiment of the present invention is:

[0063] Collect the drilled data of Block XX and train the prediction model of leakage position based on the drilling process; DJ7-4, a well drilled in Block XX, is considered a new well. The data of this well are used to verify the prediction accuracy of the drilling process leakage position prediction model. The probability distribution of the leakage position predicted by the drilling process leakage position prediction model and the depth measurement results are shown in the figure. Figure 3-Figure 4 shown.

[0064] Depend on Figure 3-Figure 4 It can be seen that the drilling process leakage location prediction model constructed using the deep reinforcement learning method can predict the probability distribution of the leakage location of the new well. The probability distribution of the leakage location of the new well is consistent with the drilling fluid density curve measured while drilling the well. The drilling fluid density can directly explain the leakage situation of the well.

[0065] The comparison results show that the leakage location prediction model based on deep reinforcement learning provided by the present invention can accurately predict the location of leakage in new wells, and can provide accurate data reference for formulating leakage prevention measures for new wells on site, which greatly reduces the additional cost caused by leakage in the drilling process and improves the drilling construction efficiency.

[0066] Figure 2 Schematic diagram of the main components of the drilling process leakage position prediction system based on deep reinforcement learning in the embodiment of the present invention; Figure 2 As shown, a drilling process leakage position prediction system 1 based on deep reinforcement learning provided by an embodiment of the present invention includes a preprocessing module 10, a construction module 20, a training module 30, and a prediction module 40: The preprocessing module 10 is used to collect the drilled data and perform data preprocessing on the drilled data to obtain preprocessed data; the data preprocessing includes data cleaning, filling missing values, removing outliers and data normalization; The construction module 20 is used to construct a drilling process leakage location prediction model with a deep Q network as an intelligent agent based on deep reinforcement learning, and to construct a drilling process simulation environment based on the above preprocessed data; The training module 30 is used to train the above-mentioned drilling process leakage position prediction model based on the above-mentioned drilling process simulation environment to obtain the optimal drilling process leakage position prediction model; during the training process, the intelligent agent continuously interacts with the drilling process simulation environment to obtain the Q value output by the deep Q network, update the Q value by the time difference method, select the action strategy based on the updated Q value, and optimize the action strategy by the reward function; calculate the Q value by formula (1); calculate the target value of time step t by formula (2); calculate the loss function by formula (3); minimize the predicted Q value and the target value The difference between the two is used to update the parameters of the deep Q network; it is determined whether the above-mentioned intelligent agent successfully predicts the well leakage location or successfully avoids the well leakage. If the intelligent agent successfully predicts the well leakage location or successfully avoids the well leakage, a positive reward is given; if the intelligent agent fails to successfully predict the well leakage location or a well leakage occurs, a negative reward is given; the multi-stage data of the drilling process is stored through the experience replay mechanism. During training, a small batch of data is randomly sampled from the experience replay mechanism for the training of the leakage location prediction model of the above-mentioned drilling process, breaking the temporal correlation between the training samples; The prediction module 40 is used to continuously interact with the environment of the drilling process through the intelligent agent of the optimal drilling process leakage position prediction model during the drilling process, and execute corresponding action strategies to successfully predict the leakage position or successfully avoid leakage.

[0067] Figure 5 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as Figure 5 As shown, the electronic device 50 includes: a processor 501 (processor), a memory 502 (memory) and a bus 503; The processor 501 and the memory 502 communicate with each other via the bus 503; The processor 501 is used to call the program instructions in the memory 502 to execute the methods provided by the above-mentioned method embodiments, for example, including: collecting drilled data, performing data preprocessing on the drilled data to obtain preprocessed data; constructing a drilling process leakage position prediction model with a deep Q network as an intelligent agent based on deep reinforcement learning, and constructing a drilling process simulation environment based on the preprocessed data; training the drilling process leakage position prediction model based on the drilling process simulation environment to obtain an optimal drilling process leakage position prediction model; during the training process, the intelligent agent continuously interacts with the drilling process simulation environment to obtain a Q value output by the deep Q network, the Q value is updated by a time difference method, an action strategy is selected based on the updated Q value, and the action strategy is optimized by a reward function; during the drilling process, the intelligent agent of the optimal drilling process leakage position prediction model continuously interacts with the drilling process environment to execute a corresponding action strategy to successfully predict the well leakage position or successfully avoid well leakage.

[0068] The present embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable a computer to execute the methods provided by the above-mentioned method embodiments, for example, including: collecting drilled data, performing data preprocessing on the drilled data to obtain preprocessed data; constructing a drilling process leakage position prediction model with a deep Q network as an intelligent agent based on deep reinforcement learning, and constructing a drilling process simulation environment based on the preprocessed data; training the drilling process leakage position prediction model based on the drilling process simulation environment to obtain an optimal drilling process leakage position prediction model; during the training process, the intelligent agent continuously interacts with the drilling process simulation environment to obtain a Q value output by the deep Q network, the Q value is updated by a time difference method, an action strategy is selected based on the updated Q value, and the action strategy is optimized by a reward function; during the drilling process, the intelligent agent of the optimal drilling process leakage position prediction model continuously interacts with the drilling process environment to execute a corresponding action strategy to successfully predict the well leakage position or successfully avoid well leakage.

[0069] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various storage media that can store program codes.

[0070] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for predicting the location of leakage in the drilling process based on deep reinforcement learning, characterized in that: include: Collecting drilled data, and performing data preprocessing on the drilled data to obtain preprocessed data; Based on deep reinforcement learning, a drilling process leakage location prediction model with a deep Q network as an intelligent agent is constructed, and a drilling process simulation environment is constructed based on the preprocessed data; Training the drilling process leakage position prediction model based on the drilling process simulation environment to obtain an optimal drilling process leakage position prediction model; During the training process, the intelligent agent continuously interacts with the drilling process simulation environment to obtain the Q value output by the deep Q network, the Q value is updated by the time difference method, an action strategy is selected based on the updated Q value, and the action strategy is optimized by the reward function; During the drilling process, the intelligent agent of the optimal drilling process leakage location prediction model continuously interacts with the drilling process environment and executes the corresponding action strategy to successfully predict the leakage location or successfully avoid leakage.

2. The method for predicting the location of leakage in the drilling process based on deep reinforcement learning according to claim 1, characterized in that: The data preprocessing includes data cleaning, filling missing values, removing outliers and data normalization.

3. The method for predicting the location of leakage in the drilling process based on deep reinforcement learning according to claim 1, characterized in that: The updating of the Q value by a time difference method comprises: The Q value is calculated by formula (1); Formula (1): In the formula, The state at time t is Take action when The Q value is the expected cumulative reward for taking a specific action in a specific state. is the learning rate, is the immediate reward obtained at time t, is the discount factor, is the state at time t+1, For the status The Q value corresponding to the action with the largest Q value among all possible actions is taken.

4. The method for predicting the location of leakage in the drilling process based on deep reinforcement learning according to claim 1, characterized in that: The updating of the Q value by a time difference method comprises: Calculate the target value at time step t using formula (2); Formula (2): In the formula, is the target value at time step t, is the immediate reward obtained at time t, is the discount factor, is the state at time step t+1 The maximum Q value is taken for all possible actions.

5. The method for predicting the location of leakage in the drilling process based on deep reinforcement learning according to claim 3, characterized in that: The updating of the Q value by the time difference method further includes: The loss function is calculated by formula (3); Formula (3): In the formula, is the loss function, For mathematical expectations, is the target value at time step t, is the Q function, which is used to estimate the state Take action Expected cumulative rewards; By minimizing the predicted Q value and the target value The difference between them is used to update the parameters of the deep Q network.

6. The method for predicting the location of leakage in the drilling process based on deep reinforcement learning according to claim 1, characterized in that: Optimizing the action strategy by using a reward function includes: It is determined whether the intelligent agent successfully predicts the location of the well leakage or successfully avoids the well leakage. If the intelligent agent successfully predicts the location of the well leakage or successfully avoids the well leakage, a positive reward is given; if the intelligent agent fails to successfully predict the location of the well leakage or a well leakage occurs, a negative reward is given.

7. The method for predicting the location of leakage in the drilling process based on deep reinforcement learning according to claim 1, characterized in that: The method for predicting the location of leakage in the drilling process based on deep reinforcement learning also includes: The multi-stage data of the drilling process is stored through the experience replay mechanism. During training, a small batch of data is randomly sampled from the experience replay mechanism for training the leakage position prediction model of the drilling process, thereby breaking the temporal correlation between the training samples.

8. A system for predicting the location of leakage in the drilling process based on deep reinforcement learning, characterized in that: include: A preprocessing module, used for collecting the drilled data, performing data preprocessing on the drilled data, and obtaining preprocessed data; A construction module is used to construct a drilling process leakage location prediction model with a deep Q network as an intelligent agent based on deep reinforcement learning, and to construct a drilling process simulation environment based on the preprocessed data; A training module, used for training the drilling process leakage position prediction model based on the drilling process simulation environment to obtain an optimal drilling process leakage position prediction model; During the training process, the intelligent agent continuously interacts with the drilling process simulation environment to obtain the Q value output by the deep Q network, the Q value is updated by the time difference method, an action strategy is selected based on the updated Q value, and the action strategy is optimized by the reward function; The prediction module is used to continuously interact with the environment of the drilling process through the intelligent agent of the optimal drilling process leakage position prediction model during the drilling process, and execute the corresponding action strategy to successfully predict the leakage position or successfully avoid leakage.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images