A Monitoring Method and System for Logging Instrument Sticking Based on Reinforcement Learning

Abstract

The present invention discloses a method and system for monitoring obstructions and jams in logging instruments based on reinforcement learning, and belongs to the technical field of petroleum logging engineering. After pre-processing the data, a training set and a test set are selected from a large amount of logging data. Then, a reinforcement learning simulation environment is built using the selected training set, and a reinforcement learning network structure is established according to the environment. The reinforcement learning network model is trained using the training set, and the reward and punishment strategy is modified according to the training curve and training score during the training process to further optimize the network model. The optimal model is used for testing. Reinforcement learning is used to improve the accuracy of judging obstructions and jams in the logging process, provide a better standard for judging the operating status of the instrument, reduce the errors that may be caused by manual judgment, improve the work efficiency of the staff, and reduce the cost loss in the logging construction process.

Description

Technical Field

[0001] The present invention belongs to the technical field of petroleum well logging engineering, and relates to an intelligent judgment and analysis of obstructions and jams encountered by an instrument during a logging acquisition process, and specifically to a method and system for monitoring obstructions and jams encountered by a logging instrument based on reinforcement learning. Background Art

[0002] As demand for oil continues to grow, oil well extraction and exploration technologies continue to advance, with continuous innovations in logging and other methods to aid in the discovery of oil resources. During the logging process, numerous sensors integrated into the instrument acquire well and formation information. However, due to complex downhole conditions, borehole diameter reduction, mud and rock interference, and high mud density, the lowering and raising of logging instruments inevitably encounter obstacles, typically manifesting as obstructions during lowering and jamming during raising.

[0003] Obstructions and stuck conditions during logging can disrupt normal operation, damage instruments, or cause them to fall into the wellbore, leading to project delays and increased costs. Currently, the diagnosis of obstructions and stuck conditions relies primarily on the judgment of logging instrument operators. This presents several challenges, including heavy reliance on operator experience, inconsistent judgment standards across operators, and high rates of missed and incorrect judgments, making them difficult to address. Summary of the Invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art in which obstruction and stuck faults in the logging process rely on manual judgment resulting in a high error rate, the purpose of the present invention is to provide a method and system for monitoring obstruction and stuck faults of logging instruments based on reinforcement learning.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for monitoring obstruction and stuck conditions of a logging instrument based on reinforcement learning, comprising the following steps:

[0007] Step 1) obtaining raw data from the operation of the logging instrument, and processing the raw data to obtain a data set;

[0008] Step 2) Create training and test sets based on the dataset;

[0009] Step 3) Building a reinforcement learning simulation environment based on Python, further establishing a reinforcement learning network model, and obtaining a state strategy table; the reinforcement learning network model includes an input layer, a hidden layer, and an output layer;

[0010] Step 4) training the reinforcement learning network model based on the training set;

[0011] Step 5) The trained reinforcement learning network model is tested based on the test set to evaluate the performance of the reinforcement learning network model, obtain the optimal reinforcement learning network model, and perform real-time monitoring of the obstruction and jam status based on the optimal reinforcement learning network model.

[0012] Preferably, in step 1), processing the raw data includes eliminating invalid values, filtering the measurement curve, and standardizing the data of different blocks.

[0013] Preferably, during the standardization process, the standardization formula used is:

[0014]

[0015] Among them, mean(X) represents the average value of the original tension curve data, std(X) represents the variance of the original tension curve data, and X i Represents the original tension curve, Z i Indicates the normalized tension curve value.

[0016] Preferably, in step 3), the reinforcement learning simulation environment built based on Python includes tension curve conditions, pulling actions, and resistance and jamming states;

[0017] The execution process of the reinforcement learning simulation environment is as follows: in the normal state, the lifting is rewarded, and in the normal state, the stopping is punished. In the state of encountering obstacles or jams, the lifting is punished, and in the state of encountering obstacles or jams, the stopping is rewarded.

[0018] The input layer is used to load data; the hidden layer is used to make the environment return the next environment state vector after performing a certain action, and to define the reward and penalty rules for the specific execution action; the output layer is the state strategy table.

[0019] Preferably, in step 3), the process of building the reinforcement learning network model is:

[0020] Determine the environmental state and individual action. When the individual takes the corresponding action in a certain state at time t, he will be rewarded at time t+1.

[0021] Establish an individual's strategy as a basis for the individual to choose actions;

[0022] The reward value function is established using pre-set reward and penalty rules, specifically:

[0023] v π (S)=E π (R t+1 +rR t+2 +r 2 R t+3 +...|S t =S)

[0024] Among them, v π (S) represents the reward value function; π represents the individual's strategy, which is used to select actions; R t+1 、R t+2 、R t+3 represents the rewards at multiple moments after time t; S t represents the state at time t; r represents the reward attenuation factor, and r takes a value between [0, 1];

[0025] Based on the reward value function, iterative optimization is performed to generate a state strategy table.

[0026] Preferably, in step 3), there are two hidden layers.

[0027] Preferably, the specific process of step 4) is:

[0028] Input the training set into the reinforcement learning network model, set the number of training rounds and model update frequency, update the state policy table, and draw a score curve of the score and loss. Select the corresponding number of training rounds based on the score curve so that the model can stably identify obstacles in the last 10 times and the score is greater than the set threshold. In this case, stop training; otherwise, continue training.

[0029] Preferably, the update formula of the state strategy table is:

[0030] newQ(s t+1 ,a t+1 )=Q(s t ,a t )+α*[R(s t ,a t )+γ*maxQ′(s′,a′)-Q(s t ,a t )]

[0031] Among them, Q(s t+1 ,a t+1 ) is the score value at the next moment, Q(s t ,a t ) is the score value at the current moment, α is the learning rate, R(s t ,a t ) is the reward or penalty at that moment, γ is the decay rate, and maxQ′(s′,a′) is the maximum expected score value at the next moment.

[0032] A logging tool obstruction and stuck monitoring system based on reinforcement learning, comprising:

[0033] A data acquisition unit, used to acquire raw data from the operation of the logging instrument;

[0034] The data processing unit interacts with the data acquisition unit and is used to process the original data to obtain a data set, and to create a training set and a test set based on the data set;

[0035] The model building unit interacts with the data processing unit to build a reinforcement learning simulation environment based on Python and further establish a reinforcement learning network model;

[0036] The model training and testing unit interacts with the model building unit to train the reinforcement learning network model based on the training set, and to test the trained reinforcement learning network model based on the test set to evaluate the performance of the reinforcement learning network model;

[0037] The detection unit interacts with the model training and testing unit to monitor the obstruction and stuck status of the logging instrument in real time based on the optimal reinforcement learning network model.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention discloses a method for monitoring obstructions and jams of logging instruments based on reinforcement learning. First, data is preprocessed. Then, training data and test data are selected from a large amount of logging data. A training set is used to train the network, and a test set is used to test the network performance. In order to make the trained network have better generalization performance, the division of the training set needs to conform to the distribution of the data. Then, a reinforcement learning simulation environment is built using the selected training set, and a reinforcement learning network structure is established according to the environment. The training set is used to train the reinforcement learning network model, and the reward and punishment strategy is modified based on the training curve and training score during the training process to further optimize the network model. Finally, the optimal model obtained in the previous step is used for testing. The reinforcement learning network model after training is tested based on the test set to evaluate the performance of the reinforcement learning network model. The learning ability of the network model is analyzed based on the generalization performance of the network and the actual effect of the test. Through reinforcement learning, the accuracy of judging obstructions and jams of the instrument during the logging process is improved, a better standard for judging the operating status of the instrument is provided, the errors that may be caused by manual judgment are reduced, the work efficiency of the staff is improved, and the cost loss during the logging construction process is reduced.

[0040] Furthermore, the logging data contains a large amount of environmental noise and errors caused by the testing process. The effects of noise and outliers are eliminated through operations such as logging data standardization and outlier removal.

[0041] Furthermore, the reinforcement learning environment includes three parts: environmental state, execution action, and reward and punishment. The three main components are combined to build a network structure, and the network model is used to learn the decision function. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1This is a flow chart of the method for judging obstacles and card encounters based on reinforcement learning of the present invention;

[0043] Figure 2 This is a schematic diagram of a curve showing a real obstruction or jamming situation handled in an example of the present invention;

[0044] Figure 3 Schematic diagram of the model structure and data flow of reinforcement learning of the present invention;

[0045] Figure 4 is a graph showing the accumulated scores during the training process of the present invention;

[0046] FIG5 is a diagram showing the judgment result of the present invention for the real curve, (a) is a normal state, and (b) is a state of encountering obstruction or jamming. DETAILED DESCRIPTION

[0047] The present invention is described in further detail below with reference to the accompanying drawings:

[0048] Example 1

[0049] A method for monitoring obstruction and stuck conditions of a logging instrument based on reinforcement learning, comprising the following steps:

[0050] Step 1) obtaining raw data from the operation of the logging instrument, and processing the raw data to obtain a data set;

[0051] Step 2) Create training and test sets based on the dataset.

[0052] Step 3) Build a reinforcement learning simulation environment and further establish a reinforcement learning network model. The reinforcement learning network model includes an input layer, a hidden layer, and an output layer. First, create a reinforcement learning network model with a network structure of 4 layers: input layer → 2 hidden layers → output layer. The number of network nodes in each layer is 2, 16, 32, and 2. The activation function of each hidden layer is set to the RELU activation function. Secondly, for the action prediction part of the model, random numbers are used to generate random actions in the first 1000 steps to increase the number of learning samples and improve learning accuracy. Finally, the model sets an experience replay memory pool to accumulate experience samples, and then randomly extracts a certain number of samples from the memory pool to train the network and update the Q table.

[0053] Step 4) training the reinforcement learning network model based on the training set;

[0054] Step 5) The trained reinforcement learning network model is tested based on the test set to evaluate the performance of the reinforcement learning network model, obtain an optimized reinforcement learning network model, and perform real-time monitoring of the obstruction and jam status based on the optimized reinforcement learning network model.

[0055] Example 2

[0056] Except for the following contents, the rest are the same as Example 1.

[0057] The processing of raw data includes invalid value elimination, filtering of measurement curves and standardization of data in different blocks. The standardization formula used in standardization is:

[0058]

[0059] Among them, mean(X) represents the average value of the original tension curve data, std(X) represents the variance of the original tension curve data, and X i Represents the original tension curve, Z i Indicates the normalized tension curve value.

[0060] The process of building a reinforcement learning model is as follows:

[0061] S301, determine the environmental state S, the state of the environment at time t S t It is a state in the state of its environment;

[0062] S302, determine individual action A, the action A taken by the individual at time t t It is a certain action in its action concentration;

[0063] S303, the reward R of the environment, the individual at time t is in state S t Action A t The corresponding reward R t+1 It will be obtained at time t+1.

[0064] S304, the individual's strategy π, represents the basis for the individual to take action, that is, the individual will choose an action based on the strategy π. The most common way to express the strategy is a conditional probability distribution π(A|S), that is, the probability of taking action A in state S. That is, π(A|S)=P(A t =A|S t =S). At this time, the action with a high probability is more likely to be chosen by the individual.

[0065] S305, the value of the individual after taking action when the strategy is π and the state is S, usually expressed as v π (S) indicates that this value is generally an expected function. Although the current action will give a delayed reward R t+1 However, it is not enough to just look at the delayed reward, because a high current delayed reward does not mean that the subsequent rewards at time t+1, t+2, etc. will also be high. Therefore, the value should take into account both the current delayed reward and the subsequent delayed rewards. Value function v π (S) can generally be expressed as the following formula. Different algorithms will have corresponding value function variants, but the idea is the same.

[0066] v π (S)=E π (R t+1 +rR t+2 +r 2 R t+3 +...|S t =S)

[0067] Among them, R t+1 、R t+2 、R t+3 represents the rewards at multiple moments after time t, S t represents the state at time t, and r is the reward decay factor, ranging from [0 to 1]. If it is 0, it is a greedy approach, where the value is determined solely by the current delayed reward. If it is 1, all subsequent state rewards are treated equally to the current reward. Most of the time, we choose a number between 0 and 1, where the current delayed reward is weighted more heavily than subsequent rewards.

[0068] S306, the state transition model of the environment, can be understood as a probabilistic state machine, which can be expressed as a probability model, that is, the probability of taking action A in state S and transitioning to the next state S′.

[0069] S307, the exploration rate λ, is used during the iterations of reinforcement learning training. While the action that maximizes the value of the current iteration is generally chosen, this can result in some good but previously unexecuted actions being missed. Therefore, when selecting the optimal action during training, there is a certain probability that λ will not select the action that maximizes the value of the current iteration, but instead select a random action.

[0070] Example 3

[0071] A method for monitoring obstructions and jams during logging based on reinforcement learning. The process is as follows: Figure 1 As shown, the following steps are included:

[0072] The first step: data preprocessing. The data in this embodiment are taken from the obstruction and jamming data of a certain oil field block. The data curve is as follows: Figure 2 First, the raw data was preprocessed to remove outliers, such as 0 and -99999, that were outside the reasonable range, ensuring that the real data used was within the normal range. Second, since there were significant differences in the values of the curves of each well, the training used uniform features and the data was standardized.

[0073] Step 2: Create training and test sets. The dataset obtained after the first step of preprocessing is randomly divided into training and test sets with a ratio of 0.8:0.2. The training set will be used to train the reinforcement learning model, and the test set will be used to verify and evaluate the model's performance.

[0074] Step 3: Build a reinforcement learning simulation environment. Based on the real-world logging environment and action characteristics, a simulation environment for reinforcement learning training was constructed. This included state feedback and reward and penalty feedback for executed actions. The reward and penalty rules are shown in Table 1. The reward and penalty rule settings include, but are not limited to, those described above. Reward and penalty parameter values can be flexibly set based on the input data.

[0075] Table 1. Reward and Punishment Rules

[0076]

[0077] Step 4: Build a reinforcement learning network model. First, since the number of states in this database is not large, a 4-layer network consisting of input layer → 2 hidden layers → output layer is created based on the characteristics. The number of nodes in each layer is 2, 16, 32, and 2, and the activation function of each hidden layer is set to RELU activation function. Secondly, set up an experience memory pool and store it in queue data format. The actions of the first 1000 steps of the training process are randomly generated to obtain more possible states. The model organization structure is as follows: Figure 3 shown.

[0078] Random actions are set using random numbers, and the formula is as follows:

[0079] μ<δ-step*ε

[0080] Among them, μ is the normal distribution between [0,1], δ is the upper threshold, and ε is the step size.

[0081] The experience replay memory pool stores samples (s, a, r, ns) generated at each previous step, where s is the state at the current moment, a is the action performed at the current moment, r is the reward or penalty returned by the environment, and ns is the state at the next moment returned by the environment. When the accumulated sample size reaches 1000, samples are randomly extracted from it for network training.

[0082] Step 5: Train the reinforcement learning model. Implement the environment and model instance built in steps 3 and 4, load the training set data set in step 2 into the environment, set the model update frequency to update the model every 100 steps, set the training to 50 rounds, save the model after training and plot the training score, and accumulate the score curve as shown below: Figure 4As shown in the figure, the network model's cumulative score is very low at the beginning of training. After a period of training, the network model has achieved a higher score, but the overall score is still low, and the model is stuck in a local optimum. Because the network incorporates a step that randomly selects actions to be executed, the network model breaks out of the local optimum after a period of training. After multiple iterations, the network model finally learns the optimal decision function, and the network model score stabilizes around 1000. Based on the score curve, select an appropriate number of training rounds so that the model consistently identifies obstacles in the last 10 times. Training is stopped when the score exceeds the set threshold.

[0083] The Q table update formula is:

[0084] newQ(s t+1 ,a t+1 )=Q(s t ,a t )+α*[R(s t ,a t )+γ*maxQ′(s′,a′)-Q(s t ,a t )]

[0085] Among them, the meaning and specific settings of each parameter are: Q(s t+1 ,a t+1 )The Q value at the next moment, Q(s t ,a t ) is the Q value at the current moment, α is the learning rate, R(s t ,a t ) is the reward or penalty at that moment, γ is the decay rate, and maxQ′(s′,a′) is the maximum expected Q value at the next moment.

[0086] Step 6: Testing and verification of model inference results. After the model training is completed, the reinforcement learning model is tested using the test training set in the second step to verify the stability of the model and evaluate the model performance. The schematic diagram of the judgment result of extracting an obstruction and jamming curve from the test set is shown in Figure 5. It can be seen from the figure that the dotted line simulates the position of the logging curve data acquisition at the current moment, and from left to right is the order of logging data acquisition. In Figure (a), it is currently in a normal state, and the network model determines that the current output action is to continue pulling. In Figure (b), it is currently encountering an obstruction and jamming state. The network model determines that it has encountered an obstruction and jamming, and the action output by the network model is to stop pulling. Combined with Figure 2 The model's judgment is accurate. From the results, we can see that: Figure 2 It can be seen from the figure that the selected part is the part that encounters resistance or jamming, while Figure 5 shows that the execution action of stopping pulling is output at the same position, which determines that the current position has encountered resistance or jamming. Combining the two figures verifies the judgment ability of the model.

[0087] Example 3

[0088] A logging instrument obstruction and stuck monitoring system based on reinforcement learning, characterized by comprising:

[0089] A data acquisition unit, used to acquire raw data from the operation of the logging instrument;

[0090] The data processing unit interacts with the data acquisition unit and is used to process the original data to obtain a data set, and to create a training set and a test set based on the data set;

[0091] The model building unit interacts with the data processing unit to build a reinforcement learning simulation environment and further establish a reinforcement learning network model;

[0092] The model training and testing unit interacts with the model building unit, trains the reinforcement learning network model based on the training set, tests the trained reinforcement learning network model based on the test set, and evaluates the performance of the reinforcement learning network model.

[0093] The monitoring unit interacts with the model training and testing unit to monitor the obstruction and stuck status of the logging instrument in real time based on the optimized reinforcement learning network model.

[0094] In summary, this invention, based on reinforcement learning, employs a DQN reinforcement learning model to monitor and identify instrument obstructions and stuck conditions during logging, reducing errors caused by manual judgment and improving the efficiency of logging personnel. This method, based on reinforcement learning, constructs a method for monitoring obstructions and stuck conditions in logging instruments. This innovative application of reinforcement learning enables real-time, full-process intelligent monitoring of possible obstructions and stuck conditions during logging. This method is stable, efficient, and highly accurate, reducing the burden on personnel and assisting them in improving the accuracy of identifying obstructions and stuck conditions.

[0095] The above content is only for explaining the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the claims of the present invention.

Claims

1. A method for monitoring obstruction and stuck conditions of a logging instrument based on reinforcement learning, characterized in that: The steps include: Step 1) obtaining the raw data of the logging instrument operation, processing the raw data to obtain a data set; Step 2) Create training and test sets based on the dataset; Step 3) Build a reinforcement learning simulation environment based on Python, further establish a reinforcement learning network model, and obtain a state strategy table; The reinforcement learning network model includes an input layer, a hidden layer, and an output layer; Step 4) Train the reinforcement learning network model based on the training set; Step 5) Test the trained reinforcement learning network model based on the test set to evaluate its performance, obtain the optimal reinforcement learning network model, and monitor the obstruction and jam status in real time based on the optimal reinforcement learning network model. In step 1), the processing of the original data includes the elimination of invalid values, filtering of the measurement curve, and standardization of the data of different blocks; the standardization formula used in the standardization process is: in, Represents the average value of the original tension curve data, represents the variance of the original tension curve data, represents the original tension curve, Indicates the value of the normalized tension curve; In step 3), the reinforcement learning simulation environment built based on Python includes tension curve conditions, pulling actions, and resistance and jamming states; The execution process of the reinforcement learning simulation environment is as follows: in the normal state, the lifting is rewarded, and in the normal state, the stopping is punished. In the state of encountering obstacles or jams, the lifting is punished, and in the state of encountering obstacles or jams, the stopping is rewarded. The input layer is used to load data; the hidden layer is used to make the environment return the next environment state vector after performing a certain action, and to define the reward and penalty rules for the specific execution action; the output layer is the state strategy table.

2. The method for monitoring obstruction and stuck of a logging instrument based on reinforcement learning according to claim 1, characterized in that: In step 3), the process of building the reinforcement learning network model is as follows: Determine the environmental state and individual action. When the individual takes the corresponding action in a certain state at time t, he will be rewarded at time t+1. Establish an individual's strategy as a basis for the individual to choose actions; The reward value function is established using pre-set reward and penalty rules, specifically: in, represents the reward value function; π represents the individual’s strategy, which is used to select actions; Represents the rewards at multiple moments after time t; represents the state at time t; represents the reward decay factor, and The value is between [0, 1]; Based on the reward value function, iterative optimization is performed to generate a state strategy table.

3. The method for monitoring obstruction and stuck of a logging instrument based on reinforcement learning according to claim 1, characterized in that: In step 3), there are two hidden layers.

4. The method for monitoring obstruction and stuck conditions of a logging instrument based on reinforcement learning according to claim 1, wherein: The specific process of step 4) is as follows: Input the training set into the reinforcement learning network model, set the number of training rounds and model update frequency, update the state policy table, and draw a score curve of the score and loss. Select the corresponding number of training rounds based on the score curve so that the model can stably identify obstacles in the last 10 times and the score is greater than the set threshold. In this case, stop training; otherwise, continue training.

5. The method for monitoring obstruction and stuck of a logging instrument based on reinforcement learning according to claim 4, characterized in that: The update formula of the state strategy table is: in, is the score value at the next moment, is the score value at the current moment, is the learning rate, For the reward or punishment at that moment, is the attenuation rate, is the maximum expected score value at the next moment.

Images