A method and system for discriminating multi-stage working conditions of offshore drilling based on reinforcement learning

Abstract

The application belongs to the technical field of working condition recognition, and discloses a marine drilling multi-stage working condition discrimination method and system based on reinforcement learning, which comprises the following steps: collecting a multi-sensor data sequence and extracting a multi-dimensional feature vector, inputting a pre-trained state probability prediction model, obtaining a state probability vector representing the probability of occurrence of each working condition state in the future T time, and sending the state probability vector to a worker; and monitoring the operation actions of the worker in the future T time, obtaining the state probability vector again after T time, calculating a reward value in combination with the operation actions of the worker, and continuously updating the model parameters through the reinforcement learning method with the reward value as a signal. The application converts the comprehensive analysis of multi-dimensional sensor data into an automatic calculation process of the model, realizes quantitative probability prediction of the drilling working condition, and continuously optimizes the model through online reinforcement learning, so that the prediction accuracy is continuously improved with the progress of the operation.

Description

Technical Field

[0001] This invention relates to the field of working condition recognition technology, and more specifically, to a method and system for multi-level working condition discrimination in marine drilling based on reinforcement learning. Background Technology

[0002] Offshore drilling operations are conducted continuously in complex geological environments, and various types of downhole anomalies may occur during the drilling process, including formation fluid intrusion, drilling fluid loss, drill string obstruction, and abnormal drill string vibration. If these anomalies are not identified and handled in a timely manner, they can range from affecting drilling efficiency to causing well control accidents and significant safety losses. Therefore, real-time monitoring and assessment of various downhole conditions during the drilling process are crucial for ensuring the safety of offshore drilling operations.

[0003] Currently, offshore drilling platforms are widely equipped with sensors to monitor key physical quantities such as wellbore pressure, inlet and outlet flow rates, drill string stress, rotational parameters, and vibration levels, enabling real-time acquisition of various measurement data reflecting the drilling status. Regarding downhole condition assessment, current practices primarily rely on manual judgment by operators based on the aforementioned measurement data and their professional experience. Operators observe changes in various measurement data, relying on their understanding of drilling engineering patterns to determine the presence and type of anomalies, and decide whether to take operational action accordingly. However, the evolution of various downhole anomalies during drilling involves complex, interconnected changes in multiple measurement data over time. Human experience struggles to systematically and comprehensively analyze multidimensional measurement data, and even more so to provide quantitative estimates of the probability of various conditions occurring when early anomaly characteristics are not yet significant. Operators typically provide qualitative, subjective judgments, with results varying from person to person, and this is particularly challenging when multiple anomalies occur concurrently. As offshore drilling moves towards deeper and ultra-deep water, formation conditions are becoming increasingly complex. The requirements for early identification of downhole conditions are constantly increasing, demanding multi-dimensional data analysis capabilities. Existing judgment methods that rely on human experience are no longer sufficient to meet the actual operational needs for quantitative and objective prediction of downhole conditions. Summary of the Invention

[0004] To overcome the aforementioned problems in the prior art, this invention proposes a method and system for determining multi-level working conditions in marine drilling based on reinforcement learning, which is used to solve the above problems.

[0005] This invention provides the following technical solution: A reinforcement learning-based method for determining multi-level working conditions in marine drilling includes: Collect multi-sensor data sequences and extract multi-dimensional feature vectors from the multi-sensor data sequences; The multidimensional feature vector is input into the pre-trained state probability prediction model to obtain the state probability vector, which represents the probability of occurrence of each working condition within the next time period T. Each working condition includes the normal state and each risk state. Send the state probability vector to the staff and monitor the staff's actions within the next time period T. After time T, multi-sensor data sequences are collected and multi-dimensional feature vectors are extracted. These vectors are then input into the state probability prediction model to obtain a new state probability vector. Calculate the reward value based on the two state probability vectors and the operation actions; The parameters of the state probability prediction model are iteratively updated using reinforcement learning methods, with reward values ​​serving as signals.

[0006] Preferably, the extraction of multi-dimensional feature vectors from multi-sensor data sequences includes: Extract the mean, variance, extreme values, and extreme value locations of each sensor data within the acquisition time window as time-series features; The skewness, kurtosis, and quantiles of the data from each sensor are extracted as statistical features. Extract the slope and acceleration of the change trend of each sensor data within the acquisition time window as trend features; Extract the dominant frequency and power spectral density distribution of each sensor data sequence as frequency domain features; By concatenating time-series features, statistical features, trend features, and frequency domain features, a multidimensional feature vector is obtained.

[0007] Preferably, the extraction of the dominant frequency and power spectral density distribution of each sensor data sequence includes: Spectral analysis was performed on the data sequences from each sensor to extract the frequency component with the most concentrated energy as the dominant frequency; Extract the power spectral density within a preset frequency band as the frequency domain energy distribution feature; The main frequency and frequency domain energy distribution characteristics of each sensor are spliced ​​together to form the frequency domain characteristics.

[0008] Preferably, the training method for the state probability prediction model includes: Extract multi-sensor data sequences from historical drilling data and extract multi-dimensional feature vectors; Label each multidimensional feature vector with a working condition status label. The working condition status label is determined based on whether each working condition actually occurs within a time T after the end of the time window corresponding to the multidimensional feature vector. Using multidimensional feature vectors as input and operating condition labels as supervision targets, the state probability prediction model is trained by multi-label classification to obtain a pre-trained state probability prediction model.

[0009] Preferably, the calculation of the reward value based on the two state probability vectors and the operation action includes: Read the types and adjustments of operation parameters actually performed by staff within time T, and match them with a pre-established risk reduction operation mapping library. The risk reduction operation mapping library records the risk status corresponding to the types and adjustments of operation parameters. If a match is successful, the corresponding risk status is obtained, and the reward value is calculated based on the risk status and the probability vectors of the two statuses. If a match is not found, the reward value will be recorded as zero.

[0010] Preferably, the calculation of the reward value based on the risk state and the probability vectors of the two states includes: Extract the probability of the risk state and the probability of the normal state from the two state probability vectors respectively; Based on the change in the probability of the risk state between the two state probability vectors, the initial reward for the risk state is calculated. Based on the change in the probability of the normal state between the two state probability vectors, the initial reward value is corrected to obtain the corrected reward. The probability values ​​corresponding to each risk state in the previous time-instance state probability vector are weighted, and the reward value is obtained by weighted summation of each correction reward.

[0011] Preferably, calculating the initial reward for the risk state based on the change in the probability of the risk state between the two state probability vectors includes: The probability change of the risk state is obtained by subtracting the probability value of the risk state in the state probability vector at the next time step from the probability value of the risk state in the state probability vector at the previous time step, and this change is used as the initial reward. The method of correcting the initial reward value based on the change in the probability of the normal state between two state probability vectors to obtain the corrected reward includes: The change in the probability of the normal state is obtained by subtracting the probability value of the normal state in the state probability vector of the previous time step from the probability value of the normal state in the state probability vector of the next time step. The modified reward is obtained by adding the product of the preset correction coefficient and the change in the probability of the normal state to the initial reward.

[0012] Preferably, the parameters of the state probability prediction model iteratively updated using reinforcement learning methods include: The continuously calculated reward values ​​are accumulated in chronological order with a discount to obtain the long-term cumulative return estimate at the current moment. The discount factor is greater than zero and less than one, so that the reward value further away from the current moment contributes less to the long-term cumulative return estimate. The prediction quality of the current state probability prediction model output strategy is measured by the long-term cumulative return estimate, and the parameters of the state probability prediction model are updated in the direction of improving the long-term cumulative return, using the prediction quality as a signal. The above process is repeated continuously to iteratively optimize the parameters of the state probability prediction model.

[0013] Preferably, updating the parameters of the state probability prediction model using prediction quality as a signal along the direction of improving long-term cumulative returns includes: The policy gradient is constructed by multiplying the logarithm of the predicted probability of each working condition state by the weighted sum of the corresponding working condition state labels and the long-term cumulative return estimate. The gradient of the policy gradient with respect to the network parameters of the state probability prediction model is obtained, and the parameter update amount is obtained by multiplying the obtained gradient by the preset learning rate. The parameter update amount is added to the current network parameters to complete one iteration update.

[0014] This invention also provides a reinforcement learning-based multi-level operating condition discrimination system for marine drilling, used to implement a reinforcement learning-based multi-level operating condition discrimination method for marine drilling, comprising: The feature extraction module is used to collect multi-sensor data sequences and extract multi-dimensional feature vectors from the multi-sensor data sequences. The state probability prediction module is used to input multi-dimensional feature vectors into a pre-trained state probability prediction model to obtain a state probability vector. The state probability vector represents the probability of occurrence of each working condition within the next time period T, where each working condition includes normal state and each risk state. The information interaction module is used to send the state probability vector to the staff and monitor the staff's actions within the next time period T. The repetitive execution module is used to collect multi-sensor data sequences and extract multi-dimensional feature vectors after time T, and then input them into the state probability prediction model again to obtain a new state probability vector. The reward calculation module is used to calculate the reward value based on two state probability vectors and the operation action; The strategy optimization module is used to continuously update the parameters of the state probability prediction model using reinforcement learning methods, with reward values ​​as signals.

[0015] This invention provides a method and system for determining multi-level working conditions in marine drilling based on reinforcement learning, which has the following beneficial effects: This invention collects multi-sensor data sequences and extracts multi-dimensional features including time series, statistics, trends, and frequency domain. The multi-dimensional feature vectors are then input into a state probability prediction model, which outputs the probability of each operating condition occurring within a future time period T. This provides workers with an objective, quantitative reference in the form of probability vectors. The state probability vectors encompass the independent probabilities of normal states and various risk states, with each dimension output independently, simultaneously reflecting the concurrent occurrence of multiple types of risks. Compared to existing methods that rely on workers' subjective experience for qualitative judgment, this invention transforms the comprehensive analysis process of multi-dimensional sensor data into an automated calculation process by the model. The judgment results are presented as probability values, unaffected by individual differences in worker experience, thus exhibiting stronger objectivity and consistency.

[0016] This invention reads the worker's actions at the end of each inference cycle and matches them with a risk reduction operation mapping library. It calculates a reward value based on the change in the probability of the corresponding risky state and the probability of the normal state in the state probability vectors before and after the operation. This reward value is then used as a signal to continuously update the parameters of the state probability prediction model using a policy gradient method. This step enables the model to continuously adjust its prediction strategy based on the worker's response behavior during actual operations. As effective reward signals accumulate, the model's adaptability continuously increases, and the prediction accuracy improves with each operation. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a reinforcement learning-based method for determining multi-level working conditions in marine drilling according to the present invention. Figure 2 This is a schematic diagram of a multi-level working condition discrimination system for marine drilling based on reinforcement learning, according to the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1

[0020] Please see Figure 1 In this embodiment, a method for determining multi-level working conditions in marine drilling based on reinforcement learning includes: S1. Collect multi-sensor data sequences and extract multi-dimensional feature vectors from the multi-sensor data sequences; The extraction of multi-dimensional feature vectors from multi-sensor data sequences includes: Extract the mean, variance, extreme values, and extreme value locations of each sensor data within the acquisition time window as time-series features; The skewness, kurtosis, and quantiles of the data from each sensor are extracted as statistical features. Extract the slope and acceleration of the change trend of each sensor data within the acquisition time window as trend features; Extract the dominant frequency and power spectral density distribution of each sensor data sequence as frequency domain features; By concatenating time-series features, statistical features, trend features, and frequency domain features, a multidimensional feature vector is obtained.

[0021] The extraction of the dominant frequency and power spectral density distribution of each sensor data sequence includes: Spectral analysis was performed on the data sequences from each sensor to extract the frequency component with the most concentrated energy as the dominant frequency; Extract the power spectral density within a preset frequency band as the frequency domain energy distribution feature; The main frequency and frequency domain energy distribution characteristics of each sensor are spliced ​​together to form the frequency domain characteristics.

[0022] In this embodiment, it should be noted that, based on drilling engineering experience, early signs of major operational anomalies are usually reflected in sensor readings within tens of seconds. However, it typically takes several seconds to tens of seconds for relevant personnel to perceive the anomaly and complete operational adjustments. Considering both aspects, T can be selected within the range of 20 to 120 seconds according to operational needs. In this embodiment, T=30 seconds is chosen.

[0023] At the beginning of each operating cycle, the system collects various types of sensor data reflecting the current drilling status from the drilling platform's sensor network. Key physical quantities during the drilling process include wellbore pressure, fluid inflow / outflow, drill bit stress and motion, and drill string vibration. Abnormal changes in these physical quantities directly characterize various operational risks. This embodiment uses seven types of sensors—standby pressure, return flow rate, mechanical drilling rate, torque, drilling pressure, rotational speed, and vibration acceleration—as examples. Each sensor continuously collects readings at 1-second intervals, and the system uses the most recent 30 seconds of collected data to form a multi-sensor data sequence.

[0024] For each type of sensor, the system extracts features from a 30-second reading sequence at four levels. The first level focuses on the overall distribution of values ​​within the sequence, extracting the mean, variance, maximum, minimum values, and their temporal positions within the sequence. Taking riser pressure as an example, the mean reflects whether the pressure level is within the normal range, the variance reflects the severity of pressure fluctuations, and the positions of the maximum and minimum values ​​reveal the direction of pressure change; if the minimum value consistently appears at the end of the sequence, it indicates a continuous decrease in pressure. The second level focuses on the shape characteristics of the distribution, extracting skewness, kurtosis, the 25th percentile, and the 75th percentile. Again using riser pressure as an example, under normal operating conditions, the pressure distribution is relatively symmetrical, with skewness close to zero; if a unilateral pressure peak occurs, the skewness will significantly deviate from zero, and the kurtosis will increase accordingly. These features are highly sensitive for capturing short-term, sudden anomalies. The third level focuses on the dynamic evolution direction of the sequence, performing linear fitting on the reading sequence with the time sequence as the independent variable. The fitting slope characterizes the overall trend of the readings within the window, and the difference in slope between the two halves estimates the acceleration of change. Taking the outflow rate as an example, a positive slope indicates a continuously rising flow rate. If the acceleration is also positive, it suggests that the flow rate is accelerating, a typical characteristic of the early evolution of overflow risk. The fourth level focuses on the frequency domain structure of the sequence. By performing spectral analysis on a 30-second reading sequence, the frequency component with the largest amplitude in the power spectrum is extracted as the dominant frequency, and the power spectral density within the 0.1 Hz to 1 Hz band is extracted as the frequency domain energy distribution characteristic. Taking vibration acceleration as an example, the energy distribution within this band is relatively uniform during normal drilling. When the drill string experiences stick-slip vibration, a concentrated spectral peak will be generated at a specific frequency. Anomalies in the dominant frequency and the power spectral density of this band are important discriminative features for stick-slip vibration risk.

[0025] S2. Input the multidimensional feature vector into the pre-trained state probability prediction model to obtain the state probability vector. The state probability vector represents the probability of occurrence of each working condition within the next T time period, where each working condition includes the normal state and each risk state. The training method for the state probability prediction model includes: Extract multi-sensor data sequences from historical drilling data and extract multi-dimensional feature vectors; Label each multidimensional feature vector with a working condition status label. The working condition status label is determined based on whether each working condition actually occurs within a time T after the end of the time window corresponding to the multidimensional feature vector. Using multidimensional feature vectors as input and operating condition labels as supervision targets, the state probability prediction model is trained by multi-label classification to obtain a pre-trained state probability prediction model.

[0026] In this embodiment, it should be noted that drilling conditions describe the current state of the drilling process, including normal drilling conditions and various risk conditions that may affect operational safety and efficiency. This embodiment uses five types of conditions—normal drilling, overflow risk, lost circulation risk, stuck pipe risk, and stick-slip vibration risk—as examples. In practical applications, the types of conditions can be expanded according to the geological characteristics of the work area and engineering needs.

[0027] The output of the state probability prediction model is a vector with the same dimension as the number of operating conditions. In this embodiment, it is five-dimensional. Each dimension corresponds to the probability of a certain type of operating condition occurring in the next 30 seconds. The output values ​​of each dimension are between 0 and 1 and are independent of each other. The sum of the dimensions is unrestricted. This design allows multiple risk states to have high probability outputs simultaneously, truly reflecting the situation of concurrent risks.

[0028] The model can employ a multi-layer fully connected network structure. The input layer receives multi-dimensional feature vectors, which are then transformed nonlinearly through several hidden layers to extract high-level feature representations. The output layer is independently activated for each of the five operating conditions, outputting the probability value for each condition. This structure supports gradient calculation via backpropagation and is fully compatible with the policy gradient update method in the subsequent reinforcement learning stage. The pre-trained network parameters can be directly used as initial parameters for the reinforcement learning stage.

[0029] The model's pre-training was performed using historical drilling data. Multi-sensor data sequences were extracted from the historical data in 30-second time windows, and multi-dimensional feature vectors were extracted to form the training sample set. For each training sample, historical engineering event records within 30 seconds after the end of the corresponding time window were reviewed. Five-dimensional labels for each working condition were assigned based on whether it actually occurred within that time period, with each dimension taking either 0 or 1 (1 for actual occurrence, 0 for non-occurrence). For example, if a stuck pipe risk operation was recorded within 30 seconds after the end of the corresponding time window, the label for this training sample would be 1, and the other dimensions would be labeled according to the actual records from the same period. If no other risk events occurred during the same period, the other dimensions would all be 0, resulting in a label of [0,0,0,1,0]. The engineering event records came from drilling logs including well control operation records, leak sealing operation records, stuck pipe operation records, and vibration anomaly alarm records. The correspondence between various events and working conditions was pre-defined and uniformly executed by drilling engineers to ensure consistent labeling. By training with multi-label classification using 0 / 1 labels, the model learns to estimate the probability of occurrence of the corresponding working condition for each output dimension. After training convergence, for a new input feature vector, the model outputs probability estimates for each dimension. For example, an inference output of [0.18, 0.71, 0.04, 0.09, 0.06] indicates that under the current features, the probability of normal drilling within the next 30 seconds is 0.18, the probability of overflow risk is 0.71, the probability of lost circulation risk is 0.04, the probability of stuck pipe risk is 0.09, and the probability of stick-slip vibration risk is 0.06. Pre-training enables the model to have basic recognition capabilities for the main working condition features, providing a reasonable starting point for subsequent online parameter updates in reinforcement learning.

[0030] S3. Send the state probability vector to the staff and monitor the staff's actions within the next T time period. In this embodiment, it should be noted that the output of the state probability prediction model can be sent to relevant personnel in real time to the drilling platform monitoring terminal, so that they can combine their own experience to make judgments on the working conditions and operational decisions.

[0031] Within 30 seconds of the state probability vector being pushed, the system continuously records all adjustments made to the operating parameters by relevant personnel. The records include the parameter type, adjustment amount, and adjustment time, where the adjustment amount is a directional value: positive values ​​indicate an increase, and negative values ​​indicate a decrease. Recording is achieved by reading the operation logs of the drilling platform's SCADA system (Supervisory and Data Acquisition System). The SCADA system collects the setpoints of each operating parameter at a rate of one second, automatically capturing each adjustment action by relevant personnel by comparing the differences in setpoints between adjacent sampling times. For example, if the SCADA system records that the pump displacement setpoint changes from 800 liters per minute to 700 liters per minute at a certain moment, the system automatically records an operation record: parameter type is pump displacement, adjustment amount is -100 liters per minute, and adjustment time is the corresponding timestamp. After 30 seconds, the system summarizes all operation records within that period as input for reward value calculation.

[0032] S4. After time T, collect multi-sensor data sequences and extract multi-dimensional feature vectors, then input them into the state probability prediction model again to obtain a new state probability vector. In this embodiment, after time T, the system re-acquires the latest multi-sensor data sequence and extracts multi-dimensional feature vectors, then inputs them again into the state probability prediction model to obtain a new state probability vector. This new state probability vector reflects the system's latest prediction of the current operating condition after a time interval T. At this point, the system possesses the previous state probability vector, the new state probability vector, and the operation action records within the last 30 seconds, and proceeds to the reward value calculation stage.

[0033] S5. Calculate the reward value based on the two state probability vectors and the operation actions; The calculation of the reward value based on the two state probability vectors and the operation action includes: Read the types and adjustments of operation parameters actually performed by staff within time T, and match them with a pre-established risk reduction operation mapping library. The risk reduction operation mapping library records the risk status corresponding to the types and adjustments of operation parameters. If a match is successful, the corresponding risk status is obtained, and the reward value is calculated based on the risk status and the probability vectors of the two statuses. If a match is not found, the reward value will be recorded as zero.

[0034] The calculation of the reward value based on the risk state and the probability vectors of the two states includes: Extract the probability of the risk state and the probability of the normal state from the two state probability vectors respectively; Based on the change in the probability of the risk state between the two state probability vectors, the initial reward for the risk state is calculated. Based on the change in the probability of the normal state between the two state probability vectors, the initial reward value is corrected to obtain the corrected reward. The probability values ​​corresponding to each risk state in the previous time-instance state probability vector are weighted, and the reward value is obtained by weighted summation of each correction reward.

[0035] The calculation of the initial reward for the risk state based on the change in the probability of the risk state between the two state probability vectors includes: The probability change of the risk state is obtained by subtracting the probability value of the risk state in the state probability vector at the next time step from the probability value of the risk state in the state probability vector at the previous time step, and this change is used as the initial reward. The method of correcting the initial reward value based on the change in the probability of the normal state between two state probability vectors to obtain the corrected reward includes: The change in the probability of the normal state is obtained by subtracting the probability value of the normal state in the state probability vector of the previous time step from the probability value of the normal state in the state probability vector of the next time step. The modified reward is obtained by adding the product of the preset correction coefficient and the change in the probability of the normal state to the initial reward.

[0036] In this embodiment, the reward value calculation uses the operational actions of relevant personnel as a reference signal to evaluate the prediction quality of the model. A risk reduction operation mapping library is pre-established based on expert experience in the drilling engineering field, recording the correspondence between operational parameter adjustments with a clear risk reduction intention and the corresponding risk states. Each record in the mapping library includes the type of operational parameter, the direction and range of the adjustment amount, and the corresponding risk state. Taking the five types of working conditions in this embodiment as examples, typical records in the mapping library include: a decrease in pump displacement exceeding 10% of the current set value, corresponding to a overflow risk; an increase in drilling fluid density exceeding 0.05 g / cm³, corresponding to a overflow risk; a decrease in drilling pressure exceeding 15% of the current set value or drill string lifting, corresponding to a stuck pipe risk; and a decrease in rotational speed exceeding 20% ​​of the current set value, corresponding to a stick-slip vibration risk. The adjustment threshold is set based on distinguishing between routine efficiency optimization fine-tuning and risk-responsive adjustments. The former is usually within 5% of the set value, while the latter is usually exceeding 10%. Each threshold can be adjusted according to specific operating procedures.

[0037] After reading the operation records within 30 seconds, the system matches each record with the mapping library. Taking a recorded pump displacement adjustment of -100 liters per minute and a current setting of 800 liters per minute as an example, the adjustment range is 12.5%, which exceeds the 10% threshold corresponding to overflow risk in the mapping library. The match is successful, and the corresponding risk status is obtained as overflow risk.

[0038] Taking the previous state probability vector [0.18, 0.71, 0.04, 0.09, 0.06] and the new state probability vector [0.52, 0.38, 0.04, 0.08, 0.05] as an example, the probabilities of overflow risk extracted from the two state probability vectors are 0.71 and 0.38, respectively, and the probabilities of normal state are 0.18 and 0.52, respectively. Subtracting the new overflow risk probability of 0.38 from the previous overflow risk probability of 0.71 yields an initial reward of 0.33. As the overflow risk probability decreases, the initial reward becomes positive. The change in normal state probability is 0.52 minus 0.18, which equals 0.34. Multiplying this by the preset correction coefficient of 0.5 by 0.34 yields 0.17, which is then added to the initial reward to obtain a corrected reward of 0.50. In this example, only one risk state is matched, so the corrected reward is the final reward value of 0.50.

[0039] If, within 30 seconds, relevant personnel simultaneously reduce the pump displacement by 12.5% ​​of the set value and the rotation speed by 22% of the set value, then both overflow risk and stick-slip vibration risk are matched. After calculating the correction reward for each, the overflow risk probability (0.71) and stick-slip vibration risk probability (0.06) in the previous state probability vector are used as weights to perform a weighted summation to obtain the final reward value.

[0040] If the relevant staff does not perform any operation exceeding the adjustment threshold within 30 seconds, or if the operation performed is not within the range of the mapping library record, the match is unsuccessful, the reward value for this time is recorded as zero, and the model parameters are not updated in this cycle. The correction coefficient controls the weight of the impact of the probability change of the normal state on the reward. In this embodiment, it is set to 0.5, but in practical applications, it can be adjusted within the range of 0.3 to 0.7.

[0041] S6. Continuously use the reward value as a signal and employ reinforcement learning methods to iteratively update the parameters of the state probability prediction model.

[0042] The parameters of the state probability prediction model iteratively updated using reinforcement learning methods include: The continuously calculated reward values ​​are accumulated in chronological order with a discount to obtain the long-term cumulative return estimate at the current moment. The discount factor is greater than zero and less than one, so that the reward value further away from the current moment contributes less to the long-term cumulative return estimate. The prediction quality of the current state probability prediction model output strategy is measured by the long-term cumulative return estimate, and the parameters of the state probability prediction model are updated in the direction of improving the long-term cumulative return, using the prediction quality as a signal. The above process is repeated continuously to iteratively optimize the parameters of the state probability prediction model.

[0043] The method of updating the parameters of the state probability prediction model in the direction of improving long-term cumulative returns using prediction quality as a signal includes: The policy gradient is constructed by multiplying the logarithm of the predicted probability of each working condition state by the weighted sum of the corresponding working condition state labels and the long-term cumulative return estimate. The gradient of the policy gradient with respect to the network parameters of the state probability prediction model is obtained, and the parameter update amount is obtained by multiplying the obtained gradient by the preset learning rate. The parameter update amount is added to the current network parameters to complete one iteration update.

[0044] In this embodiment, after each reward value calculation, the system adds the reward value to the historical reward sequence. The long-term cumulative return estimate is obtained by accumulating the historical reward sequence with a discount. The value of the discount factor determines the system's emphasis on long-term rewards. In this embodiment, it is set to 0.95, corresponding to the contribution of long-term rewards decaying to about 5% after about 60 steps (about 30 minutes), covering the typical timescale from the initial appearance to the confirmation of drilling anomalies. In practical applications, it can be adjusted within the range of 0.9 to 0.99. The system performs a parameter update every 20 steps of accumulated reward values ​​to balance the update frequency and the stability of gradient estimation. In practical applications, it can be adjusted within the range of 10 to 50 steps.

[0045] Parameter updates are implemented using the policy gradient method. For the current inference, the model outputs the predicted probabilities of five working conditions. The actual working conditions observed after time T are recorded with 0 / 1 labels, using the same labeling method as in the pre-training phase. The logarithmic sum of the predicted probabilities of each working condition and their corresponding labels is weighted and summed, then multiplied by the long-term cumulative reward estimate to construct the policy gradient. When the long-term cumulative reward estimate is positive, the gradient update strengthens the parameters consistent with the current prediction direction; when the long-term cumulative reward estimate is negative, the gradient update suppresses the parameters corresponding to the current prediction direction, guiding the model to evolve towards a more accurate prediction policy. The gradient of the policy gradient with respect to the network parameters is calculated. The parameter update amount is obtained by multiplying the resulting gradient by a preset learning rate (0.0001 in this embodiment, but can be adjusted within the range of 0.00005 to 0.001 according to training stability in practical applications), and then superimposed on the current network parameters to complete one step of the update.

[0046] The above process is continuously repeated as drilling operations proceed. When the reward value is zero for several consecutive cycles, the parameter update is zero, and the model parameters remain unchanged, ensuring that model updates are always driven by meaningful operational behaviors, and that the pre-trained parameters do not degrade due to invalid signals. As effective reward signals accumulate, the prediction accuracy of the state probability prediction model continuously improves.

[0047] Example 2

[0048] Please see Figure 2This invention provides a reinforcement learning-based multi-level operating condition discrimination system for marine drilling, used to implement a reinforcement learning-based multi-level operating condition discrimination method for marine drilling, comprising: The feature extraction module is used to collect multi-sensor data sequences and extract multi-dimensional feature vectors from the multi-sensor data sequences. The state probability prediction module is used to input multi-dimensional feature vectors into a pre-trained state probability prediction model to obtain a state probability vector. The state probability vector represents the probability of occurrence of each working condition within the next time period T, where each working condition includes normal state and each risk state. The information interaction module is used to send the state probability vector to the staff and monitor the staff's actions within the next time period T. The repetitive execution module is used to collect multi-sensor data sequences and extract multi-dimensional feature vectors after time T, and then input them into the state probability prediction model again to obtain a new state probability vector. The reward calculation module is used to calculate the reward value based on two state probability vectors and the operation action; The strategy optimization module is used to continuously update the parameters of the state probability prediction model using reinforcement learning methods, with reward values ​​as signals.

[0049] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only one method, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0050] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

[0051] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for determining multi-level working conditions in marine drilling based on reinforcement learning, characterized in that, include: Collect multi-sensor data sequences and extract multi-dimensional feature vectors from the multi-sensor data sequences; The multidimensional feature vector is input into the pre-trained state probability prediction model to obtain the state probability vector, which represents the probability of occurrence of each working condition within the next time period T. Each working condition includes the normal state and each risk state. Send the state probability vector to the staff and monitor the staff's actions within the next time period T. After time T, multi-sensor data sequences are collected and multi-dimensional feature vectors are extracted. These vectors are then input into the state probability prediction model to obtain a new state probability vector. Calculate the reward value based on the two state probability vectors and the operation actions; The parameters of the state probability prediction model are iteratively updated using reinforcement learning methods, with reward values ​​serving as signals.

2. The method for determining multi-level working conditions in marine drilling based on reinforcement learning according to claim 1, characterized in that, The extraction of multi-dimensional feature vectors from multi-sensor data sequences includes: Extract the mean, variance, extreme values, and extreme value locations of each sensor data within the acquisition time window as time-series features; The skewness, kurtosis, and quantiles of the data from each sensor are extracted as statistical features. Extract the slope and acceleration of the change trend of each sensor data within the acquisition time window as trend features; Extract the dominant frequency and power spectral density distribution of each sensor data sequence as frequency domain features; By concatenating time-series features, statistical features, trend features, and frequency domain features, a multidimensional feature vector is obtained.

3. The method for determining multi-level working conditions in marine drilling based on reinforcement learning according to claim 2, characterized in that, The extraction of the dominant frequency and power spectral density distribution of each sensor data sequence includes: Spectral analysis was performed on the data sequences from each sensor to extract the frequency component with the most concentrated energy as the dominant frequency; Extract the power spectral density within a preset frequency band as the frequency domain energy distribution feature; The main frequency and frequency domain energy distribution characteristics of each sensor are spliced ​​together to form the frequency domain characteristics.

4. The method for determining multi-level working conditions in marine drilling based on reinforcement learning according to claim 1, characterized in that, The training method for the state probability prediction model includes: Extract multi-sensor data sequences from historical drilling data and extract multi-dimensional feature vectors; Label each multidimensional feature vector with a working condition status label. The working condition status label is determined based on whether each working condition actually occurs within a time T after the end of the time window corresponding to the multidimensional feature vector. Using multidimensional feature vectors as input and operating condition labels as supervision targets, the state probability prediction model is trained by multi-label classification to obtain a pre-trained state probability prediction model.

5. The method for determining multi-level working conditions in marine drilling based on reinforcement learning according to claim 1, characterized in that, The calculation of the reward value based on the two state probability vectors and the operation action includes: Read the types and adjustments of operation parameters actually performed by staff within time T, and match them with a pre-established risk reduction operation mapping library. The risk reduction operation mapping library records the risk status corresponding to the types and adjustments of operation parameters. If a match is successful, the corresponding risk status is obtained, and the reward value is calculated based on the risk status and the probability vectors of the two statuses. If a match is not found, the reward value will be recorded as zero.

6. The method for determining multi-level working conditions in marine drilling based on reinforcement learning according to claim 5, characterized in that, The calculation of the reward value based on the risk state and the probability vectors of the two states includes: Extract the probability of the risk state and the probability of the normal state from the two state probability vectors respectively; Based on the change in the probability of the risk state between the two state probability vectors, the initial reward for the risk state is calculated. Based on the change in the probability of the normal state between the two state probability vectors, the initial reward value is corrected to obtain the corrected reward. The probability values ​​corresponding to each risk state in the previous time-instance state probability vector are weighted, and the reward value is obtained by weighted summation of each correction reward.

7. The method for determining multi-level working conditions in marine drilling based on reinforcement learning according to claim 6, characterized in that, The calculation of the initial reward for the risk state based on the change in the probability of the risk state between the two state probability vectors includes: The probability change of the risk state is obtained by subtracting the probability value of the risk state in the state probability vector at the next time step from the probability value of the risk state in the state probability vector at the previous time step, and this change is used as the initial reward. The method of correcting the initial reward value based on the change in the probability of the normal state between two state probability vectors to obtain the corrected reward includes: The change in the probability of the normal state is obtained by subtracting the probability value of the normal state in the state probability vector of the previous time step from the probability value of the normal state in the state probability vector of the next time step. The modified reward is obtained by adding the product of the preset correction coefficient and the change in the probability of the normal state to the initial reward.

8. The method for determining multi-level working conditions in marine drilling based on reinforcement learning according to claim 1, characterized in that, The parameters of the state probability prediction model iteratively updated using reinforcement learning methods include: The continuously calculated reward values ​​are accumulated in chronological order with a discount to obtain the long-term cumulative return estimate at the current moment. The discount factor is greater than zero and less than one, so that the reward value further away from the current moment contributes less to the long-term cumulative return estimate. The prediction quality of the current state probability prediction model output strategy is measured by the long-term cumulative return estimate, and the parameters of the state probability prediction model are updated in the direction of improving the long-term cumulative return, using the prediction quality as a signal. The above process is repeated continuously to iteratively optimize the parameters of the state probability prediction model.

9. The method for determining multi-level working conditions in marine drilling based on reinforcement learning according to claim 8, characterized in that, The method of updating the parameters of the state probability prediction model in the direction of improving long-term cumulative returns using prediction quality as a signal includes: The policy gradient is constructed by multiplying the logarithm of the predicted probability of each working condition state by the weighted sum of the corresponding working condition state labels and the long-term cumulative return estimate. The gradient of the policy gradient with respect to the network parameters of the state probability prediction model is obtained, and the parameter update amount is obtained by multiplying the obtained gradient by the preset learning rate. The parameter update amount is added to the current network parameters to complete one iteration update.

10. A reinforcement learning-based multi-level operating condition discrimination system for marine drilling, used to implement the reinforcement learning-based multi-level operating condition discrimination method for marine drilling as described in any one of claims 1-9, characterized in that, include: The feature extraction module is used to collect multi-sensor data sequences and extract multi-dimensional feature vectors from the multi-sensor data sequences. The state probability prediction module is used to input multi-dimensional feature vectors into a pre-trained state probability prediction model to obtain a state probability vector. The state probability vector represents the probability of occurrence of each working condition within the next time period T, where each working condition includes normal state and each risk state. The information interaction module is used to send the state probability vector to the staff and monitor the staff's actions within the next T time period. The repetitive execution module is used to collect multi-sensor data sequences and extract multi-dimensional feature vectors after time T, and then input them into the state probability prediction model again to obtain a new state probability vector. The reward calculation module is used to calculate the reward value based on two state probability vectors and the operation action; The strategy optimization module is used to continuously update the parameters of the state probability prediction model using reinforcement learning methods, with reward values ​​as signals.

Images