Responsive real-time drilling performance system

Abstract

The present invention relates to a machine learning model training method for monitoring well drilling performance, and the monitoring method using the machine learning model; and the monitoring method comprises the steps of receiving at least two data inputs, each data input comprising data of a different modality, the data relating to the well drilling, processing the received data inputs to generate a multi-modal representation of the data inputs by using a data fusion scheme, and inputting the multi-modal representation of the data inputs into a machine learning model so as to generate a result relating to a drilling performance.

Description

[0001] RESPONSIVE REAL-TIME DRILLING PERFORMANCE SYSTEM

[0002] FIELD OF THE INVENTION

[0003] The invention relates to a responsive real-time drilling performance system. Specifically, the invention relates to a responsive real-time drilling performance monitoring system for improving the efficiency of drilling operations by forecasting non-productive time.

[0004] TECHNICAL BACKGROUND

[0005] Drilling operations involve drilling a wellbore through an underground layer of, e.g., hydrocarbons. The drilling time can be divided into productive time, also known as PT, when the drill bit is advancing the wellbore, and non-productive time, also known as NPT, when the drill bit is idle or performing other tasks. NPT can have various causes, such as unskilled workers, faulty equipment, or environmental factors. NPT can also increase the costs of drilling operations by more than about one third. Therefore, it is desirable to minimize NPT and its causes. One way to measure NPT is by using invisible lost time, also known as ILT, which is the time wasted due to suboptimal performance of the drilling operations. For instance, ILT can occur when the drilling parameters are not adjusted properly, or when the cementing process takes longer than expected. However, ILT is not easy to quantify and is usually estimated after the drilling operations are finished by comparing the actual duration with a standard benchmark. This method is not very accurate and does not identify the root cause of ILT.

[0006] Currently, the market is dominated by manual monitoring solutions that require human operators to observe and record the drilling operations. These solutions are labor-intensive and error-prone, as they depend on the skill and attention of the operators. Moreover, they result in delayed responses and incomplete data due to continuous human oversight challenges. Some semi-automatic solutions exist that use sensors and software to collect and analyze the drilling data, but they are not real-time and do not provide proactive recommendations to optimize the drilling operations. These solutions are inadequate to address the complex and dynamic nature of the drilling operations and the ILT factors. Furthermore, none of the conventional approaches provide an automated, real-time, and intelligent solution that can predict and prevent ILT and enhance the efficiency of the drilling operations. The conventional approaches cannot respond to dynamic changes in the monitored environment either.

[0007] Hence, there is a need for a better way to assess and reduce ILT. More specifically, there is a need for a system and method for monitoring the drilling performance that is holistic, efficient, accurate, and has real-time capabilities of monitoring diverse systems.

[0008] SUMMARY OF THE INVENTION

[0009] Some, or all of the above objectives are achieved by the invention as defined by the features of the independent claims. Preferred embodiments of the invention are defined by the features of the dependent claims.

[0010] According to a first aspect of the present disclosure, there is provided a computer- implemented method for training a machine learning model for monitoring well drilling performance, comprising the steps of: receiving at least two data inputs, each data input comprising data of a different modality, the data relating to the well drilling; processing the received data inputs to generate a multi-modal representation of the data inputs by using a data fusion scheme; and training the machine learning model with the multi-modal representation of the data inputs.

[0011] The incorporation of multi-modal data sources, including e.g. time series, text, and video, allows for a more comprehensive and nuanced understanding of drilling activities. Specifically, by strategically fusing data through artificial intelligence (Al), the arrangement of the first aspect ensures a more nuanced and highly accurate drilling activity segmentation.

[0012] In a second aspect of the present disclosure, according to the first aspect, the data fusion scheme is an early fusion scheme. In a third aspect of the present disclosure, according to the first or second aspects, the processing step comprises the step of embedding the received data inputs so as to generate the multi-modal representation of the data inputs.

[0013] The early fusion scheme according to the second aspect and the third aspect has the benefit of providing a truly multi-modal feature representation of drilling performance since all the features are merged from the beginning. It is more efficient, as it does not require separate processing of multiple streams of data and separate training steps.

[0014] In a fourth aspect of the present disclosure, according to the first aspect, the data fusion scheme is a late fusion scheme.

[0015] In a fifth aspect of the present disclosure, according to the first or the preceding aspects, the processing step comprises the steps of: training a set of multiple machine learning models with the received data inputs, embedding a set of outputs from the trained set of machine learning models, and combining the embedded set of outputs to generate the multi-modal representation of the data input; and the machine learning model training step comprises the step of training a machine learning model which is not comprised in the set of multiple machine learning models, with the multi-modal representation of the data input.

[0016] The benefit of late fusion according to the fourth and fifth aspects is that it combines the outputs of the Al model of each classifier or modality to produce new outputs that are more precise and reliable.

[0017] In a sixth aspect of the present disclosure, according to any one of the above aspects, the data inputs comprise times series data, textual data, visual data and / or audio data. Preferably, a machine learning model is trained with domain-based rules if time-series data input is included in the data input; transformers for code / subcode classification is trained if textual data is included in the data input; and a Convolutional Neural Network for pose estimation, object detection, and / or activity recognition is trained if visual data is included in the data input. In a seventh aspect of the present disclosure, according to any one of the above aspects, the times series data includes surface sensor time series data, and / or along string measurements data, preferably representing pressure, temperature and / or vibrations at different joints of the string of a drill pipe; the textual data includes daily drilling reports; the visual data includes video data and / or image data of the surface of the well, preferably of the rig floor; and / or the audio data includes data representing machinery noise of the drilling pipe.

[0018] According to an eighth aspect of the present disclosure, there is provided a computer- implemented method for automated and real-time monitoring of well drilling performance using artificial intelligence technology, comprising the steps of: receiving at least two data inputs, each data input comprising data of a different modality, the data relating to the well drilling; processing the received data inputs to generate a multi-modal representation of the data inputs by using a data fusion scheme; and inputting the multi-modal representation of the data inputs into a machine learning model so as to generate a result relating to a drilling performance.

[0019] Similar to the first aspect, the incorporation of multi-modal data sources, including time series, text, and video, allows for a more comprehensive and nuanced understanding of drilling activities.

[0020] In a nineth aspect of the present disclosure, according to the preceding aspect, the machine learning model has been trained according to the method of any one of the first to the seventh aspects.

[0021] In a tenth aspect of the present disclosure, according to the eighth or nineth aspects, the data fusion scheme is an early fusion scheme.

[0022] In an eleventh aspect of the present disclosure, according to the eighth to tenth aspects, the processing step comprises the step of embedding the received data inputs so as to generate the multi-modal representation of the data inputs.

[0023] Similar to the first to the third aspects, the early fusion scheme according to the eighth aspect to the eleventh aspect has the benefit of providing a truly multi-modal feature representation of drilling performance since all the features are merged from the beginning. It can be more efficient, as it does not require separate processing of multiple streams of data and separate training steps.

[0024] In a twelfth aspect of the present disclosure, according to the eighth or nineth aspects, the data fusion scheme is a late fusion scheme.

[0025] In a thirteenth aspect of the present disclosure, according to the eighth, nineth, or twelfth aspects, the processing step comprises the steps of: inputting the received data inputs of the different modality of data into a set of multiple machine learning models, embedding a set of outputs from the set of machine learning models, and combining the embedded set of outputs to generate the multi-modal representation of the data input; wherein the generating step comprises the step of inputting the multi-modal representation of the data input into a machine learning model which is not comprised by the set of machine learning models, so as to generate a result relating to the drilling performance.

[0026] Similar to the fourth and fifth aspects, the benefit of late fusion, according to the eighth, nineth, twelfth and thirteenth aspects, is that it combines the outputs of the Al model of each classifier or modality to produce new outputs that are more precise and reliable.

[0027] In a fourteenth aspect of the present disclosure, according to any one of the eighth to thirteenth aspects, the result relating to the drilling performance includes at least a Key Performance Indicator, Rig States and / or an invisible lost time period.

[0028] In a fifteenth aspect of the present disclosure, according to the preceding aspect, the Key Performance Indicator includes a macro Key Performance Indicator and / or a micro Key Performance Indicator.

[0029] In a sixteenth aspect of the present disclosure, according to any one of the eighth to fourteenth aspects, the result relating to the drilling performance is generated based on at least the output of the machine learning model and a domain-based drilling performance analysis method.

[0030] In a seventeenth aspect of the present disclosure, according to any one of the eighth to fifteenth aspects, the data inputs comprise times series data, textual data, visual data and / or audio data. Preferably, a machine learning model is utilized with domain-based rules if time-series data input is included in the data input; transformers for code / subcode classification are utilized if textual data is included in the data input; and a Convolutional Neural Network for pose estimation, object detection, and / or activity recognition is utilized if visual data is included in the data input.

[0031] In an eighteenth aspect of the present disclosure, according to any one of the eighth to sixteenth aspects, the times series data includes surface sensor time series data, and / or along string measurements data, preferably representing pressure, temperature and / or vibrations at different joints of the string of a drill pipe; the textual data includes daily drilling reports; the visual data includes video data and / or image data of the surface of the well, preferably of the rig floor; and / or the audio data includes data representing machinery noise of the drilling pipe.

[0032] According to a nineteenth aspect of the present disclosure, there is provided a computer-implemented method for automated and real-time monitoring of well drilling performance, comprising the steps of: receiving at least two data inputs, each data input comprising data of a different modality, the data relating to the well drilling, processing the received data inputs to generate a multi-modal representation of the data inputs by using a late fusion scheme; and outputting a result relating to a drilling performance based on the multi-modal representation of the data inputs.

[0033] In a twentieth aspect of the present disclosure, according to the preceding aspect, the processing step further comprises the steps of: inputting the received data inputs of the different modality of data into a set of multiple machine learning models, embedding a set of outputs from the set of machine learning models, and combining the embedded set of outputs to generate the multi-modal representation of the data input. Similar to the above aspects, the benefit of late fusion according to the nineteenth and twentieth aspects is that it combines the outputs of the Al model of each classifier or modality to produce new outputs that are more precise and reliable.

[0034] According to a twenty-first aspect of the present disclosure, there is provided a computer program product comprising instructions which, when executed by a processor, causes the processor to cariy out the method of any one of the preceding aspects.

[0035] BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Fig. i shows a schematic view of a wellsite, according to an embodiment of this invention;

[0037] Fig. 2 shows a flowchart of a method of monitoring a drilling performance using a late fusion scheme according to an embodiment of this invention;

[0038] Fig. 3 shows a schematic structural diagram of a method of monitoring a drilling performance using a late fusion scheme according to an embodiment of this invention;

[0039] Fig. 4 shows a schematic structural diagram of an example of a method of monitoring a drilling performance using a late fusion scheme with multi-modal heterogenous (multiresolution) representations of the asset (Rig + Crew) alignment of multiple sources of data describing the same activity according to an embodiment of this invention;

[0040] Fig. 5 shows a schematic structural diagram of an example of a method of monitoring a drilling performance using a late fusion scheme by processing of different modality resulting in a multi-resolution complementary prediction and at different granularity of details according to an embodiment of this invention;

[0041] Fig. 6 shows a schematic structural diagram of a fusion step in a method of monitoring a drilling performance using a late fusion scheme according to an embodiment of this invention;

[0042] Fig. 7 shows a flowchart of a method of monitoring a drilling performance using early fusion according to an embodiment of this invention;

[0043] Fig. 8 shows a schematic structural diagram of a method of monitoring a drilling performance using an early fusion scheme according to an embodiment of this invention; and Fig. 9 shows a flowchart of a method of a machine learning model according to an embodiment of this invention.

[0044] DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0045] Preferred embodiments of the present invention are described hereinafter and in conjunction with the accompanying drawings.

[0046] [SYSTEM]

[0047] This invention relates to and is applied to drilling operations of a well or wellbore (which refers to a hole in a formation made by drilling or insertion of a conduit into a formation) which can be done by a control and management system.

[0048] The control or management system has a drilling system 11 and a drilling performance monitoring system io that can monitor the drilling operations by processing data with artificial intelligence technology. The monitoring system io can also transfer signals that mimic the signals from a driller. The drilling system 11 can control the trajectory of the drilling by using unmanned survey systems and steering logic.

[0049] Fig. 1 shows the control or management system with the monitoring system io and the drilling system 11 that can drill automatically according to an embodiment of this invention. The drilling system 11 is at formation. The drilling system 11 may have a drilling platform, a pump, a drill string, and a bottom hole assembly. The drill string is made of drill pipes that are added one by one to the drill string as the well is drilled in the formation. The drilling platform has a carriage, a rotary drive system, and a pipe handling system. The drilling platform can drill the well and move the drill string and the bottom hole assembly into the formation. There is an annular opening between the outside of the drill string and the walls of the well. There may be a casing in the well. The casing may cover the whole well or a part of it. The bottom hole assembly has a drill collar, a mud motor, a drill bit, and a measurement while drilling (MWD) tool of the monitoring system io.

[0050] The monitoring system io includes an artificial intelligence (Al) engine. The Al engine may be configured to employ machine learning, of any variety, such as neural networks or supervised learning (e.g., a support-vector machine), to name just two examples among many contemplated. The artificial intelligence engine may be trained to detect and monitor the drilling events and drilling activities based on the multi-modal data mentioned below. The Al engine, such as a computer, is adapted to receive input data signals, perform sampling or other necessary computations upon the data signals, and output time synchronized data corresponding to the unsynchronized input data signals. As used herein, “adapted” refers to the ability of an apparatus to achieve a specified result, through the operation of software and hardware.

[0051] [REAL-TIME & MULTI-MODAL DATA]

[0052] Most of the current solutions that assesses the performance of drilling operations rely on historical data, and often use single historical data, such as data based on a specific sensor or a specific sensor group, for example, only time-series data. This may limit the accuracy and comprehensiveness of the analysis, as historical data may not reflect the current or future conditions of the data entities. Moreover, single- or mono-modal data, such as time series data from drilling report, may not capture the interactions and correlations among different sensors, data sources, and data entities. Relying solely on one source also poses a considerable risk. This limitation is mainly caused by the fact that analyzing large amounts of data can be challenging for the user, especially when the data comes from multiple sources attached to various objects or drilling control or management systems, such as geological features, that the user is interested in.

[0053] Therefore, this invention uses real-time information about the properties of these objects, the geologic environment, the control or management systems (e.g., measurements taken by the sensors over time), and / or models created to estimate or predict their properties. For instance, in some cases, the data may include information about different geological features or structures related to oil drilling, such as regions, fields, reservoirs, and / or wells. The user (e.g., an evaluator, operator, or analyst) may use this data to assess the performance of the well drilling. In this context, a “data entity” or “entity” refers to any object or system that can provide data, such as from sensors or other equipment, which are discussed in detail below. Examples of data entities are individual oil wells, groups of oil wells, oil platforms, geographic regions, reservoirs, fields, etc. Data entities may also be other objects or systems not related to oil well development, such as electronic devices, landfills, robotics, mechanical systems, etc. For example, various systems and methods for analyzing data entities and visualizing data entity series are disclosed herein. In other words, this invention utilizes multi-modal data for monitoring the drilling performance. The multi-modal data or the data of multiple modalities refer to data obtained or derived from multiple sensor types or domains, including cyber domains. Some examples of the multi modal data are: data packets, network flow, computer records, signal data, visible light, video footage, audio, infrared, multi-spectral, hyperspectral, synthetic aperture radar, moving target indicator radar, technical data, reports and so on. Unlike existing solutions that predominantly rely on mono-modal data inputs, the present invention integrates diverse data streams, providing a comprehensive and nuanced understanding of drilling operations. This comprehensive approach ensures a more accurate representation of the monitored system, enhancing the reliability of information obtained. Specifically, this multi-modal approach allows for a 360-degree view of activities within the rig, enabling more effective monitoring and the prompt detection of inefficiencies.

[0054] As shown in FIG. 1, the geologic environment may have layers (e.g., stratification) with a reservoir and other features like faults, geo-body, and so on.

[0055] The MWD tool of the monitoring system 10 may be equipped with various measurement and detecting units 12, such as, sensors, detectors, actuators, etc. In other words, The MWD tool has sensors that measure aspects in the drilling system, the well, and / or the formation. For instance, a land equipment (not shown) may have communication circuitry to send and receive information from one or more networks that connect the measurement and detecting units 12 and the land equipment. Specifically, in the well in Fig. 1, the monitoring system 10 has various sensors (such as temperature, pressure, and flow rate sensors) along the string measurements that gather real-time data for computing; for example, one or more series related to the oil well (such as how much oil the well produces over time). Real-time data may be obtained based on measurements from the sensors working at the present time. For instance, a temperature sensor may be used to produce real-time data for a temperature series of a region entity. Real-time data may also be obtained through other ways, such as data derived from measurements from one or more sensors, regular observation logs, data retrieved from a database, and the like. The kinds of sensors that may be used include temperature sensors, pressure sensors, depth sensors, oil / gas rating sensors (e.g., to measure a quality rating of extracted oil or gas), liquid volume sensors (e.g., to measure a volume of extracted oil), water level sensors, and the like sensors that are arranged along the string. For example, the pump may have pressure sensors, a suction flow meter, and a return flow meter. The pressure sensors can measure the pressure of the fluid in the drilling system 11. In one example, one of the pressure sensors measures the standpipe pressure. The flow meters can measure how much fluid is going in and out of the drill string. Also, one or more models can be built to estimate series values related to the oil well to predict how the oil well will perform in the future. The past data and models can be seen as data sources together.

[0056] The monitoring system 11 may further comprise an image capture unit 14 recording visual data of geofences, which comprise cameras 14 on devices such as drones or mobile towers. In the preferred embodiment, the image capture unit 14 records visual data of the rig surface. The image capture unit 14 can perform various functions such as recording, detecting, obstructing, or tracking objects or areas within or outside the geofence, depending on the entitlements, instructions, and / or requirements. Specifically, the system uses entitlements, instructions, and / or requirements that are embedded in the fencing agent and can be updated by the owner or licensee of the geofence in real-time or near real-time. The system can also adapt to different factors such as time, weather, or location. Specifically, the image capture unit 14 takes photos or videos of the objects or areas within or outside the geofence. For example, the video footage on the rig floor is analyzed for a better understanding of operation sequences, durations, and potential areas for improvement. In another example, the image capture unit 14 stays within a certain distance from the objects or areas within or outside the geofence. The image capture unit 14 can be any type of device that captures images, such as digital or non-digital cameras, IR cameras, UV cameras, night vision cameras, thermal imaging cameras, x-ray cameras, gamma ray cameras, radio wave cameras, microwave cameras, radar, ultrasound, or any other imaging systems. In one example, the set of visual signals may originate from videos or images stored in a format like JPEG (or JPG), PNG, GIF, MPEG-2, MPEG-4.

[0057] The monitoring system 11 may further comprise audio capture unit (not shown). The audio capture unit may comprise a set of audio signals corresponding with the geologic environment and the drilling system 11. In one example, the set of audio signals may originate from audio stored in an uncompressed audio format, for example WAV, and / or a lossy compression audio format, for example MP3. The audio capture unit may be microphone recording equipment. Specifically, the audio signals are made up from a number of individual signals including machinery noise, caused by, for example, the engine(s), propeller shaft(s), fuel pump(s) etc.

[0058] The monitoring system 11 may further comprise a wellsite cloud framework. A wellsite cloud framework can mean an information technology system, like a single rig in the oilfield. In this case, the wellsite cloud framework can be a system of connected devices or the “Internet of Things” (loT). For example, a wellsite “cloud” framework can be part of the cloud, such as a network that connects other networks, or a separate network.

[0059] Other equipment may be situated away from a well site and have sensing, detecting, emitting or other circuitries. This equipment, such as a satellite system, may have storage and communication circuitry to store and transmit data, instructions, etc.

[0060] Furthermore, other kinds of data sources may be used, which include data relating to human experiences that provide solutions and guidance for a variety of situations. Data set in the case-base reflect interesting situations that require the attention of the operators. Interesting situations are identified by examining human experience in daily drilling reports (or DDR) from the operator 13, best practices and other related documents. The guidance part and the static data in a data set are created (manually by experts) using information in document knowledge management systems. The situation description part is generated automatically using actual logs from previous drilling operations. From the raw data in the logs, interpreted events and processing functions produce their output, and the relevant data is added to the case. The manually entered and the automatically generated data together form the completed data set.

[0061] The use of various data types or multi-modal data enhances the granularity of insights, contributing to a more accurate and responsive system for optimizing drilling processes.

[0062] [ARTIFICIAL INTELLEGENCE MODEL]

[0063] In order to further improve the ability of assessing the drilling performance, the monitoring system 10 utilizes a data fusion scheme and artificial intelligence (Al) technology through a multi-modal approach. More specifically, with the Al engine, the monitoring system 10 has the ability to employ early and late fusion strategies for robust Al performance. The data fusion strategy or data fusion process is the combination or integration of data or information from multiple sources to achieve improved accuracy and more specific inferences than can be obtained from the use of, for example, a single sensor alone. The monitoring system 10 takes inputs from various sources as mentioned above, such as time series, text, and video data, and can seamlessly scale to incorporate new data sources like loT and microphone data, which analyzes machineiy noise to identify activities. This adaptive fusion approach allows for the optimal combination of information from different modalities at various stages of the monitoring process.

[0064] Figs. 2 and 3 show flowcharts of the monitoring method using late fusion strategy implemented by the Al engine.

[0065] In the start phase of the operation, the monitoring system 10 collects (S11) and preprocesses (S12) multi-modal data from, for example, the geologic environment, and, especially, the drilling rig. The raw multi-modal data is processed so as to remove noise and inconsistencies existing in the raw data. The processed data is then inputted into multiple machine learning models for further processing (S13).

[0066] As shown in the figure, in the preferred embodiment, the data of each modality has a corresponding machine learning model aligned for processing.

[0067] Specifically, the measurement and detecting unit 12 collects time-series data, which is a critical input for this method. The time-series data is then received by the Al engine, which analyze the time-series data using a combination of machine learning algorithms and rules derived from extensive drilling knowledge.

[0068] Simultaneously, the text data, such as DDR, is input by the operator and is input into the Al engine and is processed by the Al engine using advanced Natural Language Processing (NLP) techniques, including but not limited to transformer models, to extract valuable information from textual sources related to drilling operations.

[0069] Video data collected by, for example, CCTV 14 are transferred to the Al engine to undergo analysis through multiple Convolutional Neural Networks (CNNs) for various tasks such as pose estimation, localization, object detection, and tracking.

[0070] Other data, such as audio data collected by microphones recording machine noises of the drilling system 11, may also be transferred to the Al engine so as to undergo analysis through a machine learning model.

[0071] The use of multi-modal data enhances the granularity of insights, contributing to a more accurate and responsive system for optimizing drilling processes.

[0072] In the middle phase, the monitoring system 10 employs a debiasing mechanism by taking multiple views of the drilling operations. Specifically, the preprocessed data is further processed by the Al engine with a data fusion strategy. This ensures that decision-making is robust and not influenced by a single source, thereby enhancing the reliability of the monitoring process. The fusion modules, either through data fusion (early fusion) or decision fusion (late fusion) architectures, strategically combine information from different modalities.

[0073] In this embodiment, the monitoring system io fuses insights from time series, text, and video data with a late fusion strategy to provide a holistic understanding of the drilling environment, enabling more accurate and comprehensive performance monitoring.

[0074] Specifically, features or representation data that are representative of the outputs from the Al models are extracted or embedded separately per modality into a lowerdimensional space (S14). In other words, the uni-modal embeddings are mapped and integrated into a multi-modal common space, so as to generate a multi-modal representation of the data input.

[0075] As shown in Figs. 4 and 5, since the sensors 12, the DDR report 13 and the CCTV 14 can acquire data at different sampling rates or resolutions, some form of data synchronization may be required in order for information to be combined in a data fusion sense. Specifically, Fig. 4 shows multi-modal heterogenous (multi-resolution) representations of the asset (Rig + Crew) alignment of multiple sources of data of different modality describing the same activity. Fig. 5 shows the processing of data of different modality resulting in a multi-resolution complementary prediction as the right frame shows, and at different granularity of details as the left frame shows. These different resolutions are a result of the nature of the data. E.g., DDR text can describe hours’ worth of information in one sentence, while sensors are data points with a close to a second resolution. Accordingly, time windows are defined and the data and / or feature information received (at different data rates or resolutions) are processed within the windows in order to synchronize the data for subsequent analysis and fusion. As a result of this alignment process, data may have a feature extracted, such as a representation representing the presence of abnormal operation. Accordingly, multiple features are extracted per modality with reference to its data source. The multiple features, i.e., the mono-modal or uni-modal representation of the corresponding data input of the modalities, are combined into the multi-modal representation of the data input.

[0076] In the final phase, the monitoring system 10 generates actionable insights and recommendations based on the analyzed multi-modal data, i.e., the multi-modal representation of the data input. Fig. 6 shows the fusion of the multi-modal results into a multi-level output. The actionable insights and recommendations could include identifying inefficiencies, predicting potential issues, and determining their root causes.

[0077] In the present invention, this analyzing process is implemented by a machine learning model. This model can, for example, define the cause-and-effect relationships between monitored data, analyzed fault detections, fault isolations and analytical and empirical model outputs. Negative information from the monitored data is an essential component of reasoning. It refers to the expected features or measurements that would indicate a specific fault hypothesis. If these features or measurements are missing, then the fault hypothesis can be rejected. This reasoning is based on a causal network that establishes the connections between causes and effects and the confidence level of the diagnosis, so that this system io ensures robust operation. For example, in the event of a connectivity issue with one data source, the multi-modal nature of this approach enables the system to seamlessly continue functioning, ensuring high availability.

[0078] Accordingly, the monitoring system io of this invention can predict different drilling operations at different resolutions as shown in Fig. 5. These predictions are also complementary one to the other because of the granularity of the data and point of view describing the activity. For example, by coupling time series and video data, a stationery and a safety meeting can be determined. Then these predictions are aggregated in a way that enables the extraction of the maximum possible information related to drilling activity.

[0079] As explained above, not only the monitoring system 10, specifically the machine learning model, is able to aggregate and complement the outputs one from the other, but it is also able to generate different levels of aggregated output granularity including Rig States, Macro-KPIs, and Micro-KPIs.

[0080] In another embodiment of this invention, the Al engine uses the early fusion scheme that consist of using a single Al model that processes all the different inputs at once as Fig. 7 shows. The model makes sure after the training phase to output a unique result that is the most appropriate to the input. As shown in Fig. 8, all modalities are taken as input, and a joint multi-modal embedding is learned or implemented directly.

[0081] Specifically, similar to the late fusion scheme, in the start phase, the monitoring system 10 collects (S21) and preprocesses (S22) multi-modal data from the geologic environment, and especially the drilling rig. In the later phase, the raw multi-modal data are combined and embedded to an embedding space so as to generate a joint multi-modal embedding, i.e., the multimodal representation of all the data inputs from different modalities (S23).

[0082] In the final phase, the joint multi-modal embedding is fed to a neural network that processes it and outputs the activities (S24, S25). Preferably, the machine learning model is similar or identical to the one mentioned above for the late fusion scheme in a way such that it can implement the analyzing process. In other words, the machine learning model is not only able to aggregate and complement the outputs one from the other, but it is also able to generate different levels of aggregated output granularity including Rig States, Macro-KPIs, and Micro-KPIs. As is known, Key Performance Indicators (KPIs) refer to various metrics measured during a drilling procedure. The user may be able to observe the well through the KPI such as recorded times for various procedures, the operator present, and estimated delays, as well as other types of information.

[0083] Additionally, the monitoring system 10 comprises a user interface that can provide the above-mentioned result relating to the drilling performance. For example, the user interface can help to track the efficiency of the rig, produce reports, and communicate with others. It can also receive user input of efficiency metrics or estimates, as well as tool face set point values or ranges based on the result generated from the Al engine.

[0084] As a modified embodiment to the above embodiments, the system can also operate with data from just one modality.

[0085] In order to train the above-mentioned machine learning model, as shown in Fig. 9, in the beginning stage, the historical and / or real-time data set from different modalities are collected as input (S31). The collected data is then preprocessed to remove noise and inconsistencies existing in the raw data. In the middle stage, the data is processed to generate a multi-modal representation of the data inputs in the above-mentioned data fusion schemes (S32). Specifically, if an early fusion scheme is taken, the data of different modalities is embedded into the multi-modal representation directly. Alternatively, if a late fusion scheme is taken, a set of multiple machine learning models are trained with the preprocessed received data inputs. Specifically, for time-series data, a machine learning model is trained in combination with a domain-based drilling performance analysis method; for textual data, transformers for code / subcode classification are trained; and for visual data, a Convolutional Neural Network for pose estimation, object detection, and / or activity recognition is trained. The set of outputs from the trained set of machine learning models is embedded and combined to generate the multi-modal representation of the data input. The multi-modal representation is then fed into the machine learning model for outputting actionable insights and recommendations for the drilling operation (S33).

[0086] The ability to integrate and analyze diverse data types using advanced Al techniques positions this invention as a pioneering solution in drilling operations monitoring.

[0087] Specifically, the system 10 and the method implemented by the system 10 surpass the limitations of mono-modal systems, providing a 360-degree view of operations that significantly enhances accuracy and responsiveness. Unlike conventional methods that rely on single data types, the system and method 10 of this invention have the ability to seamlessly scale to new sources, such as loT and microphone data, further futureproofs the monitoring process, ensuring adaptability to evolving technological landscapes. From a technical standpoint, the utilization of advanced Al techniques, including machine learning for time series analysis, natural language processing for text processing, and Convolutional Neural Networks for video analysis, contributes to a more sophisticated and accurate monitoring system. The system’s debiasing mechanism, through the incorporation of multiple views and fusion strategies, mitigates the risks associated with relying on a single data source, improving the overall reliability of decision-making. In economic terms, the invention drives efficiency in drilling activities by adeptly identifying root causes and pinpointing sources of inefficiencies. This proactive system and method 10 not only minimizes visible downtime but also significantly reduces invisible lost time (ILT), enhancing overall operational efficiency. The economic benefits of the invention go beyond immediate cost reductions, playing a pivotal role in fostering long-term sustainability and bolstering competitiveness within the drilling industry. By systematically addressing inefficiencies and optimizing operational parameters, the invention contributes to sustained economic advantages, positioning drilling operations for enduring success in an increasingly competitive market landscape.

[0088] Overall, the invention’s technical prowess and economic efficiencies position it as a transformative advancement in the field of drilling operations monitoring.

[0089] The described approach can be implemented through computer-based methods, computer-readable media with instructions, and computer systems with memory and processors. Implementations of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Software implementations of the described subject matter can be implemented as one or more computer programs, that is, one or more modules of computer program instructions encoded on a tangible, non-transitory, computer- readable computer-storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively, or additionally, the program instructions can be encoded in / on an artificially generated propagated signal, for example, a machinegenerated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer-storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of computer-storage mediums. Configuring one or more computers means that the one or more computers have installed hardware, firmware, or software (or combinations of hardware, firmware, and software) so that when the software is executed by the one or more computers, particular computing operations are performed.

[0090] The term "real-time," "real time," "realtime," "real (fast) time (RFT)," "near(ly) realtime (NRT)," "quasi real-time," or similar terms (as understood by one of ordinary skill in the art), means that an action and a response are temporally proximate such that an individual perceives the action and the response occurring substantially simultaneously. For example, the time difference for a response to display (or for an initiation of a display) of data following the individual's action to access the data may be less than i ms, less than i sec, or less than 5 sees. While the requested data need not be displayed (or initiated for display) instantaneously, it is displayed (or initiated for display) without any intentional delay, taking into account processing limitations of a described computing system and time required to, for example, gather, accurately measure, analyze, process, store, or transmit the data.

[0091] The terms "data processing apparatus," "computer," or "electronic computer device" (or equivalent as understood by one of ordinary skill in the art) refer to data processing hardware and encompass all kinds of apparatus, devices, and machines for processing data, including by way of example, a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include special purpose logic circuitry, for example, a central processing unit (CPU), an FPGA (field programmable gate array), or an ASIC (application-specific integrated circuit). In some implementations, the data processing apparatus or special purpose logic circuitry (or a combination of the data processing apparatus or special purpose logic circuitry) may be hardware- or software-based (or a combination of both hardware- and software-based). The apparatus can optionally include code that creates an execution environment for computer programs, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of execution environments. The present disclosure contemplates the use of data processing apparatuses with or without conventional operating systems, for example LINUX, UNIX, WINDOWS, MAC OS, ANDROID, IOS, or any other suitable conventional operating system.

[0092] A computer program, which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, for example, files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network. While portions of the programs illustrated in the various figures are shown as individual modules that implement the various features and functionality through various objects, methods, or other processes, the programs may instead include a number of sub-modules, third-party services, components, libraries, and such, as appropriate. Conversely, the features and functionality of various components can be combined into single components, as appropriate. Thresholds used to make computational determinations can be statically, dynamically, or both statically and dynamically determined.

[0093] The methods, processes, or logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The methods, processes, or logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, for example, a CPU, an FPGA, or an ASIC.

[0094] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors, both, or any other kind of CPU. Generally, a CPU will receive instructions and data from and write to a memory. The essential elements of a computer are a CPU, for performing or executing instructions, and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to, receive data from or transfer data to, or both, one or more mass storage devices for storing data, for example, magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, for example, a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, for example, a universal serial bus (USB) flash drive, to name just a few.

[0095] Computer-readable media (transitory or non-transitory, as appropriate) suitable for storing computer program instructions and data includes all forms of permanent / nonpermanent or volatile / non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, for example, random access memory (RAM), read-only memory (ROM), phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable readonly memory (EEPROM), and flash memory devices; magnetic devices, for example, tape, cartridges, cassettes, internal / removable disks; magneto-optical disks; and optical memory devices, for example, digital video disc (DVD), CD-ROM, DVD+ / -R, DVD- RAM, DVD-ROM, HD-DVD, and BLURAY, and other optical memory technologies. The memory may store various objects or data, including caches, classes, frameworks, applications, modules, backup data, jobs, web pages, web page templates, data structures, database tables, repositories storing dynamic information, and any other appropriate information including any parameters, variables, algorithms, instructions, rules, constraints, or references thereto. Additionally, the memory may include any other appropriate data, such as logs, policies, security or access data, reporting files, as well as others. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device, for example, a CRT (cathode ray tube), LCD (liquid crystal display), LED (Light Emitting Diode), or plasma monitor, for displaying information to the user and a keyboard and a pointing device, for example, a mouse, trackball, or trackpad by which the user can provide input to the computer. Input may also be provided to the computer using a touchscreen, such as a tablet computer surface with pressure sensitivity, a multi-touch screen using capacitive or electric sensing, or other type of touchscreen. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, for example, visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

[0096] The term "graphical user interface," or "GUI," maybe used in the singular or the plural to describe one or more graphical user interfaces and each of the displays of a particular graphical user interface. Therefore, a GUI may represent any graphical user interface, including but not limited to, a web browser, a touch screen, or a command line interface (CLI) that processes information and efficiently presents the information results to the user. In general, a GUI may include a plurality of user interface (UI) elements, some or all associated with a web browser, such as interactive fields, pulldown lists, and buttons. These and other UI elements maybe related to or represent the functions of the web browser.

[0097] Implementations of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, for example, as a data server, or that includes a middleware component, for example, an application server, or that includes a front-end component, for example, a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of wireline or wireless digital data communication (or a combination of data communication), for example, a communication network. Examples of communication networks include a local area network (LAN), a radio access network (RAN), a metropolitan area network (MAN), a wide area network (WAN), Worldwide Interoperability for Microwave Access (WIMAX), a wireless local area network (WLAN) using, for example, 802.11 a / b / g / n or 802.20 (or a combination of 802.1 lx and 802.20 or other protocols consistent with this disclosure), all or a portion of the Internet, or any other communication system or systems at one or more locations (or a combination of communication networks). The network may communicate with, for example, Internet Protocol (IP) packets, Frame Relay frames, Asynchronous Transfer Mode (ATM) cells, voice, video, data, or other suitable information (or a combination of communication types) between network addresses.

[0098] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0099] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular implementations of particular inventions. Certain features that are described in this specification in the context of separate implementations can also be implemented, in combination, in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations, separately, or in any suitable sub-combination. Moreover, although previously described features may be described as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can, in some cases, be excised from the combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination.

[0100] Particular implementations of the subject matter have been described. Other implementations, alterations, and permutations of the described implementations are within the scope of the following claims as will be apparent to those skilled in the art. While operations are depicted in the drawings or claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed (some operations may be considered optional), to achieve desirable results. In certain circumstances, multitasking or parallel processing (or a combination of multitasking and parallel processing) may be advantageous and performed as deemed appropriate.

[0101] Moreover, the separation or integration of various system modules and components in the previously described implementations should not be understood as requiring such separation or integration in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0102] Accordingly, the previously described example implementations do not define or constrain this disclosure. Other changes, substitutions, and alterations are also possible without departing from the spirit and scope of this disclosure.

[0103] Furthermore, any claimed implementation is considered to be applicable to at least a computer-implemented method; a non-transitory, computer-readable medium storing computer-readable instructions to perform the computer-implemented method; and a computer system comprising a computer memory interoperably coupled with a hardware processor configured to perform the computer-implemented method or the instructions stored on the non-transitory, computer-readable medium.

Claims

CLAIMS1. A computer-implemented method for training a machine learning model for monitoring well drilling performance, comprising the steps of: receiving at least two data inputs, each data input comprising data of a different modality, the data relating to the well drilling; processing the received data inputs to generate a multi-modal representation of the data inputs by using a data fusion scheme; and training the machine learning model with the multi-modal representation of the data inputs.

2. The training method of the preceding claim, wherein the processing step comprises the step of embedding the received data inputs so as to generate the multimodal representation of the data inputs.

3. The training method of claim 1, wherein the processing step comprises the steps of: training a set of multiple machine learning models with the received data inputs, embedding a set of outputs from the trained set of machine learning models, and combining the embedded set of outputs to generate the multi-modal representation of the data input; wherein the machine learning model training step comprises the step of training a machine learning model which is not comprised in the set of multiple machine learning models, with the multi-modal representation of the data input.4- The training method of any one of the preceding claims, wherein the data inputs comprise times series data, textual data, visual data and / or audio data.

5. The training method of the preceding claim, wherein the times series data includes surface sensor time series data and / or along string measurements data, preferably representing pressure, temperature and / or vibrations at different joints of the string of a drill pipe; the textual data includes daily drilling reports; the visual data includes video data and / or image data of the surface of the well, preferably of the rig floor; and / or the audio data includes data representing machinery noise of the drilling pipe.

6. A computer-implemented method for automated and real-time monitoring of well drilling performance using artificial intelligence technology, comprising the steps of: receiving at least two data inputs, each data input comprising data of a different modality, the data relating to the well drilling; processing the received data inputs to generate a multi-modal representation of the data inputs by using a data fusion scheme; and inputting the multi-modal representation of the data inputs into a machine learning model so as to generate a result relating to a drilling performance.

7. The drilling performance monitoring method of the preceding claim, wherein the machine learning model has been trained according to the method of any one of claims 1 to 5.

8. The drilling performance monitoring method of claims 6 or 7, wherein the processing step comprises the step of embedding the received data inputs so as to generate the multi-modal representation of the data inputs.

9. The drilling performance monitoring method of claims 6 or 7, wherein the processing step comprises the steps of: inputting the received data inputs of the different modality of data into a set of multiple machine learning models, embedding a set of outputs from the set of machine learning models, and combining the embedded set of outputs to generate the multi-modal representation of the data input; and wherein the generating step comprises the step of inputting the multi-modal representation of the data input into a machine learning model which is not comprised by the set of machine learning models, so as to generate a result relating to the drilling performance.

10. The drilling performance monitoring method of any one of claims 6 to 9, wherein the result relating to the drilling performance includes at least a Key Performance Indicator, Rig States and / or an invisible lost time period.

11. The drilling performance monitoring method of the preceding claim, wherein the Key Performance Indicator includes a macro Key Performance Indicator and / or a micro Key Performance Indicator.

12. The drilling performance monitoring method of any one of claims 6 to 11, wherein the result relating to the drilling performance is generated based on at least the output of the machine learning model and a domain-based drilling performance analysis method.13- A computer program product comprising instructions which, when executed by a processor, causes the processor to carry out the method of any one of claims i to 12.

Images