Validating machine learning models for generating synthetic rock files

Abstract

The ML validation and validation system validates ML models using cross validation, consistency check, and physical modeling. Cross validation includes dividing a set of original rock files into a training subset and a validation subset, and generating a composite rock file for validation on the validation subset. Consistency checking includes generating a plurality of synthetic rock files using different implementations of the ML model and comparing their consistency. The validation engine generates a synthetic physical simulation and an original physical simulation using the physical model, and compares the two simulations.

Description

BACKGROUND

[0001] Downhole drilling systems use cutting structures such as drill bits to advance the depth of a wellbore. Drill bits include a plurality of cutting elements. The cutting elements have different geometries and shapes. A rock file is used to describe the cutting behavior of a particular cutting element (having a particular shape and a particular orientation or cutter-rock interaction state) on a particular rock type at a particular confining pressure. The rock file gives cutting forces when cutting the rock at a set of orientations and a set of cutting depths. During drill bit design, the rock file of a cutting element can be used to model the drill bit. Conventionally, the rock file is generated using experimental data, which can be limited based on experiments and other limitations. SUMMARY

[0002] In some embodiments, the technology described herein relates to a method. The method includes receiving a target configuration of a cutting element. The target configuration is related to an original configuration of an original rock file. The method includes applying a machine learning (ML) model to the target configuration to generate a synthetic rock file for the cutting element. The synthetic rock file includes a cutting force associated with the target configuration. An ML validation and confirmation system validates the ML model by generating a consistency ratio for the synthetic rock file based on a validation synthetic rock file generated by the ML model. The validation synthetic rock file is generated by using a different implementation of the ML model of the target configuration. The ML model is further validated by performing a physical simulation using the synthetic rock file and performing an original physical simulation using the original rock file.

[0003] In some embodiments, the technology described herein relates to a method. The method can include training a first implementation of a machine learning (ML) model of a cutting element using a first training subset of a plurality of original rock files. The method can also include applying the first implementation of the ML model to a target configuration of the cutting element to generate a first synthetic rock file. The target configuration can match an original configuration of a first validation subset of the plurality of original rock files. The ML validation and verification system can train a second implementation of the ML model using a second training subset of the plurality of original rock files and can apply the second implementation of the ML model to the target configuration of the cutting element to generate a second synthetic rock file. The second target configuration can match an original configuration of a second validation subset of the plurality of original rock files. The ML validation and verification system can validate the ML model by adjusting hyperparameters of the ML model based on a first cross-validation ratio between the first synthetic rock file and the first validation subset and a second cross-validation ratio between the second synthetic rock file and the second validation subset. A validation engine of the ML validation and verification system can generate a consistency ratio between the first synthetic rock file and the second synthetic rock file for the ML model. Using a physical model, the validation engine can perform a first synthetic physical simulation on the first synthetic rock file and a second synthetic physical simulation on the second synthetic rock file. The validation engine can also perform a first original physical simulation on the first validation subset and a second original physical simulation on the second validation subset. The validation engine can generate a simulation comparison ratio using the first synthetic physical simulation, the second synthetic physical simulation, the first original physical simulation, and the second original physical simulation.

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional features and aspects of implementations of the present disclosure will be set forth in part in the description below and, in part, will be apparent from the description, or can be learned by practice of such implementations. BRIEF DESCRIPTION OF DRAWINGS

[0005] In order to describe the manner in which the above-recited and other features of the disclosure can be obtained, a more particular description will be rendered by reference to specific implementations thereof, which are illustrated in the appended drawings. For better understanding, identical elements are denoted by the same reference numbers throughout the various drawings. While some of the drawings can be schematic or exaggerated representations of concepts, others can be drawn to scale. Understanding that the drawings depict some example implementations, the implementations will be described and explained with additional specificity and detail through the use of the accompanying drawings in which: Figure 1is a representation of a drilling system for drilling earth formations 101 according to at least one embodiment of the present disclosure; Figure 2 is a representation of ML validation and verification 212 according to at least one embodiment of the present disclosure; Figure 3 is a representation of a ML validation and verification system according to at least one embodiment of the present disclosure; Figure 4 is a representation of a cross-validation system according to at least one embodiment of the present disclosure; Figure 5 is a representation of a consistency check validation system according to at least one embodiment of the present disclosure; Figure 6 is a representation of a physical comparison validation system according to at least one embodiment of the present disclosure; Figure 7 is a representation of a radar plot generated from a physical simulation comparison of a drill bit according to the use of a synthetic rock file and an original rock file to simulate a cutting element in a drill bit device according to at least one embodiment of the present disclosure; Figure 8 is a representation of a box plot generated from a physical simulation comparison of a drill bit indicating variation in different applications of a physical model according to at least one embodiment of the present disclosure; Figure 9 is a representation of a rock file plot according to at least one embodiment of the present disclosure; Figure 10 is a flowchart of a method for validating a ML model according to at least one embodiment of the present disclosure; Figure 11 is a flowchart of a method for validating a ML model according to at least one embodiment of the present disclosure; and Figure 12 is a representation of a computing system according to at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0006] The present disclosure relates generally to apparatuses, systems, and methods for training and validating a machine learning (ML) model for generating synthetic rock files for cutting elements utilized on drill bits. The ML model can be trained from original rock files of cutting element geometries, orientations, confining pressures, and rock types. The original rock files can be generated experimentally. In some cases, the amount of information available from the original rock files can be limited. The ML model can generate a large number of synthetic rock files based on a relatively small dataset of original rock files. In some cases, the ML model can be trained to generate synthetic rock files for cutting elements having geometries different from the geometries used in the original rock files.

[0007] According to at least one embodiment of the present disclosure, the ML model can be validated using numerical error minimization by hyperparameter tuning of the ML model. For example, the ML model can be validated by cross-validation, such as by 90-10 cross-validation. In some embodiments, the ML model can be validated by consistency checks on multiple different implementations of the ML model. For example, synthetic rock files generated using different implementations of the ML model can be compared to determine bias between the synthetic rock files. In some embodiments, the ML model can be validated by simulating operation of a drill bit using the synthetic rock files with a physical model of the drill bit. Results of the physical model can be compared to results of the physical model implemented using the relevant original rock files.

[0008] Utilization of cross-validation, consistency checks, and physical model evaluation can facilitate improved validation and confirmation of the ML model. For example, based on results of at least one of cross-validation, consistency checks, and physical model evaluation, the ML model can be retrained. This can facilitate improved accuracy and / or relevance of synthetic rock files generated by the ML model. This can facilitate drilling personnel generating a library of synthetic rock files for cut element configurations for which original rock files do not exist. A validated ML model can allow for improved drill bit (and other cutting structure) design using synthetic rock files. This can improve drilling efficiency of drilling systems utilizing drill bits designed with synthetic rock files.

[0009] As shown by the foregoing discussion, the present disclosure utilizes various terminology to describe features and benefits of the synthetic rock file generation system. Additional details regarding the meaning of such terminology are now provided. For example, as used herein, the term “rock file” refers to a set of numbers describing physical forces generated when a cutting element interacts with a rock. Specifically, the term “rock file” can include information regarding a shape of the cutting element, such as a 3-dimensional shape, size, thickness, orientation (e.g., back rake angle, side rake angle), chamfer angle, any other information, and combinations thereof. In some embodiments, the term “rock file” can include information regarding a formation, such as a formation name, rock type, rock hardness, rock unconfined compressive strength, confining pressure, confining pressure medium, fracture and joint information, any other rock information, and combinations thereof. In some embodiments, the term “rock file” can include cutting forces, such as vertical force (e.g., F v ), lateral force (F side ), cutting force (F cut), any other force, and combinations thereof. The rock files can be determined using experimental data. Such rock files can be “raw rock files.” In some embodiments, the rock files can be determined using models or algorithms, such as physical models, analytical models, numerical simulation models, machine learning models, any other model, and combinations thereof. Such models can be “synthetic rock files.”

[0010] As specific, non-limiting examples, a raw rock file can include cutting forces derived through experiments for 3,000 psi confining pressure and 9,000 psi confining pressure. A synthetic rock model can include cutting forces at 6,000 psi confining pressure. In some examples, a raw rock file can include cutting forces derived through experiments for cutting element sizes including 9 mm, 13 mm, and 19 mm. A synthetic rock file can include cutting forces for cutting element sizes including 11 mm and 16 mm. In some examples, a raw rock file can include cutting forces for a flat shape, while a synthetic rock file can include cutting forces for a wedge-shaped cutting element. In some examples, a raw rock file can include cutting forces associated with a 5° back rake angle, while a synthetic rock file can include cutting forces associated with 0°, 10°, 15°, 20°, etc. back rake angles. In some examples, a raw rock file can include cutting forces associated with a 5° side rake angle, while a synthetic rock file can include cutting forces associated with 0°, 10°, etc. side rake angles.

[0011] As used herein, a “machine learning model” or “ML model” refers to a computer algorithm or model (e.g., a classification model, a regression model, a language model, an object detection model) that can be adjusted (e.g., trained) based on inputs or a set of inputs (e.g., training inputs) to generate an output of unknown information. For example, a machine learning model can refer to a neural network (e.g., a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN)) or other machine learning algorithm or architecture that learns and approximates complex functions and generates an output based on a plurality of inputs provided to the machine learning model.

[0012] As used herein, “physical simulation” or “physical model” refers to a physical model of a drilling system. The physical simulation can receive one or more rock files of a cutting element as input. Using the rock files, the physical simulation can analyze the impact of the cutting element in the drilling system. For example, the physical simulation can simulate the operation of a drill bit in a downhole drilling system. The drill bit can include a plurality of cutting elements in a drill bit design. Each cutting element can have an associated configuration (e.g., shape, orientation, rock type). The physical simulation can use the rock files of the cutting elements in a particular formation and / or a particular rock type to simulate the forces experienced by the drill bit design. This can result in one or more of an unbalanced force, a radial unbalanced force, a circumferential unbalanced force, a side- tilt unbalanced force, a weight on bit (WOB), a bit torque, any other bit force, and combinations thereof. While embodiments of the present disclosure can discuss a drill bit or a drill bit configuration, the physical simulation can simulate any other cutting device or combination of cutting elements, such as a reamer, a casing cutter, any other cutting device, and combinations thereof. In some embodiments, the physical simulation can utilize any simulation method, such as finite element analysis.

[0013] As used herein, the term “validation” refers to a confirmation of an ML model. Validating an ML model can include determining whether the outputs produced by the ML model represent the real-world elements that the outputs are modeling. For example, validating an ML model of the present disclosure can include determining whether the cutting forces in a synthetic rock file represent the forces exhibited by a physical cutting element having the properties in the synthetic rock file. As will be discussed in further detail herein, validation can be performed using one or more mechanisms, such as cross-validation, consistency checks, physical simulations, any other validation mechanism, and combinations thereof.

[0014] Figure 1 One example of a drilling system 100 for drilling into an earth formation 101 to form a wellbore 102 is shown. The drilling system 100 includes a rig 103 for rotating a drilling tool assembly 104 that extends downhole into the wellbore 102. The drilling tool assembly 104 can include a drill string 105, a bottom hole assembly (“BHA”) 106, a drill bit 110 attached to a downhole end of the drill string 105.

[0015] Drill string 105 can include several joints of drill pipe 108 connected end-to-end by tool joints 109. Drill string 105 transmits drilling fluid through a central bore and rotational power from rig 103 to BHA 106. In some embodiments, drill string 105 can also include additional components, such as subs, pup joints, and the like. Drill pipe 108 provides a hydraulic passageway through which drilling fluid is pumped from the surface. The drilling fluid is discharged through nozzles, jets, or other orifices of selected size in drill bit 110 for cooling drill bit 110 and cutting structures thereon, and for lifting cuttings out of the wellbore as it is drilled.

[0016] BHA 106 can include drill bit 110 or other components. Example BHA 106 can include additional or other components (e.g., coupled between drill string 105 and drill bit 110). Examples of additional BHA components include drill collars, stabilizers, measurement-while-drilling (“MWD”) tools, logging-while-drilling (“LWD”) tools, downhole motors, under-reamers, casing shoes, hydraulic slip joints, jars, vibration or shock absorption tools, other components, or combinations of the foregoing. BHA 106 can also include a rotary steerable system (RSS). The RSS can include a directional drilling tool that changes the direction of drill bit 110, thereby changing the trajectory of the wellbore. At least a portion of the RSS can maintain a geostationary position relative to an absolute reference frame, such as gravity, magnetic north, and / or true north. Using measurements obtained through the geostationary position, the RSS can position drill bit 110, change the course of drill bit 110, and direct the directional drilling tool on a projected trajectory.

[0017] In general, drilling system 100 can include other drilling components and accessories, such as special valves (e.g., kelly cocks, blowout preventers, and safety valves). Additional components included in drilling system 100 can be considered part of drilling tool assembly 104, drill string 105, or BHA 106, depending on their location in drilling system 100.

[0018] The drill bit 110 in the BHA 106 can be any type of drill bit suitable for degrading downhole materials. For example, the drill bit 110 can be a drill bit suitable for drilling into the overburden formation 101. An example type of drill bit for drilling into a formation is a fixed-cutter or drag bit. In other embodiments, the drill bit 110 can be a mill used to remove metal, composite materials, elastomers, other downhole materials, or combinations thereof. For example, the drill bit 110 can be used with a whipstock to mill into a casing 107 that is run over the wellbore 102. The drill bit 110 can also be a flat mill used to mill out tools, plugs, cement, other materials, or combinations thereof within the wellbore 102. Cuttings or other drill cuttings formed by use of the mill can be lifted to the surface or can be allowed to fall downhole.

[0019] According to at least some embodiments of the present disclosure, the cutting elements on the drill bit 110 or other cutting structure can be described by a rock file of a particular cutting element in a particular configuration. As discussed herein, one or more of the rock files can be an original rock file developed at least in part using experimental data. For example, the cutting forces of an original rock file can be measured directly in an experimental apparatus by cutting into a formation with a cutting element having a particular configuration and measuring the resultant forces on the cutting element. According to at least one embodiment of the present disclosure, the rock file used to describe the cutting elements on the drill bit 110 can be a synthetic rock file. A synthetic rock file can include information about a cutting element that is not generated through experimentation. In some embodiments, at least a portion of the cutting forces of a cutting element are generated through experimentation and a portion are generated using one or more models (e.g., analytical models and / or ML models discussed herein). In some embodiments, none of the information in a synthetic rock file can be generated through experimentation.

[0020] A synthetic rock file used to design the drill bit 110 can be determined using a ML model. For example, the ML model can be applied to a target configuration of a cutting element and used to develop a cutting force associated with the target configuration. In some embodiments, the target configuration can be the same, similar, and / or related to an original configuration of an original rock file. In some embodiments, the target configuration can be different from an original configuration of an original rock file. For example, as discussed herein, a ML model can be validated against an original rock file by generating a synthetic rock file related to an original configuration of the original rock file.

[0021] According to at least one embodiment of the present disclosure, the ML model can be validated to improve the accuracy and / or representativeness of the resulting synthetic rock files for physical conditions. The ML model can be validated using one or more validation mechanisms. For example, as discussed in further detail herein, the ML model can be validated using a cross-validation mechanism, a consistency check, and a physical simulation of the drilling system in which a cutting element described by a synthetic rock file can be used.

[0022] In some embodiments, the physical simulation can simulate the operation of the BHA 106 and / or individual elements of the BHA 106. This can allow the drill tool designer and / or the drill tool planner to determine how the BHA 106 can perform under a particular set of drilling conditions, such as the surface formation 101, the depth of the BHA 106, the confining pressure of the surface formation 101 at the BHA 106, the rock type of the surface formation 101 at the BHA 106, the configuration of the BHA 106, any other elements of the drilling system 100, and combinations thereof.

[0023] Figure 2 is a representation of an ML validation and confirmation system 212 according to at least one embodiment of the present disclosure. The ML validation and confirmation system 212 includes an ML model 214. The ML model 214 can be trained using original rock files 216. The ML model 214 can be trained to output one or more synthetic rock files 218.

[0024] The ML model 214 can be validated using a validation engine 220. The validation engine 220 can determine whether the synthetic rock files 218 generated by the ML model 214 represent the actual cutting forces experienced by the associated physical cutting element. In some embodiments, the ML validation and confirmation system 212 can utilize cross-validation. For example, to validate the ML model 214, the ML validation and confirmation system 212 can divide the original rock files 216 into a training subset 222 and a validation subset 224 and one or more test subsets. The training subset 222 can be used to train the ML model 214. For example, the training subset 222 can be used to train the ML model 214 to generate synthetic rock files 218 that represent portions of the original rock files 216 in the training subset 222. The test subsets can be used to test the trained ML model 214.

[0025] The validation subset 224 can be used by the validation engine 220 to validate the ML model 214. For example, the ML model 214 can generate synthetic rock files 218 using known cutting element configurations, such as the cutting element configurations in the validation subset 224. The validation engine 220 can compare the synthetic rock files 218 to the validation subset 224 to determine whether the synthetic rock files 218 match the validation subset 224.

[0026] The ML validation and confirmation system 212 can determine 226 whether the synthetic rock file 218 is validated or whether the synthetic rock file 218 matches or approximately matches the validation subset 224. For example, the ML validation and confirmation system 212 can determine 226 how much the synthetic rock file 218 differs from the validation subset 224. If the synthetic rock file 218 matches the validation subset 224 or exceeds a threshold match, the ML conditioner 228 can retrain the ML model 214 using the synthetic rock file 218.

[0027] In some embodiments, the ML conditioner 228 can condition one or more hyperparameters of the ML model 214. The hyperparameters of the ML model 214 can be parameters in the ML model 214 that control the learning process. Other parameters of the ML model 214 can be adjusted or derived during training of the ML model 214. For example, the ML conditioner 228 can identify which portions of the synthetic rock file 218 do not match the validation subset 224 of the original rock file. The ML conditioner 228 can determine which hyperparameter or hyperparameters can adjust the outputted synthetic rock file 218. Adjusting or conditioning the hyperparameters can adjust the outputted synthetic rock file 218.

[0028] In some embodiments, if the ML validation and confirmation system 212 determines 226 that the synthetic rock file 218 has been validated by the validation subset 224, the ML validation and confirmation system 212 can output one or more validated synthetic rock files 230. The validated synthetic rock files 230 can represent the physical conditions experienced by the cutting element.

[0029] In some embodiments, the validation engine 220 can implement cross-validation with any proportion of the original rock file 216. For example, the validation engine 220 can implement 90-10 cross-validation. In 90-10 cross-validation, the validation engine 220 can divide the original rock file 216 into 10 portions, resulting in 10 different training subsets 222 and 10 different validation subsets 224. The ML validation and confirmation system 212 can generate a synthetic rock file 218 for each of the 10 portions. Each synthetic rock file 218 can be validated using the associated validation subset 224. In some embodiments, each synthetic rock file 218 can be generated using a configuration or one of the configurations of the associated validation subset 224.

[0030] Each of the synthetic rock files 218 can be validated by the validation engine 220, and the ML validation and confirmation system 212 can determine 226 whether the synthetic rock files 218 match the validation subset 224. The ML tuner 228 can tune one or more hyperparameters of the ML model 214 based on the synthetic rock files 218 generated using the 90-10 cross-validation. In some embodiments, the ML tuner 228 can tune the ML model 214 using each of the synthetic rock files 218 in the 90-10 cross-validation. In some embodiments, the ML tuner 228 can tune the ML model 214 using the synthetic rock files 218 that do not match the associated validation subset 224. In some embodiments, the ML tuner 228 can tune the ML model 214 using the synthetic rock file(s) 218 that do match the associated validation subset 224. For example, if one or more of the synthetic rock files 218 do not match the associated validation subset 224, the ML tuner 228 can tune the ML model 214 using the synthetic rock files 218 that do match the validation subset 224.

[0031] In some embodiments, the validation engine 220 can validate the ML model 214 using a consistency check of the synthetic rock files 218 generated by different implementations of the ML model 214. For example, the ML model 214 can include different implementations trained by different training subsets 222. The resulting synthetic rock files 218 can be generated using the same cutting element configuration. The validation engine 220 can compare the cutting forces of the synthetic rock files 218 generated by the different implementations of the ML model 214. The validation engine 220 can generate a consistency ratio based on the differences between the various synthetic rock files 218. If the consistency ratio is different from a consistency ratio threshold, the ML validation and confirmation system 212 can determine 226 that the ML model 214 is not validated, and can retrain the ML model 214 using the differences between the synthetic rock files 218. This can help improve the consistency of the ML model 214.

[0032] In some embodiments, the validation engine 220 can perform a physical simulation (e.g., a synthetic physical simulation) on a drill bit or other cutting structure having cutting elements simulated by the synthetic rock files 218. In some embodiments, the validation engine 220 can perform a simulation (e.g., an original physical simulation) on a drill bit or other cutting structure having cutting elements simulated by the validation subset 224, where the synthetic rock files 218 and the validation subset 224 have the same configuration.

[0033] The validation engine 220 can compare the synthetic physical simulation using the synthetic rock files 218 to the original physical simulation using the validation subset 224 of the original rock files 216. For example, the validation engine 220 can generate a simulation comparison ratio of one or more bit performance generated by the synthetic physical simulation and the original physical simulation. If the simulation comparison ratio is above a simulation comparison ratio threshold, the validation engine 220 can validate the ML model 214. If the simulation comparison ratio is below the simulation comparison ratio threshold, the validation engine 220 can not validate the ML model 214, and the validation engine 220 can retrain the ML model 214 based on the synthetic physical simulation and the original physical simulation.

[0034] In some embodiments, the validation engine 220 can use each of the validation mechanisms to validate the ML model 214 simultaneously. For example, the validation engine 220 can use all three of cross-validation, consistency checks, and physical simulation to validate the ML model 214. This multi-factor cross-validation can help improve the validation and confirmation of the ML model 214. In some embodiments, the validation engine 220 can use only one or more of the validation mechanisms to validate the ML model 214.

[0035] According to at least one embodiment of the present disclosure, the ML model 214 can be trained using a relatively small dataset. For example, as discussed herein, the number of original rock files 216 can be limited, such as limited by constraints on the availability of experimental data used to generate the original rock files 216.

[0036] In some embodiments, the ML model 214 can be relatively small. For example, the ML model 214 can include less than 5 hidden layers. In some examples, the ML model 214 can include 50 to 100 parameters. In one particular non-limiting example, the ML model 214 has one hidden layer and 66 parameters. The small size of the ML model 214 can allow the ML model 214 to be trained based on a small dataset. As discussed herein, if the resulting synthetic rock files 218 are not validated using cross-validation, consistency checks, and physical simulation, the validation engine 220 can validate and retrain the ML model 214. The ML model 214 can be retrained until the synthetic rock files 218 are validated. When the synthetic rock files 218 are validated, the ML model 214 can be validated, and the ML model 214 can be used to generate additional synthetic rock files 218.

[0037] In some embodiments, the validated ML model 214 can be used to generate synthetic rock files 218 for cutting element configurations that are not included in the original rock files 216. To generate the synthetic rock files 218, the ML model 214 can receive a target configuration for a cutting element, which includes a shape of the cutting element, an orientation of the cutting element (e.g., a back rake angle, a side rake angle), a depth of cut, a rock type of the cutting element, any other portion of the configuration, and combinations thereof. In some examples, the target configuration can be the same, similar, or related to the original configuration of the original rock files. In some examples, new synthetic rock files 218 can be generated for target orientations that have new geometries, orientations, shapes, or other configurations of the synthetic rock files 218 that are not included in the original rock files 216. For example, new synthetic rock files 218 can be generated for new configurations that are different from the original configurations. In this way, the ML validation and confirmation system 212 can generate synthetic rock files 218 for cutting elements and / or configurations of cutting elements that have not yet been physically tested. This can help drilling personnel design and manufacture drill bits and understand forces and associated cutting actions. This can help improve drilling efficiency of drill bits and drilling systems.

[0038] Figure 3 is a representation of a ML validation and confirmation system 312 in accordance with at least one embodiment of the present disclosure. Each of the components of the ML validation and confirmation system 312 can include software, hardware, or both. For example, a component can include one or more instructions stored on a computer-readable storage medium and executable by a processor of one or more computing devices, such as a client device or a server device. The computer-executable instructions of the ML validation and confirmation system 312, when executed by the one or more processors, can cause the computing device to perform the methods described herein. Alternatively, a component can include hardware, such as a specialized processing device for performing a certain function or group of functions. Alternatively, a component of the ML validation and confirmation system 312 can include a combination of computer-executable instructions and hardware.

[0039] Further, the components of the ML validation and confirmation system 312 can be implemented, for example, as one or more operating systems, one or more standalone applications, one or more modules of an application, one or more plug-ins, one or more library functions, or functions callable by other applications, and / or a cloud computing model. Thus, a component can be implemented as a standalone application, such as a desktop or mobile application. Further, a component can be implemented as one or more web-based applications hosted on a remote server. A component can also be implemented in a suite of mobile device applications or “apps.”

[0040] The ML validation and verification system 312 includes a ML model 314. The ML model 314 can be trained to generate synthetic rock files when applied to a target configuration of a cutting element. As discussed herein, the target configuration can be the same, similar, or related to an original configuration of an original rock file. In some examples, the target configuration can be different from the original configuration. The ML model 314 can be trained in any manner. For example, the ML model 314 can be trained using a deep learning training system 332. The deep learning training system 332 can train the ML model 314 by receiving original rock files as input and identifying connections and / or comparisons between elements in the original rock files. In this manner, the deep learning training system 332 can train the ML model 314 to generate synthetic rock files using the identified connections between the original rock files.

[0041] In some embodiments, the ML model 314 can be trained using a transfer learning training system 334. The transfer learning training system 334 can train the ML model 314 to generate synthetic rock files using different data. Transfer learning can include transferring at least a portion of a first model trained based on a first set of data. The transferred portion can be used as a starting point for the ML model 314. The ML model 314 can then be trained for a second set of data that is different from the data initially used to train the first model. This can allow the ML model 314 to generate synthetic rock files for cutting elements having different geometries and / or shapes from the cutting elements used to generate the original rock files.

[0042] As a non-limiting example, a first ML model can be trained using original rock files generated with cylindrical cutting elements. The first ML model, or at least a portion of the first ML model, can be used as a starting point for the ML model 314. The transfer learning training system 334 can train the ML model 314 to generate synthetic rock files for cutting elements having different shapes, such as non-cylindrical, including conical, wedge-shaped, biased wedge-shaped, biased conical, convex, concave, any other shape, and combinations thereof. In some embodiments, the ML model 314 can be retrained using the validation engine 320 or other retraining mechanism to further improve and validate the ML model 314.

[0043] The ML validation and verification system 312 includes a validation engine 320. The validation engine 320 can validate the ML model 314 to help the generated synthetic rock files represent actual forces experienced by a physical cutting element. This can help improve the accuracy and / or reliability of the ML model 314.

[0044] The validation engine 320 can validate the ML model 314 in any manner. For example, as discussed herein, the validation engine 320 can validate the ML model 314 using cross-validation 336. As discussed herein, the cross-validation 336 can include generating a plurality of training original rock file subsets and validation original rock file subsets. The ML model 314 can be trained based on each of the training original rock files and generate an associated synthetic rock file. The synthetic rock files can be compared to the validation original rock files to determine how similar the synthetic rock files are to the validation original rock files. If the synthetic rock files are not within a cross-validation threshold ratio, the ML model 314 can be retrained using the results of the cross-validation 336.

[0045] In some embodiments, the cross-validation 336 can include a 90-10 cross-validation, where 10% of the original rock files are reserved for validation and 90% of the original rock files are used to train the ML model 314. This can allow for 10 independent instances or implementations of the ML model 314. In some embodiments, the cross-validation 336 can include any cross-validation ratio, such as 50-50, 60-40, 70-30, 80-20, 90-10, 95-5, 99-1, or any ratio in between.

[0046] In some embodiments, the validation engine 320 can validate the ML model 314 using a consistency check 338. The consistency check 338 can include generating synthetic rock files for a plurality of implementations of the ML model 314. The implementations of the ML model 314 can be versions of the ML model 314 trained using separate training subsets of the original rock files. The different implementations of the ML model 314 can generate synthetic rock files for the same target configuration of the cutting element. The consistency check 338 of the validation engine 320 can compare the generated synthetic rock files and generate a consistency ratio to determine the variation in the synthetic rock files. If the consistency ratio is below a consistency ratio threshold, the consistency check 338 of the validation engine 320 can result in the ML model 314 being retrained to reduce the variation in the synthetic rock files. If the consistency ratio is above the consistency ratio threshold, the consistency check 338 of the validation engine 320 can determine that the ML model 314 is at least partially validated.

[0047] The verification engine 320 can further verify the ML model 314 using a physical simulation 340. The physical simulation 340 can simulate a drill bit or other cutting structure using cutting information for a synthetic rock file for cutting elements that can be used on the drill bit (e.g., a synthetic physical simulation). The physical simulation 340 can produce a drill bit performance representing forces that the drill bit can experience. In some embodiments, the physical simulation 340 can be performed on a synthetic rock file and an original rock file with the same cutting element configuration (e.g., an original physical simulation). This can allow the verification engine 320 to compare drill bit performance between the synthetic rock file and the original rock file. The verification engine 320 can generate a simulation comparison ratio between the synthetic simulation and the original simulation. If the simulation comparison ratio is below a simulation comparison ratio threshold, the verification engine 320 can cause the ML model 314 to be retrained to reach the simulation comparison ratio threshold. If the simulation comparison ratio is above the simulation comparison ratio threshold, the physical simulation 340 of the verification engine 320 can determine that the ML model 314 is at least partially verified. In some embodiments, the verification engine 320 can generate a performance comparison ratio for each individual drill bit performance.

[0048] In some embodiments, the verification engine 320 can perform the physical simulation 340 (one or more of a synthetic physical simulation or an original physical simulation) using a plurality of ROPs in a ROP control mode. For example, the verification engine 320 can perform the physical simulation 340 using ROPs including 1 foot per hour (ft / hr.), 5 ft / hr., 10 ft / hr., 25 ft / hr., 50 ft / hr., 100 ft / hr., 250 ft / hr., 500 ft / hr., or any value between these values. The verification engine 320 can compare drill bit performance at different ROPs to determine performance of the synthetic rock file at various ROPs. In some embodiments, the verification engine 320 can compare drill bit performance at different ROPs to determine in which areas cutting forces of the synthetic rock file can be improved. In this way, the verification engine 320 can retrain the ML model 314 to further improve the synthetic rock file.

[0049] In some embodiments, the verification engine 320 can perform the physical simulation 340 using different cutting element configuration arrangements, including different configurations at different ROPs. In this way, the ML model 314 can help represent stable and / or efficient drill bit configurations.

[0050] In some embodiments, the validation engine 320 can perform a bit sensitivity analysis using the synthetic rock file to perform the physical simulation 340. For example, the validation engine 320 can perform the physical simulation 340 using random variations or noise in the synthetic rock file. In this way, the ML validation and confirmation system 312 can identify sensitivity of the bit to variations in the synthetic rock file. In some embodiments, the ML validation and confirmation system 312 can identify sensitivity of the synthetic rock file to configuration variations.

[0051] Figure 4 is a representation of a cross-validation system 442 in accordance with at least one embodiment of the disclosure. The cross-validation system 442 includes an ML model 414 that is applied to a target configuration of cutting elements. The ML model 414 can generate one or more synthetic rock files 418 for the target configuration. The synthetic rock files 418 can be analyzed by a rock file manager 444. The rock file manager 444 can receive a validation original rock file 446. The validation original rock file 446 can be an original rock file having the same configuration as the synthetic rock files 418 (e.g., the target configuration).

[0052] The rock file manager 444 can compare the synthetic rock files 418 to the validation original rock file 446 to determine a cross-validation ratio of one or more cutting forces of the synthetic rock files 418 (e.g., a force in the synthetic rock file 418 divided by a corresponding force in the validation original rock file 446). The cross-validation ratio can include any other comparison of forces or other values of the synthetic rock files 418, such as mechanical specific energy (MSE) or other comparisons. If the cross-validation ratio is not within a cross-validation ratio threshold range, the ML adjustment 428 can use the difference between the synthetic rock files 418 and the validation original rock file 446 to adjust (e.g., retrain) the ML model 414. In some embodiments, a lower boundary of the cross-validation threshold range can be within a range having an upper limit value, a lower limit value, or an upper limit value and a lower limit value, including any of 80%, 90%, 95%, 97%, 98%, 99%, 99.5%, or any value therebetween. In some embodiments, an upper boundary of the cross-validation threshold range can be within a range having an upper limit value, a lower limit value, or an upper limit value and a lower limit value, including any of 100.5%, 101%, 102%, 103%, 105%, 110%, 120%, or any value therebetween. In some embodiments, the cross-validation threshold range can be any combination of the lower boundary and the upper boundary. For example, the cross-validation threshold range can be between 99.5% and 100.5%. In some examples, the cross-validation threshold range can be between 99% and 100%. In some examples, the cross-validation threshold range can be between 98% and 102%. In some examples, the cross-validation threshold range can be any combination of the boundaries discussed herein.

[0053] In some embodiments, if the synthetic rock file 418 or the cut forces of the synthetic rock file 418 are determined by the rock file manager 444 to be within a cross-validation threshold range, the rock file manager 444 can validate or at least partially validate the ML model 414. In this way, the cross-validation system 442 can help validate the ML model 414 using validation original rock files 446 that have the same configuration as the synthetic rock file 418. This can help improve the accuracy and / or representation of the ML model 414 to physical cutting elements.

[0054] Figure 5 is a representation of a consistency check validation system 548 in accordance with at least one embodiment of the present disclosure. The consistency check validation system 548 includes a ML model 514. The ML model 514 can include a plurality of ML model implementations (collectively 550). Different ML model implementations 550 can be generated using different training data sets.

[0055] In some embodiments, the ML model implementations 550 can generate a set of validation synthetic rock files (collectively 552). The validation synthetic rock files 552 can all be generated using the same cutting element configuration. A consistency manager 554 can review the validation synthetic rock files to determine a consistency rate between two or more of the validation synthetic rock files 552.

[0056] For example, a first ML model implementation 550-1 can generate a first validation synthetic rock file 552-1. A second model implementation 550-2 can generate a second validation synthetic rock file 552-2. The consistency check validation system 548 can include n ML model implementations 550, where an nth ML model implementation 550-n generates an nth validation synthetic rock file 552-n.

[0057] The consistency manager 554 can generate a consistency rate between the validation synthetic rock files 552. For example, the consistency manager 554 can generate a consistency rate between the first validation synthetic rock file 552-1 and the second validation synthetic rock file 552-2. In some examples, the consistency manager 554 can generate a consistency rate between the first validation synthetic rock file 552-1 and the nth validation synthetic rock file 552-n. In some embodiments, the consistency manager 554 can generate a consistency rate for every combination of the validation synthetic rock files 552.

[0058] The consistency manager 554 can determine whether the consistency ratio is greater than a consistency ratio threshold. In some embodiments, the consistency ratio threshold can be in a range having an upper value, a lower value, or an upper value and a lower value, including any of 80%, 90%, 95%, 97%, 98%, 99%, 99.5%, 100.5%, 101%, 102%, 103%, 105%, 110%, 120%, or any value therebetween. For example, the consistency ratio threshold can be greater than 80%. In another example, the consistency ratio threshold can be less than 120%. In yet other examples, the consistency ratio threshold can be any value in a range between 80% and 120%. In some embodiments, a consistency ratio threshold greater than 98% can be critical to improving validation of the ML model 514.

[0059] If the consistency ratio is less than the consistency ratio threshold, the consistency manager 554 can cause the ML model 514 to undergo retraining to improve consistency of the synthetic rock files. For example, the consistency manager 554 can identify which portion of the synthetic rock files caused the disparity in the consistency ratio. The consistency manager 554 can direct training of the ML model 514 to improve consistency of the ML model 514.

[0060] Figure 6 is a representation of a physical comparison validation system 656 in accordance with at least one embodiment of the present disclosure. The physical comparison validation system 656 includes a ML model 614. The ML model 614 can be applied to a target configuration of a cutting element to generate one or more synthetic rock files 618.

[0061] The synthetic rock file parser 658 can receive the synthetic rock files 618 and apply more synthetic rock files 618 to a physical model 640. The synthetic rock file parser 658 can receive a bit design of the physical model 640 from a physical model library 660. For example, the physical model library 660 can include one or more bit or other cutting structure designs. A particular bit design can include a combination of cutting element configurations. The physical model 640 can analyze performance of a bit having cutting elements arranged in accordance with the bit design or forces experienced by the bit. The cutting element configurations can be modeled by the synthetic rock files to generate bit performance.

[0062] In some embodiments, the physical comparison validation system 656 can use the original rock files 616 having the same configuration as the synthetic rock files to generate bit performance. The physical comparison manager 662 can compare the original bit performance from the original rock files 616 to the synthetic bit performance from the synthetic rock files 618 to generate a physical comparison ratio. If the physical comparison ratio is within a physical comparison ratio threshold range, the physical comparison manager 662 can determine that the ML model 614 is at least partially validated. If the physical comparison ratio is not within the physical comparison ratio threshold range, the physical comparison manager 662 can cause the ML model 614 to be further trained or retrained to improve the representativeness of the synthetic rock files 618 to physical cutting elements.

[0063] In some embodiments, a lower boundary of the physical comparison ratio threshold range can be within a range having an upper limit value, a lower limit value, or both, including any of 80%, 90%, 95%, 97%, 98%, 99%, 99.5%, or any value therebetween. In some embodiments, an upper boundary of the physical comparison ratio threshold range can be within a range having an upper limit value, a lower limit value, or both, including any of 100.5%, 101%, 102%, 103%, 105%, 110%, 120%, or any value therebetween. In some embodiments, the physical comparison ratio threshold range can be any combination of the lower boundary and the upper boundary. For example, the physical comparison ratio threshold range can be between 99.5% and 100.5%. In some examples, the physical comparison ratio threshold range can be between 99% and 100%. In some examples, the physical comparison ratio threshold range can be between 98% and 102%. In some examples, the physical comparison ratio threshold range can be any combination of the boundaries discussed herein.

[0064] In some embodiments, the physical model 640 can generate bit performance including one or more of unbalanced force, radial unbalanced force, circumferential unbalanced force, roll unbalanced force, WOB, bit torque, any other bit force, and combinations thereof. The physical comparison manager 662 can generate a physical comparison ratio for each of the bit performance. If any of the bit performance is outside of a physical comparison ratio threshold range, the physical comparison manager 662 can cause the ML model 614 to be retrained based on that bit performance. In some embodiments, WOB can be more relevant than F v and the physical comparison manager 662 can cause the ML model 614 to be retrained to improve F v unbalanced force can be more relevant than F 侧More relevant, and if the bit torque physical comparison ratio is outside of an associated threshold range, the physical comparison manager 662 can cause the ML model 614 to be retrained to improve F 侧 . The bit torque force can be more relevant, and if the bit torque physical comparison ratio is outside of an associated threshold range, the physical comparison manager 662 can cause the ML model 614 to be retrained to improve F 切削 . 切削 .

[0065] In some embodiments, the physical comparison validation system 656 can execute multiple physical models 640 in different configurations, as discussed herein. For example, the physical comparison validation system 656 can execute multiple physical models 640 for different ROPs in an ROP control mode. This can allow the physical comparison manager 662 to determine the impact of the synthetic rock file on bit performance at various ROPs and retrain the ML model 614 accordingly. In some examples, the physical comparison validation system 656 can execute multiple physical models 640 for statistical variations of the configuration of the synthetic rock file 618. This can help determine the sensitivity of the bit and / or bit performance to noise in the synthetic rock file 618.

[0066] Figure 7 is a representation of a radar chart 764 generated from comparing physical simulations of a bit according to the present disclosure. The radar chart 764 identifies six bit performance comparisons generated based on the physical simulation results, including unbalanced force 766, radial unbalanced force 768, circumferential unbalanced force 770, side- tilt unbalanced force 772, WOB 774, and bit torque 776.

[0067] The radar chart 764 can visually identify the ratio of each of the bit performances, where a midpoint between the center and the edge of the radar chart 764 indicates a ratio of 1 : 1, i.e., 100%, indicating that the compared bit performances are the same. The radar chart 764 can include multiple lines located on the radar chart 764, where each line indicates a different application of a physical model. As discussed herein, different applications of a physical model can have different parameters. For example, different applications of a physical model can include a synthetic physical model applied to a bit modeled with a synthetic rock file, and an original physical model applied to a bit modeled with an original rock file having the same configuration. This can allow a drilling tool operator and / or a drilling tool planner to determine the difference between the synthetic rock file and the original rock file. The bit performances and / or other information identified or inferred from the radar chart 764 can be used to improve the ML model.

[0068] In some examples, different lines on the radar chart 764 can indicate bits modeled at different ROPs in the ROP control mode. This can provide the drill tooling personnel and / or the drill tooling planner with information about bit stability at different ROPs, and / or information about the synthetic rock file representation of the physical cutting elements at various ROPs. In some examples, different lines on the radar chart 764 can indicate bits modeled with noise introduced to the synthetic rock file. This can provide the drill tooling personnel and / or the drill tooling planner with information about the sensitivity of the bit to various parameters.

[0069] Figure 8 is a representation of a box plot 878 generated from a comparison of physical simulations of bits according to at least one embodiment of the disclosure, the box plot indicating variation in different applications of the physical model. The box plot 878 identifies six bit performances modeled by the physical model, including unbalanced force 866, radial unbalanced force 868, circumferential unbalanced force 870, roll unbalanced force 872, WOB 874, and bit torque 876.

[0070] The box plot 878 can identify statistical variation in different applications of the physical model, including indications of mean, standard deviation, and outliers. The drill tooling personnel and / or the drill tooling planner can review the box plot 878 to determine variation in various bits modeled using the synthetic rock file. In some embodiments, the box plot 878 can be generated using a ratio between two applications, such as a ratio between a bit generated using the synthetic rock file and a bit generated from the original rock file. In some embodiments, the box plot 878 can be generated using variation in the input synthetic rock file to identify sensitivity of the bit to variation in the synthetic rock file.

[0071] Figure 9 is a representation of rock file plots 980 of an original rock file plot 980-1 and a synthetic rock file plot 980-2 according to at least one embodiment of the disclosure. The rock file plots 980 include cutting lines 982 of cutting elements of different sizes with the same configuration (e.g., back rake angle, side rake angle, and confining pressure), where the lines 982 are plotted with cutting depth 984 on the horizontal axis (x-axis) and cutting force 986 on the vertical axis (y-axis).

[0072] It can be seen that the original cutting lines 982-1 of the original rock file plot 980-1 and the synthetic cutting lines 982-2 of the synthetic rock file plot 980-2 can be similar. This can indicate that the synthetic rock file represents the physical cutting elements represented by the original rock file.

[0073] In some implementations, synthetic rock files can "correct" for non-representative data in the original rock files. For example, the original rock files for one or more diameters of the cutting element may include errors due to experimental mistakes. Synthetic rock files generated from validated ML models can produce synthetic cutting lines 982-2 that better represent the actual physical state of the cutting element. As a particular, non-limiting example, in the original rock file figure 980-1, the force 986 of the original 9 mm cutting element line 987-1 may be higher than that of the original 11 mm cutting element line 988-1, even if larger cutting elements have higher forces due to their larger size. In the synthetic rock file figure 980-2, the force of the synthetic 9 mm cutting element line 987-2 may be lower than that of the synthetic 11 mm cutting element line 988-2. This can indicate that the synthetic rock file "corrects" the original rock file or generates forces that do not include any experimental errors.

[0074] According to at least one embodiment of this disclosure, the rock file library of drilling operators can be supplemented and / or at least partially replaced with synthetic rock files. This can result in the rock file library being less affected by experimental errors, thereby improving the quality and / or completeness of the rock file library. In some embodiments, the rock file library can be reliable, thereby improving simulation results based on the use of the rock file library.

[0075] Figure 10 and Figure 11 The corresponding text and examples provide various methods, systems, apparatuses, and computer-readable media for ML verification and validation systems. In addition to the foregoing, one or more implementation schemes can be described using flowcharts including actions for achieving specific results, such as... Figure 10 and Figure 11 As shown. Figure 10 and Figure 11 It can be performed with more or fewer actions. Furthermore, actions can be performed in different orders. Additionally, the actions described herein can be performed repeatedly or in parallel with each other, or in parallel with different instances of the same or similar actions.

[0076] As mentioned, Figure 10 A flowchart illustrating a series of actions 1089 for validating and validating an ML model according to one or more implementation schemes is shown. Although Figure 10 The actions according to one implementation are shown, but alternative implementations may omit, add, reorder, and / or modify them. Figure 10 Any action shown. Figure 10 The actions shown can be performed as part of a method. Alternatively, a computer-readable medium may include instructions that, when executed by one or more processors, cause a computing device to perform... Figure 10the actions of FIG. 10. In some embodiments, the system can perform the actions of Figure 10 the actions of FIG. 10.

[0077] At 1090, the ML validation and verification system can receive a target configuration for a cutting element. The target configuration can include an orientation of the cutting element, a shape of the cutting element, and a rock type. The target configuration can be related to an original configuration of an original rock file. At 1091, the ML validation and verification system can apply the ML model to the target configuration to generate a synthetic rock file for the cutting element. The synthetic rock file can include a cutting force associated with the target configuration.

[0078] At 1092, the ML validation and verification system can validate the ML model. At 1093, to validate the ML model, the validation engine can generate a consistency ratio for the synthetic rock file. The consistency ratio can be based on a validation synthetic rock file generated by the ML model, the validation synthetic rock file being generated by using a different implementation of the ML model with the target configuration. At 1094, the validation engine can perform a physical simulation using the synthetic rock file. In some embodiments, performing the physical simulation can include performing a synthetic physical simulation. In some embodiments, performing the physical simulation can include performing an original physical simulation. The validation engine can generate a simulation comparison ratio using the synthetic physical simulation and the original physical simulation.

[0079] In some embodiments, the ML model can generate a new synthetic rock file for a new configuration when at least one of: the simulation comparison ratio is greater than a simulation comparison ratio threshold or the consistency ratio threshold is greater than a consistency ratio threshold. The new configuration can be different from the target configuration and the original configuration. In some embodiments, the ML model can be trained based on the synthetic rock file when at least one of: the simulation comparison ratio is less than a simulation comparison ratio threshold or the consistency ratio threshold is less than a consistency ratio threshold. In some embodiments, retraining the ML model can include retraining the ML model until the simulation comparison ratio and / or the consistency ratio is greater than its respective threshold.

[0080] As mentioned, Figure 11 A flow diagram illustrating a series of actions 1100 for validating and verifying a ML model in accordance with one or more embodiments is shown. While Figure 11 actions are shown, alternative embodiments can omit, add to, reorder, and / or modify Figure 11 any of the actions shown. Figure 11 The actions shown can be performed as part of a method. Alternatively, a computer-readable medium can include instructions that, when executed by one or more processors, cause a computing device to perform the actions of Figure 11 the actions of FIG. 10. In some embodiments, the system can perform the actions of Figure 11the actions of the method.

[0081] At 1101, the ML validation and verification system can train a first implementation of a ML model of a cutting element. The ML model can be trained using a first training subset of a plurality of original rock files. At 1102, the ML validation and verification system can apply the first implementation of the ML model to a target configuration of the cutting element to generate a first synthetic rock file. The target configuration can match an original configuration of a first validation subset of the plurality of original rock files.

[0082] At 1103, the ML validation and verification system can train a second implementation of the ML model using a second training subset of the plurality of original rock files. At 1104, the ML validation and verification system can apply the second implementation of the ML model to a target configuration of the cutting element to generate a second synthetic rock file. The second target configuration can match an original configuration of a second validation subset of the plurality of original rock files.

[0083] At 1105, the ML validation and verification system can validate the ML model. At 1106, to validate the ML model, the validation engine can tune hyperparameters of the ML model. The tuning can be based on a first cross-validation ratio between the first synthetic rock file and the first validation subset and a second cross-validation ratio between the second synthetic rock file and the second validation subset. At 1107, the validation engine can generate a consistency ratio between the first synthetic rock file and the second synthetic rock file for the ML model.

[0084] At 1108, the validation engine can perform a first synthetic physical simulation on the first synthetic rock file and a second synthetic physical simulation on the second synthetic rock file using a physical model. At 1109, the validation engine can perform a first original physical simulation on the first validation subset and a second original physical simulation on the second validation subset using the physical model. At 1110, the validation engine can generate a simulation comparison ratio using the first synthetic physical simulation, the second synthetic physical simulation, the first original physical simulation, and the second original physical simulation.

[0085] Figure 12 Certain components can be included in the computer system 1200, which is illustrated in FIG. 12. One or more computer systems 1200 can be used to implement various apparatuses, components, and systems described herein.

[0086] The computer system 1200 includes processors 1201. Processors 1201 can be general- purpose single- or multi-chip microprocessors (e.g., Advanced RISC (RISC) machines (ARM)), special-purpose microprocessors (e.g., digital signal processors (DSPs)), microcontrollers, programmable gate arrays (PGAs), or the like. The processors 1201 can be referred to as central processing units (CPUs). Although illustrated as a single processor, the processors 1201 can comprise multiple processors. Although illustrated in a single computing device, the processors 1201 can be distributed across multiple computing devices.Figure 12 Only a single processor 1201 is shown in the computer system 1200, but in an alternative configuration, a combination of processors (e.g., an ARM and a DSP) could be used.

[0087] The computer system 1200 also includes memory 1203 in electronic communication with the processor 1201. The memory 1203 can be any electronic, magnetic, optical, or other physical storage device that can store electronic information. For example, the memory 1203 can be embodied as random access memory (RAM), read only memory (ROM), magnetic disk storage media, optical storage media, flash memory devices in RAM, on-board memory contained in the processor, erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), registers, and the like, including combinations of the aforementioned types of memory.

[0088] The instructions 1205 and the data 1207 can be stored in the memory 1203. The instructions 1205 can be executable by the processor 1201 to implement some or all of the functionality disclosed herein. Executing the instructions 1205 can involve the use of the data 1207 that is stored in the memory 1203. Any of the various examples of modules and means described herein can be implemented, partially or entirely, as instructions 1205 stored in memory 1203 and executed by the processor 1201. Any of the various examples of data described herein can be data 1207 stored in memory 1203 and used during execution of the instructions 1205 by the processor 1201.

[0089] The computer system 1200 can also include one or more communication interfaces 1209 for communicating with other electronic devices. The communication interface 1209 can be based on wired communication technology, wireless communication technology, or both. Some examples of communication interfaces 1209 include a Universal Serial Bus (USB) interface, an Ethernet interface, a wireless interface operating according to a Institute of Electrical and Electronics Engineers (IEEE) 802.11 wireless communication protocol, a Bluetooth® wireless interface, a ZigBee® wireless interface, or the like, including combinations of the aforementioned types of communication interfaces. ® Wireless communication adapters and Infrared (IR) communication ports.

[0090] Computer system 1200 can also include one or more input device(s) 1211, and one or more output device(s) 1213. Some examples of input device(s) 1211 include keyboard, mouse, microphone, remote control device, button(s), joystick, trackball, touchpad, and lightpen. Some examples of output device(s) 1213 include speaker(s) and printer. One specific type of output device that is often included in computer system 1200 is a display device 1215. Display device 1215 utilized with the implementations disclosed herein can utilize any suitable image projection technology, such as liquid crystal display (LCD), light-emitting diode (LED), gas plasma, electroluminescence, and the like. A display controller 1217 can also be provided, which is utilized to transform data 1207 stored in storage 1203 into text, graphics, and / or moving images (as appropriate) shown on display device 1215.

[0091] The various components of computer system 1200 can be coupled together by one or more buses, which can include a power bus, a control signal bus, a status signal bus, a data bus, etc. For the sake of clarity, not all Figure 12 In general, bus system 1219 is used for communication between the various internal and external components of computer system 1200.

[0092] Embodiments of the ML validation and verification system are described primarily with reference to wellbore drilling operations. However, it is important to note that the ML validation and verification system described herein can be used for applications other than drilling a wellbore. In other embodiments, the ML validation and verification system according to the present disclosure can be used externally of a wellbore or other downhole environment used to explore or produce natural resources. For example, the ML validation and verification system of the present disclosure can be used in boreholes used to place utility lines. Accordingly, the terms “wellbore,” “borehole,” and the like should not be interpreted to limit the tools, systems, assemblies, or methods of the present disclosure to any particular industry, field, or environment.

[0093] One or more specific embodiments of the present disclosure are described herein. These described embodiments are examples of the technologies disclosed herein. Additionally, it is to be understood that various alterations, modifications, and improvements can be made to the disclosed embodiments, and additional implementations can be made, without departing from the scope of the present disclosure. Further, it is to be understood that such alterations, modifications, and improvements are to be considered equivalents of the technologies disclosed herein, and are to be included within the scope of the present disclosure. Also, it is to be understood that one or more specific embodiments of the present disclosure can be combined with one or more other specific embodiments of the present disclosure.

[0094] Additionally, it is to be appreciated that the use of any of the present disclosure, or separate embodiments, is not intended to exclude the addition of another such embodiment. For example, any element of an embodiment described herein can be combined with any element of any other embodiment described herein. As will be appreciated by one of ordinary skill in the art upon reading this disclosure, the numerical, percentage, ratio or other values recited herein are intended to encompass both the recited value and also other values "about” or "approximately” the recited value. Thus, the recited values are to be construed broadly enough to encompass values at least close enough to the recited value to perform the intended function or achieve the intended result. The recited values include at least variations expected in a suitable manufacturing or production process, and can include values within 5%, within 1%, within 0.1%, or within 0.01% of the recited value.

[0095] In light of the present disclosure, those of ordinary skill in the art will appreciate that the various embodiments disclosed herein are susceptible to various modifications, substitutions, and alterations without departing from the spirit and scope of the disclosure. Equivalents (including functional equivalents) are intended to be encompassed by the description and claims as if individually recited herein. Applicant’s intent is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the claims. Each of the additions, deletions, and modifications of the embodiments falling within the meaning and range of equivalents of the claims is to be embraced by the claims.

[0096] The terms “about,” “approximately,” and “substantially” as used herein represent an amount that is within standard manufacturing or process tolerances or an amount that is close enough to the stated amount to perform the intended function or achieve the intended result. For example, the terms “about,” “approximately,” and “substantially” can refer to an amount that is within less than 5% of the stated amount, within less than 1% of the stated amount, within less than 0.1% of the stated amount, and within less than 0.01% of the stated amount. Furthermore, it should be understood that any directional or reference system in the foregoing description is merely a relative direction or movement. For example, any reference to “up” and “down” or “above” or “below” merely describes the relative position or movement of the relevant elements.

[0097] The present disclosure can be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The described implementations are to be considered in all respects only as illustrative and not restrictive. The scope of the disclosure is, therefore, indicated by the appended claims, rather than by the foregoing description. Changes in meaning and equivalents within the meaning and range of equivalents of the claims are to be embraced within the scope of the claims.

Claims

1. A method comprising: Receive the target configuration of the cutting element, which is related to the original configuration of the original rock file; A machine learning (ML) model is applied to the target configuration to generate a synthetic rock file for the cutting element, the synthetic rock file including the cutting forces associated with the target configuration; and Verify the ML model using the following steps: A consistency ratio is generated for the synthetic rock files based on the validation synthetic rock files generated by the ML model, wherein the validation synthetic rock files are generated using different implementations of the ML model with the target configuration; and Physical simulations were performed using the synthetic rock file and the original rock file.

2. The method of claim 1, wherein performing the physical simulation includes performing a synthetic physical simulation using the synthetic rock file and performing a raw physical simulation using the original rock file, and further includes generating a simulation comparison ratio using the synthetic physical simulation and the original physical simulation.

3. The method of claim 2, further comprising applying the ML model to a new configuration to generate a new synthetic rock file when at least one of the following exists: the simulation comparison ratio is greater than a simulation comparison ratio threshold or the consistency ratio is greater than a consistency ratio threshold, and the new configuration differs from the target configuration and the original configuration in at least one aspect of orientation, shape, or rock type.

4. The method of claim 2, further comprising training the ML model based on the synthetic rock file when at least one of the following exists: the simulation comparison ratio is less than a simulation comparison ratio threshold or the consistency ratio is less than a consistency ratio threshold, wherein performing the synthetic physics simulation includes performing the synthetic physics simulation on the synthetic rock file arranged in the drill bit configuration, and performing the original physics simulation includes performing the original simulation on the original rock file arranged in the drill bit configuration.

5. The method of claim 4, wherein training the ML model comprises training the ML model until at least one of the following exists: the simulation comparison ratio is greater than a validation threshold or the consistency ratio is less than a consistency ratio threshold.

6. The method of claim 4, wherein performing the synthetic physics simulation includes configuring the drill bit to generate synthetic drill bit performance, and performing the original physics simulation includes configuring the drill bit to generate original drill bit performance, and generating the simulation comparison ratio includes generating a performance ratio for each of the synthetic drill bit performances based on the original drill bit performance.

7. The method of claim 6, wherein performing the synthetic physics simulation includes performing the synthetic physics simulation for a plurality of penetration rates (ROPs) and generating the synthetic drill bit performance for each of the plurality of ROPs, wherein performing the raw physics simulation includes performing the raw physics simulation for the plurality of ROPs and generating the raw drill bit performance using the raw drill bit performance for each of the plurality of ROPs.

8. A method comprising: A first implementation of training a machine learning (ML) model for cutting elements using a first training subset of multiple raw rock files; The first implementation of the ML model is applied to the target configuration of the cutting element to generate a first synthetic rock file, the target configuration being matched with the original configuration of a first verification subset of the plurality of original rock files; A second implementation of training the ML model using a second training subset of the plurality of original rock files; The second implementation of the ML model is applied to the target configuration of the cutting element to generate a second synthetic rock file, the second target configuration being matched with the original configuration of a second verification subset of the plurality of original rock files; as well as Validating the ML model includes: The hyperparameters of the ML model are adjusted based on the first cross-validation ratio between the first synthetic rock file and the first validation subset and the second cross-validation ratio between the second synthetic rock file and the second validation subset. Generate a consistency ratio between the first synthetic rock file and the second synthetic rock file for the ML model; A first synthetic physics simulation is performed on the first synthetic rock file using a physical model, and a second synthetic physics simulation is performed on the second synthetic rock file. A first primitive physics simulation is performed on the first verification subset using the physical model, and a second primitive physics simulation is performed on the second verification subset; and The simulation comparison ratio is generated using the first synthetic physics simulation, the second synthetic physics simulation, the first original physics simulation, and the second original physics simulation.

9. The method of claim 8, further comprising applying the ML model to a new configuration of the cutting element when the consistency ratio is higher than a consistency ratio threshold and the simulation comparison ratio is higher than a simulation comparison ratio threshold.

10. The method of claim 8, wherein the first implementation of training the ML model and the second implementation of the ML model include training using at least one of the following: transfer learning and deep learning.

Images