Drilling simulator-oriented intelligent teaching assistant system base LLM training method

Abstract

The application provides a drilling simulator-oriented intelligent teaching assistant system base LLM training method, and belongs to the technical field of large language model training of oil drilling, and the method comprises the following steps: based on the intelligent teaching assistant system base large language model, drilling simulator simulation data is acquired, and field self-adaption pre-training is carried out; low-rank self-adaption is utilized to fine-tune the original weight, and operation steps and fault diagnosis generation are carried out; by constructing an enterprise private knowledge base, a large language model is trained by using retrieval enhancement, and a trained verification head is obtained through explicit supervision; a direct preference optimization loss function is defined, a retrieval-aware direct preference optimization loss function is obtained by combining the verification head, a large language model is optimized, and the large language model training is completed; the intelligent assistant framework capable of fusing external professional knowledge base, understanding equipment operation mechanism and having teaching guidance and error diagnosis capability is obtained, and the intelligent level of the drilling simulator is improved.

Description

Technical Field

[0001] This invention belongs to the field of large language model training technology for oil drilling, and particularly relates to an LLM training method for an intelligent teaching assistant system for drilling simulators. Background Technology

[0002] Oil drilling operations are high-risk, high-tech intensive engineering activities involving complex mechanical equipment (such as top drives, mud pumps, blowout preventers, etc.), multi-condition coordinated operation, and strict well control procedures. To ensure on-site safety and efficiency, operators must undergo rigorous training. Traditional training relies on master-apprentice training or simulator training based on fixed scripts, which suffers from high costs, limited scenarios, and delayed feedback.

[0003] In recent years, with the development of artificial intelligence (AI), especially large language models (LMs), intelligent assisted training systems have been gradually introduced into the field of industrial training. Among them, simulation training platforms based on multi-agent systems (MAS) have become an important development direction for the next generation of drilling simulators due to their advantages such as task decomposition, role collaboration, and dynamic response.

[0004] However, while general-purpose LLMs possess powerful natural language understanding and generation capabilities, their training corpora primarily originate from publicly available internet texts, lacking the ability to model tacit knowledge such as physical rules, equipment logic, and operating procedures in the oil drilling field. Furthermore, a large amount of unstructured / semi-structured technical documents accumulated within enterprises (such as operation manuals, accident cases, and expert experience) are not included in the original LLM training data. This results in the model's inability to accurately understand the professional and contextualized questions raised by users during simulation training (e.g., "Why does the torque suddenly increase during tripping?" and "How to handle abnormal pressure during cementing and mud removal?").

[0005] Therefore, there is an urgent need for an intelligent assistant architecture that can integrate external professional knowledge bases, understand equipment operation mechanisms, and have teaching guidance and error diagnosis capabilities to improve the intelligence level of drilling simulators. Summary of the Invention

[0006] To address the aforementioned shortcomings in existing technologies, this invention provides a base LLM training method for an intelligent teaching assistant system for drilling simulators. This method solves the technical problems of knowledge gaps, logical breaks, frequent illusions, and weak pedagogical effectiveness in the base LLM of existing intelligent teaching assistant systems for oil drilling platform simulation training.

[0007] To achieve the above objectives, the technical solution adopted by this invention is: an LLM training method for an intelligent teaching assistant system base for drilling simulators, comprising the following steps:

[0008] S1. Based on the intelligent teaching assistant system's base large language model, obtain drilling simulator simulation data, and perform domain-adaptive pre-training on the large language model to obtain a physically-based large language model.

[0009] S2. Obtain training data, use low-rank adaptation to fine-tune the original weights, and perform operation steps and fault diagnosis generation on the physically based large language model to obtain a domain semantically adapted large language model.

[0010] S3. Construct a private knowledge base for the enterprise and use retrieval enhancement to train a large language model that adapts to the semantics of the domain, thereby obtaining a large language model aligned with retrieval enhancement. Through explicit supervision, train the model to obtain a trained validation head.

[0011] S4. Obtain error and resolution data. Based on the retrieved and aligned large language model, define the standard direct preference optimization loss function and combine it with the trained validation head to obtain the retrieval-aware direct preference optimization loss function. Optimize the large language model to obtain the preference-optimized large language model, thus completing the training of the large language model.

[0012] The beneficial effects of this invention are as follows: This invention trains the large language model of the intelligent teaching assistant system for drilling simulators through four stages: domain adaptive pre-training, operation steps and fault diagnosis generation, retrieval enhancement alignment, training verification head, and constructing a retrieval-aware direct preference optimization loss function. These stages include laying the foundation of physical rules and engineering common sense, adapting drilling equipment and processes, aligning private knowledge retrieval consistency, and optimizing retrieval-aware direct preferences. This results in significantly enhanced domain mechanism understanding, alignment of document-based generation with engineering logic, joint optimization of the retrieval and generation modules, safe and standardized output habits with a teaching-friendly approach, and stable performance suitable for multi-agent simulation training.

[0013] Further, S1 includes the following steps:

[0014] S101. Based on the large language model of the intelligent teaching assistant system, simulation data of the drilling simulator is obtained according to the physical laws of fluid mechanics, rock mechanics and control theory involved in the drilling process.

[0015] S102. Based on the model parameters of the large language model and combined with the mask position set, the large language model is modeled using mixed language by calculating the mixed language loss function, thus obtaining the large language model modeled using mixed language.

[0016] S103. Based on the drilling simulator simulation data, obtain the initial state vector of drilling, and obtain the real evolution result through preset real simulation;

[0017] S104. Based on the linear regression head after the encoder of the large language model, the evolution loss function is obtained by calculating the predicted value of the state vector and combining it with the actual evolution results. The physical state evolution prediction is performed on the large language model modeled by mixed language to obtain the large language model predicted by physical state evolution.

[0018] S105. Based on the intervention actions and problem descriptions in the drilling simulator simulation data, and combined with the expected results annotated by experts, obtain the causal loss function;

[0019] S106. Using the causal loss function, perform causal intervention reasoning on the large language model modeled by mixed language to obtain the large language model based on physics.

[0020] Furthermore, the drilling simulator simulation data includes: ordinary text paragraphs for language modeling, structured triples containing initial conditions, intervention actions, and evolution results, as well as mathematical derivation chains;

[0021] The mathematical derivation chain is specifically a chain of premises, formulas, and conclusions.

[0022] Furthermore, the expression for the mixed language loss function is as follows:

[0023] ;

[0024] in, Represents the loss function for mixed languages. Represents the set of mask locations. Indicates that given model parameters and word position Under the condition that the next word in the sequence is exactly The probability, Represents probability. Indicates the masked number i Each word element, Indicates the current position in the sequence. i All previous morphemes, Indicates model parameters;

[0025] The expression for the state loss function is as follows:

[0026] ;

[0027] ;

[0028] in, Represents the state loss function. Represents the actual evolutionary outcome. Indicates prediction Current state This represents the post-linear regression head of the large language model encoder. Represents the initial state vector. express time;

[0029] The expression for the causal loss function is as follows:

[0030] ;

[0031] in, Represents the causal loss function. This indicates the expected results of expert annotation. This indicates a problem description. This indicates an intervention action.

[0032] The beneficial effects of the above-mentioned further solutions are as follows: By utilizing domain adaptive pre-training that includes hybrid language modeling, physical state evolution prediction, and causal intervention reasoning, this invention only fine-tunes the parameters of the transformer model at higher levels, avoiding damage to the underlying language capabilities, and improving the understanding of the basic physical laws of fluid mechanics, rock mechanics, and control theory involved in the drilling process by the large language model, enabling the large language model to master the scientific reasoning capabilities of state evolution, causal intervention, and dimensional consistency.

[0033] Furthermore, S2 includes the following steps:

[0034] S201. Based on the drilling equipment and standard operating procedures, obtain training data including operation planning data and fault diagnosis data;

[0035] S202. Using low-rank adaptive adjustment, the original weights in the physically-based large language model are fine-tuned to obtain updated weights.

[0036] S203. Based on the operation planning data, generate operation steps for the physically-based large language model to obtain the operation loss function;

[0037] S204. Based on the fault diagnosis data, generate fault diagnosis text for the physically-based large language model to obtain the fault diagnosis loss function.

[0038] S205. Using the updated weights, operational loss function, and fault diagnosis loss function, the physical-based large language model is adapted to the domain semantics to obtain the domain-adapted large language model.

[0039] Furthermore, the expression for the operational loss function is as follows:

[0040] ;

[0041] in, Represents the operational loss function. Indicates completion Number of operations required Represents probability. Indicates the first i The word element of the step operation, Indicate the goal, Indicates on-site restrictions, Indicates model parameters;

[0042] The expression for the fault diagnosis loss function is as follows:

[0043] ;

[0044] in, This represents the fault diagnosis loss function. This refers to a natural language fault analysis written by an engineer. Represents sensor time-series data, This indicates a description of the scene.

[0045] The beneficial effects of the above-mentioned further solutions are as follows: This invention adapts drilling equipment and processes through low-rank adaptive fine-tuning, operation step generation, and fault diagnosis text generation, enabling the large language model to understand the text description of equipment functions, operation logic, and abnormal phenomena.

[0046] Furthermore, step S3 includes the following steps:

[0047] S301. Parse portable document formats or standard operating procedures into text, generate embeddings using a bidirectional encoder representation method, and build an enterprise private knowledge base by storing them in a vector database.

[0048] S302. Based on the enterprise's private knowledge base, use a preset sampling method to obtain document fragments, obtain a set of search paragraphs, and concatenate the question with the document fragments to obtain the gold standard answer;

[0049] S303. Combine the question, the set of search paragraphs, and the gold standard answer to obtain the labeled triples;

[0050] S303. Based on the labeled triples, use retrieval enhancement generation, obtain the multi-document retrieval enhancement generation loss, and train the domain semantically adapted large language model to obtain the retrieval enhancement aligned large language model.

[0051] S304. Based on the gold standard answer and the average embedding of the retrieved paragraph set, the trained validation head is obtained by explicitly supervising whether the generated content is supported by the retrieval and by calculating the retrieval alignment loss.

[0052] Furthermore, the expression for the multi-document retrieval enhancement generation loss is as follows:

[0053] ;

[0054] in, This represents the enhanced generation loss for multi-document retrieval. This represents the gold standard answer. To indicate a problem, This indicates the search for a set of paragraphs. Indicates model parameters;

[0055] The expression for the retrieval alignment loss is as follows:

[0056] ;

[0057] ;

[0058] in, This represents the retrieval alignment loss. Represents the binary cross-entropy loss. This indicates the probability that the generated answer is supported by the retrieved evidence. Indicates an indicator function, This represents the weight vector of the verification header. Indicates the gold standard answer of express, Indicates the retrieval of paragraph sets Average embedding.

[0059] The beneficial effects of the above-mentioned further solutions are as follows: By introducing a private knowledge base of an enterprise and training a verification head, the present invention performs consistency alignment for private knowledge retrieval, enabling the generated content to be supported by retrieval and improving the ability of a large language model to align private documents.

[0060] Furthermore, step S4 includes the following steps:

[0061] S401. Based on the teaching needs of drilling training, use real error scenarios accumulated in drilling simulators and high-quality solutions written by engineers to obtain error and solution data.

[0062] S402. Based on the retrieved and enhanced aligned large language model, and combined with manually labeled preference pairs, define the standard direct preference optimization loss function;

[0063] S403. Based on the trained verification head, obtain the support score, and define the evidence alignment regularization term according to the preset support threshold.

[0064] S404. Using the evidence alignment regularization term, a retrieval enhancement generation alignment mechanism is introduced into the standard direct preference optimization loss function to obtain the direct preference optimization loss function.

[0065] S405. The trained validation head is used as the dynamic weight of the direct preference optimization loss function to obtain the retrieval-aware direct preference optimization loss function;

[0066] S406. Optimize the large language model by using the direct preference optimization loss function based on retrieval perception to obtain a preference-optimized large language model.

[0067] S407. Using the large language model optimized by preference, obtain the model output, score the model output according to the preset dimension, and add the sample pairs in the model output that are greater than or equal to the preset score value to the direct preference optimization training set.

[0068] S408. Utilize direct preference optimization training set, and retrain the direct preference optimization loss function of retrieval perception every month to establish a monthly evaluation and data iteration mechanism, forming a continuous evolutionary closed loop of deployment, collection, evaluation and optimization, and completing the training of the large language model.

[0069] Furthermore, the error and resolution data includes: inputs, preferred answers, and rejected answers;

[0070] The input is specifically a structured error scenario, which includes: background, parameter changes, error actions, and alarm information;

[0071] The preferred answers are specifically high-quality solutions written by engineers, including: error cause analysis, correct handling steps, explanation of the principle behind each step, relevant risk points and safety thresholds, and reminders of common misconceptions; and the cited enterprise documents are marked.

[0072] The refusal to answer specifically refers to the user's original incorrect answer or superficial explanation;

[0073] The manually labeled preference pairs include preferred answers and rejected answers.

[0074] Furthermore, the expression for the retrieval-aware direct preference optimization loss function is as follows:

[0075] ;

[0076] ;

[0077] ;

[0078] in, This represents the direct preference optimization loss function for retrieval perception. Indicates input and preferred answers The verification header score, Represents the regularity coefficient. This represents the standard direct preference optimization loss. Represents the preference of triples in the dataset Expectations Represents a preference dataset, Indicates input, Indicate preferred answer, He indicated that he refused to answer. Indicates the temperature coefficient. This represents the sigmoid function. Indicates the current training model, This represents a large language model that has undergone retrieval-enhanced alignment. Indicates input and preferred answers Support score.

[0079] Furthermore, the expression for the evidence alignment regularization term is as follows:

[0080] ;

[0081] in, This indicates the evidence alignment regularization term. Indicates input and preferred answers Support score Represents the gradient penalty coefficient. This indicates the preset support threshold. This represents the log probability gradient of the preference response.

[0082] The beneficial effects of the above-mentioned further solutions are as follows: This invention constructs a retrieval-aware direct preference optimization loss function through retrieval enhancement generation and direct preference optimization, and constructs an evidence alignment regularization term through the reuse of trained verification heads, thereby improving the teaching logic and safety standards that the output of the large language model conforms to, so that the large language model, after training, has a full-stack capability from physical rule understanding to teaching guidance generation. Attached Figure Description

[0083] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0084] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0085] Before describing this embodiment, the following terms will be explained:

[0086] LLM: Large Language Model;

[0087] token: word element;

[0088] Top-k sampling: a method for sampling from the top-ranked... k The sampling method for sampling from the token;

[0089] CLS: Special tags used to represent the entire text sequence and their corresponding output vectors;

[0090] DPO: Direct Preference Optimization;

[0091] prompt: a prompt word;

[0092] RAG: Search Enhancement Generation;

[0093] sigmoid function: S-shaped growth curve function;

[0094] RAG-Aware DPO Loss Function: Retrieval-aware direct preference optimization loss function;

[0095] SPE / OnePetro: American Society of Petroleum Engineers;

[0096] API: Application Programming Interface.

[0097] Example

[0098] In this embodiment, a four-stage progressive large language model training framework is proposed to address the technical problems of knowledge gaps, logical breaks, frequent illusions, and weak pedagogical effectiveness in the base LLM of the intelligent teaching assistant system in oil drilling platform simulation training. The framework enables the large language model to have full-stack capabilities from physical rule understanding to teaching guidance generation by injecting domain knowledge in stages, aligning private documents, and optimizing expert preferences.

[0099] like Figure 1 As shown, this invention provides an LLM training method for an intelligent teaching assistant system based on a drilling simulator, the implementation of which is as follows:

[0100] S1. Based on the large language model of the intelligent teaching assistant system, obtain drilling simulator simulation data, and perform domain-adaptive pre-training on the large language model to obtain a physically-based large language model. The specific steps are as follows:

[0101] S101. Based on the large language model of the intelligent teaching assistant system, simulation data of the drilling simulator is obtained according to the physical laws of fluid mechanics, rock mechanics and control theory involved in the drilling process.

[0102] S102. Based on the model parameters of the large language model and combined with the mask position set, the large language model is modeled using mixed language by calculating the mixed language loss function, thus obtaining the large language model modeled using mixed language.

[0103] In this embodiment, although the general large language model can process natural language, it lacks an understanding of the fundamental physical laws involved in the drilling process, such as fluid mechanics, rock mechanics, and control theory. For example, the model cannot infer "well leakage or drill bit water hole blockage" from "pump pressure increase and return decrease" because it has not learned the law of conservation of mass and Bernoulli's equation. The first stage aims to enable the model to master scientific reasoning abilities such as state evolution, causal intervention, and dimensional consistency through domain adaptive pre-training.

[0104] Therefore, it lays the foundation of physical rules and engineering common sense, and endows the model with scientific reasoning ability;

[0105] The large language model underlying the intelligent teaching assistant system is an existing model.

[0106] In this embodiment, based on the large language model of the intelligent teaching assistant system, and according to the physical laws of fluid mechanics, rock mechanics and control theory involved in the drilling process, simulation data of the drilling simulator is obtained from public textbooks (such as "Engineering Fluid Mechanics" and "Automatic Control Principles"), SPE / OnePetro academic papers, API engineering manuals and physical derivation question banks.

[0107] Data formats include: plain text paragraphs (for language modeling); structured triples: (initial conditions, intervention actions, evolutionary results); and mathematical derivation chains, specifically: premises, formulas, and conclusions.

[0108] Based on the model parameters of the large language model and combined with the mask location set, a mixed language modeling task is established, and the mixed language loss function is obtained, as shown in the following expression:

[0109] ;

[0110] in, Represents the loss function for mixed languages. Represents the set of mask locations. Indicates that given model parameters and word position Under the condition that the next word in the sequence is exactly The probability, Represents probability. Indicates the masked number i Each word element, Indicates the current position in the sequence. i All previous morphemes, The parameters represent the model parameters. The large language model is then used for mixed language modeling to obtain the large language model after mixed language modeling.

[0111] S103. Based on the drilling simulator simulation data, obtain the initial state vector of drilling, and obtain the real evolution result through preset real simulation;

[0112] S104. Based on the linear regression head after the encoder of the large language model, the evolution loss function is obtained by calculating the predicted value of the state vector and combining it with the actual evolution results. The physical state evolution prediction is performed on the large language model modeled by mixed language to obtain the large language model predicted by physical state evolution.

[0113] In this embodiment, the initial state vector of drilling is obtained based on the drilling simulator simulation data, and the actual evolution results are generated using professional simulation software. Based on the linear regression head following the large language model encoder, physical state evolution is predicted to obtain the evolution loss function, as shown in the following expression:

[0114] , ;

[0115] in, Represents the predictions of large language models Current state This indicates the back-end linearity return head of the LLM encoder. Represents the initial state vector. , Represents the vector dimension. express time, Represents the actual evolutionary outcome. The state loss function is used to obtain a large language model predicted by the evolution of physical states.

[0116] S105. Based on the intervention actions and problem descriptions in the drilling simulator simulation data, and combined with the expected results annotated by experts, obtain the causal loss function;

[0117] S106. Using the causal loss function, perform causal intervention reasoning on the large language model modeled by mixed language to obtain the large language model based on physics.

[0118] In this embodiment, causal intervention reasoning is performed. Based on the intervention actions and problem descriptions in the drilling simulator simulation data, combined with the expected results annotated by experts, a causal loss function is obtained, as shown in the following expression:

[0119] ;

[0120] in, Represents the causal loss function. This indicates the expected result as indicated by the expert (e.g., "Casing pressure increases, vertical pressure stabilizes"). This indicates a problem description (e.g., "Currently, there are obvious signs of a well blowout"). Representing intervention actions (such as "closing the annular blowout preventer") yields a physically based large language model;

[0121] In this stage, domain-adaptive pre-training is used, and only the parameters of the high-level Transformer are fine-tuned to avoid damaging the underlying language capabilities.

[0122] S2. Obtain training data, fine-tune the original weights using low-rank adaptive methods, and perform operational steps and fault diagnosis generation on the physically-based large language model to obtain a domain-semantic-adapted large language model. The specific steps are as follows:

[0123] S201. Based on the drilling equipment and standard operating procedures, obtain training data including operation planning data and fault diagnosis data;

[0124] S202. Using low-rank adaptive adjustment, the original weights in the physically-based large language model are fine-tuned to obtain updated weights.

[0125] S203. Based on the operation planning data, generate operation steps for the physically-based large language model to obtain the operation loss function;

[0126] S204. Based on the fault diagnosis data, generate fault diagnosis text for the physically-based large language model to obtain the fault diagnosis loss function.

[0127] S205. Using the updated weights, operational loss function, and fault diagnosis loss function, the physical-based large language model is adapted to the domain semantics to obtain the domain-adapted large language model.

[0128] In this embodiment, the model is given general physical reasoning ability in the first stage, but has not yet been exposed to specific drilling equipment (such as top drive and blowout preventer) and standard operating procedures (SOPs). In the simulator, the trainee may execute any sequence of operations, and the system needs to be able to dynamically generate operating steps or open domain fault explanations that conform to the procedures (rather than preset categories). Therefore, the second stage focuses on training the model's ability to understand the text description of equipment functions, operating logic and abnormal phenomena.

[0129] Training data, including operation planning data and fault diagnosis data, is obtained from equipment technical manuals, enterprise SOPs, maintenance work orders, and site logs of drilling equipment and standard operating procedures.

[0130] Operational planning data includes objectives and constraints, used for natural language step sequences (including cause explanations); fault diagnosis data includes sensor time series X and text descriptions, used for open-domain fault analysis text.

[0131] The fault diagnosis task does not pre-define fixed categories. Instead, engineers write natural language explanations (such as "abnormal torque increase may be due to increased friction caused by wellbore diameter reduction"). The large language model needs to generate text in a similar style.

[0132] In this embodiment, low-rank adaptive (LoRA) fine-tuning is performed. To efficiently inject domain knowledge, the low-rank adaptive (LoRA) technique is used to fine-tune the original weights. The updated weights are obtained by superimposing low-rank updates on top of each other, as shown in the following expression:

[0133] ;

[0134] ;

[0135] in, This indicates an update to the weights. This represents the weight update matrix. Denotes the downward projection matrix. Represents the projection matrix. This represents a low-dimensional representation, typically using 32, which significantly reduces the number of training parameters. The dimension is The parameter matrix is, The dimension is The parameter matrix;

[0136] The operation steps are generated, and the operation loss function is obtained based on the operation planning data, as shown in the following expression:

[0137] ;

[0138] in, Represents the operational loss function. Indicates completion Number of operations required Indicates the first i The word element of the step operation, Indicate the objective (e.g., "run the casing to the design depth"), Indicates site restrictions (such as "well inclination") (”);

[0139] Fault diagnosis text is generated, and the fault diagnosis loss function is obtained based on the fault diagnosis data. The expression is as follows:

[0140] ;

[0141] in, This represents the fault diagnosis loss function. This refers to a natural language fault analysis written by an engineer. This represents sensor timing data (such as drilling pressure, torque, pump pressure). It represents on-site descriptions (such as "obstruction encountered during drilling"), and the fault diagnosis loss function is the standard negative log-likelihood (NLL), which is suitable for open text generation;

[0142] By utilizing update weights, operational loss functions, and fault diagnosis loss functions, domain semantic adaptation is performed on the physically based large language model to obtain a domain semantically adapted large language model.

[0143] S3. Construct a private enterprise knowledge base and use retrieval enhancement to train a large language model that has undergone domain semantic adaptation, thereby obtaining a large language model aligned with retrieval enhancement. Then, through explicit supervision, train a trained validation head. The specific steps are as follows:

[0144] S301. Parse portable document formats or standard operating procedures into text, generate embeddings using a bidirectional encoder representation method, and build an enterprise private knowledge base by storing them in a vector database.

[0145] S302. Based on the enterprise's private knowledge base, use a preset sampling method to obtain document fragments, obtain a set of search paragraphs, and concatenate the question with the document fragments to obtain the gold standard answer;

[0146] S303. Combine the question, the set of search paragraphs, and the gold standard answer to obtain the labeled triples;

[0147] S303. Based on the labeled triples, use retrieval enhancement generation, obtain the multi-document retrieval enhancement generation loss, and train the domain semantically adapted large language model to obtain the retrieval enhancement aligned large language model.

[0148] S304. Based on the gold standard answer and the average embedding of the retrieved paragraph set, the trained validation head is obtained by explicitly supervising whether the generated content is supported by the retrieval and by calculating the retrieval alignment loss.

[0149] In this embodiment, although the large language model adapted to domain semantics in the second stage can generate reasonable explanations, it may "fabricate" operational suggestions that do not appear in the enterprise documents (such as recommending non-standard well control fluids), which poses a safety risk. To address this, the third stage introduces the enterprise's private knowledge base (SOPs, accident reports, and equipment manuals), and ensures that all key conclusions are documented through Retrieval Enhanced Generation (RAG), and trains a verification head to determine whether the generated content is supported by the retrieved evidence.

[0150] By parsing portable document formats or standard operating procedures into text, generating embeddings using a bidirectional encoder representation method, storing them in a vector database, and building a private enterprise knowledge base, metadata (doc_id and equipment_type, etc.) is attached to each document.

[0151] And perform QA pair (question-answer pair) annotation, with triplet (question) annotated by engineers. q Search paragraph set D Gold Standard Answer ), and the gold standard answer You must reference the search paragraph set D Retrieve at least one segment from the paragraph set. , Specifically, it refers to the set of Top-k document fragments retrieved, and each The concatenation and decoding processes combine all representations to generate the gold standard answer. .

[0152] In this embodiment, the multi-document retrieval enhancement generation loss is obtained by generating the loss based on the triples annotated by the engineer, as shown in the following expression:

[0153] ;

[0154] in, This represents the enhanced generation loss for multi-document retrieval. This represents the gold standard answer. To indicate a problem, This means that the retrieved paragraph set is used to train a large language model that has undergone domain semantic adaptation, resulting in a large language model that has been enhanced and aligned by retrieval.

[0155] Obtain the trained validation head. To explicitly supervise whether the generated content is supported by the retrieval, train a binary classification validation head, as shown in the following expression:

[0156] , ;

[0157] in, This indicates the probability that the generated answer is supported by the retrieved evidence. This represents the weight vector of the verification header. Indicates the gold standard answer of express, Indicates the retrieval of paragraph sets Average embedding, This indicates a trained verification head. Represents the binary cross-entropy loss. Indicates an indicator function, express exist There is insufficient evidence to support this. If and only if can be This is obtained when the content logic is derived from it;

[0158] The validation head takes the semantic representation of the generated answer and the average semantic representation of the retrieved paragraph as input, and outputs a support score through a binary classification network.

[0159] S4. Obtain error and resolution data. Based on the retrieved and aligned large language model, define the standard direct preference optimization loss function and combine it with the trained validation head to obtain the RAG-Aware DPO loss function. Optimize the large language model to obtain the preference-optimized large language model, thus completing the large language model training. The specific steps are as follows:

[0160] S401. Based on the teaching needs of drilling training, use real error scenarios accumulated in drilling simulators and high-quality solutions written by engineers to obtain error and solution data.

[0161] S402. Based on the retrieved and enhanced aligned large language model, and combined with manually labeled preference pairs, define the standard direct preference optimization loss function;

[0162] In this embodiment, the first three stages enable the large language model to possess physical knowledge, equipment understanding, and document citation capabilities, but its output style may not meet the teaching needs of drilling training. For example, the large language model may only answer "operation error" without explaining key teaching elements such as "why the error occurred," "what the correct steps are," and "which safety thresholds are involved." The fourth stage utilizes a large number of real error scenarios accumulated in the simulator and high-quality solutions written by engineers to optimize the model through DPO (Drilling Operations Program), so that its output is not only technically correct but also has teaching value, risk awareness, and evidence citation habits.

[0163] Retrieve error and resolution data, including: input prompt, preferred answer (Chosen), and rejected answer (Rejected);

[0164] The input Prompt is specifically a structured error scenario, including: {background, parameter change, error action, alarm information}. For example: {background: "Drilling to 2500m", parameters: {"Drill pressure": 80kN, "Torque": 12kN·m↑}, error action: "No grouting", alarm: "Wellhead return is zero"};

[0165] Chosen preferred answers specifically refer to: high-quality solutions written by engineers, including: error cause analysis, correct handling steps, explanation of the principle of each step, relevant risk points and safety thresholds (such as "well shut-in casing pressure > 5MPa requires well control"), and reminders of common misconceptions, while also indicating the enterprise document ID (doc_id) it references.

[0166] The term "Rejected" specifically refers to a user's original incorrect answer or superficial explanation, such as "The pump might be broken, try restarting it" or "Continue drilling and see."

[0167] In this embodiment, based on the retrieved and enhanced aligned large language model and combined with manually annotated preference pairs, the standard direct preference optimization loss function is defined, resulting in the standard DPO loss function, as shown in the following expression:

[0168] ;

[0169] in, This represents the standard DPO loss function. Represents the preference of triples in the dataset The expectation, specifically, is for each sample triple across the entire preference dataset. Calculate the loss, then calculate the expectation; the model's loss is not calculated for just one input (prompt) and one pair of responses, but rather the average loss is taken over all samples in the entire preference dataset. Represents a preference dataset, This indicates the input (prompt). Indicates a preferred answer (chosen). This indicates a refusal to answer (rejected). This represents the temperature coefficient (usually taken as 0.1–1.0). This represents the sigmoid function. Indicates the current training model, This represents the large language model that has been enhanced and aligned through retrieval, which is the frozen reference model (i.e., the model output from the third stage). This represents the log-probability ratio difference in the standard DPO loss function.

[0170] S403. Based on the trained verification head, obtain the support score, and define the evidence alignment regularization term according to the preset support threshold.

[0171] S404. By using the evidence alignment regularization term, a retrieval enhancement generation alignment mechanism is introduced into the standard direct preference optimization loss function to obtain the direct preference optimization loss function.

[0172] In this embodiment, the DPO method relies solely on manually labeled preference pairs (chosen, rejected), which cannot guarantee that the content of the chosen answer is based on retrieval evidence in the enterprise's private knowledge base. There is still a risk of the illusion that the answer is "seemingly reasonable but without basis." To address this, this invention proposes the RAG-Aware DPO loss function, which explicitly introduces retrieval consistency constraints.

[0173] To introduce the RAG alignment mechanism, an evidence alignment regularization term is added to the standard DPO loss. The total loss is expressed as follows:

[0174] ;

[0175] in, Indicates the total loss. Indicates standard DPO loss, This indicates the evidence alignment regularization term. This represents the regularization coefficient, set to 0.2, used to balance preference learning and evidence alignment;

[0176] RAG alignment items The computation relies on a verification head trained in the third stage; this verification head has learned in the third stage to determine whether the generated content is supported by the retrieved evidence, and its output is a support score. Based on this, we define as follows:

[0177] ;

[0178] in, This indicates that the third-stage verification head is paired with the input. and preferred answers The output score, This indicates the preset support threshold. This represents the gradient penalty coefficient, which is set to 0.1. This represents the log probability gradient of the preference response.

[0179] S405. The trained validation head is used as the dynamic weight of the direct preference optimization loss function to obtain the RAG-AwareDPO loss function;

[0180] S406. Optimize the large language model using the RAG-Aware DPO loss function to obtain a preference-optimized large language model.

[0181] In this embodiment, to enhance the learning weights of samples with strong evidence support, the validation head score is used as the dynamic weight of the DPO loss, resulting in the RAG-Aware DPO loss function, as shown in the following expression:

[0182] ;

[0183] ;

[0184] in, Represents the dynamic total loss. Indicates input and preferred answers The validation head score enables the large language model to prioritize learning expert answers that are highly supported by the retrieved evidence during the optimization process, achieving explicit collaboration between RAG and DPO, and obtaining a large language model optimized by preferences.

[0185] S407. Using the large language model optimized by preference, obtain the model output, score the model output according to the preset dimension, and add the sample pairs in the model output that are greater than or equal to the preset score value to the direct preference optimization training set.

[0186] S408. Utilize direct preference optimization of the training set, and establish a monthly evaluation and data iteration mechanism by retraining the RAG-Aware DPO loss function monthly. This forms a continuous evolutionary closed loop of deployment, collection, evaluation, and optimization, thereby completing the training of the large language model.

[0187] In this embodiment, to continuously improve the performance of large language applications, a monthly evaluation and data iteration mechanism is established, as follows:

[0188] Senior drilling engineers scored the output of the preference-optimized large language model according to the following five dimensions, using a 1-5 point scale:

[0189] The accuracy of the professional assessment is evaluated based on whether it conforms to physical laws and API / SOP safety specifications.

[0190] The teaching effectiveness will be assessed based on whether the teacher clearly explains "why the mistake was made" and "how to avoid it."

[0191] Scoring the completeness of the solution: Does the process cover the entire process of diagnosis, treatment, and verification?

[0192] Scoring of risk and threshold sensitivity: whether key safety thresholds (such as shut-in pressure limits) are mentioned.

[0193] The quality of retrieved evidence is scored based on whether the content of corporate documents is accurately cited.

[0194] And add the sample pairs (prompt, chosen, rejected) from the model output with a score ≥ preset score value 4.0 to the DPO training set;

[0195] The fourth-stage, preference-optimized large language model is retrained monthly, forming a continuous evolutionary closed loop of "deployment → collection → evaluation → optimization".

[0196] In this embodiment, a four-stage training framework is used to systematically solve the core technical bottleneck of industrial large models in professional training scenarios: the first stage injects physical rules to solve the problem of not understanding the mechanism;

[0197] The second phase of training focuses on open-domain fault interpretation to address the inability to handle unknown anomalies.

[0198] The third stage mandates knowledge citation to address the issues of hallucinations and security risks.

[0199] The fourth stage involves optimizing teaching styles and addressing the issue of outputs not meeting training objectives.

[0200] Crucially, the fourth-stage RAG alignment mechanism explicitly reuses the verification head from the third stage, ensuring that preference optimization is always anchored on the enterprise's proprietary knowledge, forming a technological closed loop; and acquiring the optimal intelligent teaching assistant system base LLM to complete the training of the large language model.

[0201] In this embodiment, the first stage is the scientific reasoning basis of the entire training process. The first stage, through domain adaptive pre-training, endows the model with the inherent understanding of the physical laws (such as fluid conservation, mechanical equilibrium, and control feedback) in the drilling process.

[0202] Although its output, a large language model based on physics, does not have specific device knowledge, it can still perform state evolution prediction and causal intervention reasoning, providing a reliable scientific logical foundation for subsequent stages.

[0203] The second phase of operation and fault analysis must be based on the premise of physical rationality. Therefore, the second phase uses the output model of the first phase as the starting point for training.

[0204] In this embodiment, the second stage achieves domain semantic adaptation based on the first stage. The second stage maps general physical reasoning capabilities to the context of specific drilling equipment, process flow and abnormal phenomena, so that the model can understand professional concepts such as "top drive jamming" and "well kick signs" and generate operation instructions or open domain fault explanations that conform to SOP.

[0205] The training data in the second stage (such as sensor time series + text description) implicitly contains changes in physical state. If the state evolution modeling capability of the first stage is lacking, the model will have difficulty correctly associating causal relationships such as "increased torque" and "increased wellbore friction". Therefore, the second stage must inherit and expand the model capability of the first stage.

[0206] In this embodiment, the third stage introduces enterprise knowledge constraints into the content generated in the second stage. Although the model in the second stage can generate seemingly reasonable explanations, it may "illusory" provide unauthorized operational suggestions from the enterprise (such as using non-standard well control fluid). The third stage, by introducing the enterprise's private knowledge base (SOP, accident report, equipment manual), forces all key conclusions of the model to be supported by documentation. The RAG model is based on the model in the second stage and adds a retrieval alignment mechanism.

[0207] Crucially, the trained validation head will be reused in the fourth stage as a discriminative signal for evidence alignment. Therefore, the third stage serves as a bridge connecting knowledge alignment and preference optimization.

[0208] In this embodiment, the fourth stage uses the output of the third stage as a constraint to optimize teaching behavior. The goal of the fourth stage is no longer to improve consistency with the enterprise's requirements (which have already been achieved in the third stage), but to optimize the output teaching style, the completeness of risk warnings, and the ability to guide students.

[0209] We propose the RAG-Aware DPO loss function, which explicitly calls the validation head trained in the third stage, enabling the model to favor outputs with answers supported by search results and adjust the optimization direction accordingly.

[0210] Evidence alignment regularization Its value is: when the support score of the validation head for the chosen answer is lower than the preset threshold, it is equal to the weighted value of the gradient L2 norm, otherwise it is zero; and a dynamic weight term is used to use the validation head score as the multiplicative weight of the DPO loss.

Claims

1. A training method for an intelligent teaching assistant system based on an LLM (Limited Learning Model) for drilling simulators, characterized in that, Includes the following steps: S1. Based on the intelligent teaching assistant system's base large language model, obtain drilling simulator simulation data, and perform domain-adaptive pre-training on the large language model to obtain a physically-based large language model. S2. Obtain training data, use low-rank adaptation to fine-tune the original weights, and perform operation steps and fault diagnosis generation on the physically based large language model to obtain a domain semantically adapted large language model. S3. Construct a private enterprise knowledge base and use retrieval enhancement to train a large language model that has undergone domain semantic adaptation, resulting in a retrieval-enhanced aligned large language model. Then, through explicit supervision, train a trained validation head, specifically: S301. Parse portable document formats or standard operating procedures into text, generate embeddings using a bidirectional encoder representation method, and build an enterprise private knowledge base by storing them in a vector database. S302. Based on the enterprise's private knowledge base, use a preset sampling method to obtain document fragments, obtain a set of search paragraphs, and concatenate the question with the document fragments to obtain the gold standard answer; S303. Combine the question, the set of search paragraphs, and the gold standard answer to obtain the labeled triples; S303. Based on the labeled triples, use retrieval enhancement generation, obtain the multi-document retrieval enhancement generation loss, and train the domain semantically adapted large language model to obtain the retrieval enhancement aligned large language model. S304. Based on the gold standard answer and the average embedding of the retrieved paragraph set, the trained validation head is trained by explicitly supervising whether the generated content is supported by the retrieval and calculating the retrieval alignment loss. The expression for the multi-document retrieval enhancement generation loss is as follows: in, This represents the enhanced generation loss for multi-document retrieval. Represents probability. This represents the gold standard answer. To indicate a problem, This indicates the search for a set of paragraphs. Indicates model parameters; The expression for the retrieval alignment loss is as follows: in, This represents the retrieval alignment loss. Represents the binary cross-entropy loss. This indicates the probability that the generated answer is supported by the retrieved evidence. Indicates an indicator function, This represents the weight vector of the verification header. Indicates the gold standard answer of express, Indicates the retrieval of paragraph sets Average embedding; S4. Obtain error and resolution data. Based on the retrieval-enhanced aligned large language model, define the standard direct preference optimization loss function and, combined with the trained validation head, obtain the retrieval-aware direct preference optimization loss function. Optimize the large language model to obtain the preference-optimized large language model, thus completing the large language model training. Specifically: S401. Based on the teaching needs of drilling training, use real error scenarios accumulated in drilling simulators and high-quality solutions written by engineers to obtain error and solution data. S402. Based on the retrieved and enhanced aligned large language model, and combined with manually labeled preference pairs, define the standard direct preference optimization loss function; S403. Based on the trained verification head, obtain the support score, and define the evidence alignment regularization term according to the preset support threshold. S404. Using the evidence alignment regularization term, a retrieval enhancement generation alignment mechanism is introduced into the standard direct preference optimization loss function to obtain the direct preference optimization loss function. S405. The trained validation head is used as the dynamic weight of the direct preference optimization loss function to obtain the retrieval-aware direct preference optimization loss function; S406. Optimize the large language model by using the direct preference optimization loss function based on retrieval perception to obtain a preference-optimized large language model. S407. Using the large language model optimized by preference, obtain the model output, score the model output according to the preset dimension, and add the sample pairs in the model output that are greater than or equal to the preset score value to the direct preference optimization training set. S408. Utilize direct preference optimization training set, and retrain the direct preference optimization loss function of retrieval perception every month to establish a monthly evaluation and data iteration mechanism, forming a continuous evolutionary closed loop of deployment, collection, evaluation and optimization, and completing the training of the large language model. The error and resolution data includes: input, preferred answers, and rejected answers; The input is specifically a structured error scenario, which includes: background, parameter changes, error actions, and alarm information; The preferred answers are specifically high-quality solutions written by engineers, including: error cause analysis, correct handling steps, explanation of the principle behind each step, relevant risk points and safety thresholds, and reminders of common misconceptions; and the cited enterprise documents are marked. The refusal to answer specifically refers to the user's original incorrect answer or superficial explanation; The manually labeled preference pairs include preferred answers and rejected answers; The expression for the retrieval-aware direct preference optimization loss function is as follows: in, This represents the direct preference optimization loss function for retrieval perception. Indicates input and preferred answers The verification header score, Represents the regularity coefficient. This represents the standard direct preference optimization loss. Represents the preference of triples in the dataset Expectations Represents a preference dataset, Indicates input, Indicate preferred answer, He indicated that he refused to answer. Indicates the temperature coefficient. This represents the sigmoid function. Indicates the current training model, This represents a large language model that has undergone retrieval-enhanced alignment. Indicates input and preferred answers Support score; The expression for the evidence alignment regularization term is as follows: in, This indicates the evidence alignment regularization term. Indicates input and preferred answers Support score Represents the gradient penalty coefficient. This indicates the preset support threshold. This represents the log probability gradient of the preference response.

2. The LLM training method for an intelligent teaching assistant system based on a drilling simulator according to claim 1, characterized in that, S1 includes the following steps: S101. Based on the large language model of the intelligent teaching assistant system, simulation data of the drilling simulator is obtained according to the physical laws of fluid mechanics, rock mechanics and control theory involved in the drilling process. S102. Based on the model parameters of the large language model and combined with the mask position set, the large language model is modeled using mixed language by calculating the mixed language loss function, thus obtaining the large language model modeled using mixed language. S103. Based on the drilling simulator simulation data, obtain the initial state vector of drilling, and obtain the real evolution result through preset real simulation; S104. Based on the linear regression head after the encoder of the large language model, the evolution loss function is obtained by calculating the predicted value of the state vector and combining it with the actual evolution results. The physical state evolution prediction is performed on the large language model modeled by mixed language to obtain the large language model predicted by physical state evolution. S105. Based on the intervention actions and problem descriptions in the drilling simulator simulation data, and combined with the expected results annotated by experts, obtain the causal loss function; S106. Using the causal loss function, perform causal intervention reasoning on the large language model modeled by mixed language to obtain the large language model based on physics.

3. The LLM training method for an intelligent teaching assistant system based on a drilling simulator according to claim 2, characterized in that, The drilling simulator simulation data includes: ordinary text paragraphs for language modeling, structured triples containing initial conditions, intervention actions, and evolution results, as well as mathematical derivation chains; The mathematical derivation chain is specifically a chain of premises, formulas, and conclusions.

4. The LLM training method for an intelligent teaching assistant system for drilling simulators according to claim 3, characterized in that, The expression for the mixed language loss function is as follows: in, Represents the loss function for mixed languages. Represents the set of mask locations. Indicates that given model parameters and word position Under the condition that the next word in the sequence is exactly The probability, Represents probability. Indicates the masked number i Each word element, Indicates the current position in the sequence. i All previous morphemes, Indicates model parameters; The expression for the evolutionary loss function is as follows: in, Represents the state loss function. Represents the actual evolutionary outcome. Indicates prediction Current state This represents the post-linear regression head of the large language model encoder. Represents the initial state vector. express time; The expression for the causal loss function is as follows: in, Represents the causal loss function. This indicates the expected results of expert annotation. This indicates a problem description. This indicates an intervention action.

5. The LLM training method for an intelligent teaching assistant system based on a drilling simulator according to claim 1, characterized in that, S2 includes the following steps: S201. Based on the drilling equipment and standard operating procedures, obtain training data including operation planning data and fault diagnosis data; S202. Using low-rank adaptive adjustment, the original weights in the physically-based large language model are fine-tuned to obtain updated weights. S203. Based on the operation planning data, generate operation steps for the physically-based large language model to obtain the operation loss function; S204. Based on the fault diagnosis data, generate fault diagnosis text for the physically-based large language model to obtain the fault diagnosis loss function. S205. Using the updated weights, operational loss function, and fault diagnosis loss function, the physical-based large language model is adapted to the domain semantics to obtain the domain-adapted large language model.

6. The LLM training method for an intelligent teaching assistant system base for drilling simulators according to claim 5, characterized in that, The expression for the operational loss function is as follows: in, Represents the operational loss function. Indicates completion Number of operations required Represents probability. Indicates the first i The word element of the step operation, Indicate the goal, Indicates on-site restrictions, Indicates model parameters; The expression for the fault diagnosis loss function is as follows: in, This represents the fault diagnosis loss function. This refers to a natural language fault analysis written by an engineer. Represents sensor time-series data, This indicates a description of the scene.

Images