A method and system for identifying drilling simulation operations

Abstract

The application provides a drilling simulation operation recognition method and system, and belongs to the field of drilling operation. The method comprises the following steps: constructing a knowledge graph, and generating node vector embedding in the process of constructing the knowledge graph; pre-training and adapting processing are performed on a multi-modal encoder; three candidate node paths are generated to obtain relevant nodes; after adding an intelligent subgraph to the node with the highest correlation, the node vector embedding in the intelligent subgraph is dynamically adjusted in real time; a multi-modal similarity matrix is constructed; a time sequence bias matrix is constructed by fusing time sequence difference information; a potential bottleneck is fused by calculating multi-modal attention; and based on the fusion result, a final prediction and confidence evaluation result are output. The application solves the problem that in the condition that internal operation logs and control instructions cannot be obtained in a closed industrial control system, the prior art cannot effectively associate external multi-modal observation information, so that the operation stage and user operation cannot be accurately recognized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of drilling operation, and particularly relates to a drilling simulation operation recognition method and system. BACKGROUND

[0002] The drilling operation simulation training system is the most important personnel training equipment in the oil and gas industry. The traditional training mode highly depends on manual instructors. One instructor can take a maximum of 3-5 students at a time. The training cost is high, the scale is limited, and the teaching quality is seriously dependent on the personal experience of the instructor.

[0003] The existing simulator generally adopts a completely closed architecture to ensure the fairness of training, prevent students from cheating, system stability, and most importantly, prevent leakage. The reasons for the closure include but are not limited to: 1. The system internally contains a large number of self-developed core algorithms, proprietary control logic, process parameter curves, and accident models. Once the interface or log is opened, it is considered as a technical leakage, which may involve business secrets or even state secrets; 2. The first party explicitly prohibits providing any internal variable name, state machine, real label, or operation log to third-party developers in the contract; 3. Only screen display, limited analog output, or desensitized time-structured data files are allowed to interact with the outside.

[0004] This results in that the third party cannot directly obtain the current operation of the student, the operation stage, or the real state of the system through the conventional API or log when developing intelligent training functions. Traditional teaching can only rely on the instructor's naked eye to monitor the screen. The student's expression is vague, the misoperation is frequent, and the instructor is severely fatigued, which has become the biggest bottleneck restricting large-scale, standardized, and intelligent development.

[0005] The existing drilling operation intelligent monitoring system and automatic control system generally adopt a closed architecture design for safety and stability considerations, and do not open internal data interfaces to external algorithms or third-party systems. Although this closed design improves the safety of the system, it also brings a series of technical problems:

[0006] System state and operation behavior cannot be directly identified: Since the internal operation signals, control instructions, and real-time state data of the system are not open to the outside, the external algorithm cannot directly identify the current operation (such as drilling, tripping, pump closing, pressure adjustment, etc.) through the conventional API or communication protocol, which leads to the inability to understand the user behavior or working condition changes in real time.

[0007] Multi-modal information is fragmented, and there is a lack of unified recognition mechanism: The information available in the closed system is usually limited to external display interface screenshots, sensor logs, or manually recorded texts. These information is distributed in different modalities (vision, text, numerical parameters), and the existing technology lacks joint analysis and fusion mechanism for these modalities, making it difficult to form a unified working condition cognition.

[0008] The existing visual recognition method lacks field knowledge support: the traditional image recognition algorithm (such as image text recognition model or target detection) can only recognize the surface graphics or numerical values, and cannot understand the semantic relationship between various interface elements in combination with the drilling process knowledge, for example, it cannot distinguish the difference in the meaning of "pump pressure" in different stages.

[0009] Lack of dynamic correlation modeling at the timing and semantic levels: due to the closed system and the inability to directly obtain the operation log, the existing method is difficult to infer the timing evolution and causal relationship of the operation from the external observation data, resulting in inaccurate identification of the operation stage, abnormal state or trend change.

[0010] Lack of adaptive information fusion strategy: there are quality differences in external observation signals, such as interface blur, image text recognition error, parameter update delay, etc. The existing fusion algorithm cannot dynamically adjust the weight according to the information confidence, which is easy to cause misjudgment or missed detection.

[0011] Lack of explainability and uncertainty evaluation mechanism: external identification and reasoning in a closed system are easily affected by data noise and modal differences. The existing model cannot quantify the uncertainty of the prediction result, nor can it actively request additional information or interactive correction. SUMMARY

[0012] In view of the above deficiencies in the prior art, the drilling simulation operation recognition method and system provided by the present application solve the problem that the prior art cannot effectively associate the external multi-modal observation information under the condition that the internal operation log and control instruction cannot be obtained in a closed industrial control system, thereby cannot accurately identify what operation the user is currently doing.

[0013] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows: a drilling simulation operation recognition method, comprising the following steps:

[0014] S1, a knowledge graph containing snapshot-operation-hardware-parameter is constructed, and node vector embedding is generated in the knowledge graph construction process; wherein the snapshot layer is used to represent the overall configuration state information of the drilling simulation system at the beginning of the predetermined simulation task or at a specific analysis time, each snapshot node is associated with multiple operation nodes, each operation node is connected to all hardware nodes related thereto, and each hardware is associated with all related parameter nodes;

[0015] S2, pre-training and adaptation processing of multi-modal encoder;

[0016] S3, three candidate node paths are generated to obtain the most relevant nodes related to the question raised by the current user, wherein in the process of generating three candidate nodes, the adapted multi-modal encoder is called to encode the text, visual and parameter information, and dynamic weight is used for distribution;

[0017] S4, based on the obtained most relevant related nodes, expanding the related nodes with dynamic intelligent sub-graphs, and using dynamic context strategies to dynamically adjust the node vector embedding of the nodes added to the intelligent sub-graphs on the basis of the node vector embedding generated in S1 as the initial value;

[0018] S5, based on the dynamically adjusted node vector embedding and the text features and region-level visual features obtained through the multi-modal encoder, constructing a multi-modal similarity matrix;

[0019] S6, based on the multi-modal similarity matrix, constructing a time bias matrix by fusing time difference information;

[0020] S7, based on the time bias matrix, calculating multi-modal attention;

[0021] S8, based on the multi-modal attention, fusing the latent bottleneck;

[0022] S9, based on the fusion result, outputting the final prediction and confidence evaluation result, and completing the operation recognition of the drilling simulation.

[0023] Further, the S1 comprises the following steps:

[0024] S101, constructing a knowledge graph containing snapshot-operation-hardware-parameters;

[0025] S102, collecting text descriptions for each node in the process of constructing the knowledge graph, the text descriptions containing at least node name and function description; the nodes including snapshot nodes, operation nodes, hardware nodes and parameter nodes;

[0026] S103, calling a text encoder to encode the text description information to obtain initial semantic embedding of each node;

[0027] S104, based on the initial semantic embedding and the connection relationship of the nodes in the knowledge graph, aggregating the field information of each node, so that the generated node vector embedding contains both the semantic information of the node itself and the structural information of the adjacent nodes.

[0028] Further, the S2 comprises the following steps:

[0029] S201, using different functional areas in the drilling simulation operation interface to realize region segmentation processing;

[0030] S202, based on the region segmentation result, using a pre-trained visual encoder to extract global features for the entire interface screenshot to generate an initial visual embedding representing the overall state of the interface;

[0031] S203. Based on the region segmentation results, the interface images corresponding to each functional region are cropped and encoded separately to obtain region-level visual features. At the same time, the text content extracted from the interface and its confidence level are encoded to generate corresponding text features.

[0032] S204. The historical change patterns of parameters are processed using a time-series encoder. The short-term mode captures rapid changes at the second level, the medium-term mode is used to characterize the direction, magnitude and rate of parameter change within a preset time window, and the long-term mode analyzes the changes from the snapshot to the current drilling process, thus completing the pre-training and adaptation of the multimodal encoder.

[0033] Furthermore, the three candidate node paths in S3 include:

[0034] Precise path matching is used to process user-mentioned devices, parameters, or operations;

[0035] Semantic retrieval path, used to process users' natural language descriptions;

[0036] The visual-image text recognition (OCR) retrieval path is used to process interface information.

[0037] Furthermore, step S4 includes the following steps:

[0038] S401. Take the most relevant nodes to the problem as seed nodes, and selectively expand them using edge type and weight. Prioritize expanding the most relevant hardware nodes and parameter nodes from the operation nodes to generate a set of candidate context nodes.

[0039] S402. For the candidate context node set, calculate the node correlation probability by combining the temporal state information of the corresponding parameters, and select the node with the highest correlation based on the node correlation probability to add to the smart subgraph. Stop the selection when the average value is lower than the threshold.

[0040] S403. After obtaining the smart subgraph most relevant to the current query, based on the node vector embedding generated in S1, select the node embedding belonging to the smart subgraph as the initial state, and use lightweight graph convolution to dynamically adjust the node vector embedding in the smart subgraph so that the same node can obtain differentiated representations in different job contexts.

[0041] Furthermore, the dynamic adjustment includes the following steps:

[0042] In the first layer of graph convolution, for each node in the intelligent subgraph, the current node vector embedding is used as input. Each node performs weighted aggregation processing on its own current embedding vector and the embedding vectors of all its neighboring nodes, so that the node can perceive the directly related local context information.

[0043] In the second layer graph convolution, on the basis of the updated node embedding vectors of the first layer graph convolution, each node aggregates its own current embedding vector with the embedding vectors of all adjacent nodes again, so that the local context information is globally propagated within the intelligent subgraph within a two-hop range, and dynamic adjustment is completed, wherein after the dynamic adjustment, the node embedding vectors in the intelligent subgraph have incorporated the context information.

[0044] Further, the S5 comprises the following steps:

[0045] S501, based on the dynamically adjusted node vector embedding of S4, encoding the query text used to represent the current analysis target, fusing the context information of each token, and generating enhanced representation through self-attention mechanism;

[0046] S502, cosine similarity calculation between token embedding and knowledge graph node embedding;

[0047] S503, based on the cosine similarity calculation result, adding node type weighting to enhance the correlation, and completing the calculation of the text-knowledge graph similarity matrix

[0048] S504, according to the functional area type identified in S201, determining the weight initialization strategy of the text features corresponding to the area type and the area visual features;

[0049] S505, based on the weight initialization strategy, fusing the extracted text embedding and the generated visual embedding of the area;

[0050] S506, during the fusion, dynamically correcting the initial weight according to the image clarity and the image text recognition result confidence, and completing the calculation of the visual-knowledge graph similarity matrix

[0051] S507, by calculating the semantic association strength between the token and the visual area, the calculation of the cross-modal correlation matrix is completed.

[0052] Further, the S6 comprises the following steps:

[0053] S601, collecting parameter data generated during system operation and saving;

[0054] S602, based on the collected parameter data, calculating the multi-time scale time series difference features of the collected parameters under the same time scale as the parameter processed by the S204 time series encoder, obtaining the difference time series vector, which includes short-term difference change, medium-term difference change and long-term difference change;

[0055] ​​S603, calculate the activation strength vector based on the difference time sequence vector;

[0056] S604, generate a mask matrix according to a preset field to knowledge graph parameter node ID mapping table ;

[0057] S605, calculate the time sequence activation vector of the parameter node based on the activation strength vector and the mask matrix to correspond the time sequence difference information as a dynamic feature to the parameter node embedding;

[0058] S606, construct a cross-modal time sequence bias matrix based on the text-knowledge graph similarity matrix , the visual-knowledge graph similarity matrix and the time sequence activation vector .

[0059] S606, construct the time sequence bias matrix using .

[0060] Further, the expression of the time sequence bias matrix is as follows:

[0061] ;

[0062] ;

[0063] ;

[0064] ;

[0065] ;

[0066] wherein, denotes the time sequence bias matrix, denotes the text-knowledge graph similarity matrix, denotes the visual-knowledge graph similarity matrix, denotes the diagonal matrix constructed by the parameter node, the time sequence node and the time sequence activation vector d , denotes the time sequence activation vector corresponding to the knowledge graph parameter node one by one, denotes the MLP neural network, denotes the short-term change, denotes the medium-term change, denotes the long-term change, denotes the parameter state of the system when the current user asks, denotes the parameter state information saved in the last stage database, ​​This represents the parameter status information stored in the database during the first three stages. This indicates the original parameter information of the snapshot. This represents the set of knowledge graph parameter nodes related to the current task. express transpose, This indicates transpose.

[0067] Furthermore, S7 includes the following steps:

[0068] S701, Constructing Attention Bias ;

[0069] S702, based on temporal bias matrix and attention bias Multimodal attention is calculated.

[0070] Furthermore, the expression for the multimodal attention is as follows:

[0071] ;

[0072] ;

[0073] in, This represents multimodal attention. This represents the query vector obtained by linear transformation of text features. This represents the key vector obtained by linear transformation of visual features. Indicates transpose. This represents the feature dimensions of the query vector and the key vector. express Learnable weight parameters, express Learnable weight parameters, These represent the word index in the text feature sequence and the region index in the visual feature set at the functional region level, respectively. Represents the timing bias matrix. Indicates position code, Indicates a small linear layer. This represents a nonlinear compression function. Represents the cross-modal correlation matrix. Represents the natural logarithm operation. This represents the modified linear unit activation function.

[0074] Furthermore, S8 includes the following steps:

[0075] S801. A fixed-size learnable query vector is used as an information bottleneck to limit the intermediate representation space for cross-modal information fusion.

[0076] S802. Using the query vector corresponding to the information bottleneck as the query end in the cross-attention mechanism, key information related to the current analysis task is extracted from text, vision, time series and knowledge graph.

[0077] S803. Based on the extracted key information, calculate the single confidence score for each modality, and evaluate the single confidence score according to query explicitness, image text recognition quality / image quality, change significance, and retrieval matching degree respectively.

[0078] S804. Based on the single confidence level of each modality, a weighted fusion mechanism is used to dynamically allocate weights in the potential representation space corresponding to the information bottleneck, generate the final representation, and complete the fusion of potential bottlenecks.

[0079] Furthermore, S9 includes the following steps:

[0080] S901. Using the fusion result obtained in S8 as the input of shared features, design a multi-task prediction head, where the multi-task includes operation classification, confidence prediction and anomaly prediction.

[0081] S902. Based on the fusion results and the output of the multi-task prediction head, a multi-level method is used to quantify the uncertainty of the recognition results;

[0082] S903. Based on the fusion results, if the uncertainty exceeds the threshold, a clarification question is generated for the user, and the source of uncertainty is analyzed. If the uncertainty does not exceed the threshold, the drilling simulation operation identification result is output.

[0083] The present invention also provides a drilling simulation operation identification system, comprising:

[0084] The first processing module is used to construct a knowledge graph containing snapshots, operations, hardware, and parameters, and to generate node vector embeddings during the knowledge graph construction process. The snapshot layer is used to represent the overall configuration status information of the drilling simulation system at the start of a predetermined simulation task or at a specific analysis time. Each snapshot node is associated with multiple operation nodes, each operation node is connected to all hardware nodes related to it, and each hardware node is associated with all its related parameter nodes.

[0085] The second processing module is used for pre-training and adaptation of the multimodal encoder;

[0086] The third processing module is used to obtain the most relevant node to the question raised by the current user by generating three candidate node paths. In the process of generating three candidate nodes, an adapted multimodal encoder is called to encode the text, visual and parameter information and dynamic weights are used for allocation.

[0087] The fourth processing module is used to expand the relevant nodes based on the most relevant nodes to the problem using a dynamic intelligent subgraph, and to dynamically adjust the node vector embeddings added to the intelligent subgraph in real time using a dynamic context strategy, with the generated node vector embeddings as the initial values.

[0088] The fifth processing module is used to construct a multimodal similarity matrix based on dynamically adjusted node vector embeddings and text and visual features obtained by multimodal encoder.

[0089] The sixth processing module is used to construct a temporal bias matrix based on the multimodal similarity matrix by fusing temporal difference information;

[0090] The seventh processing module is used to calculate multimodal attention based on the temporal bias matrix;

[0091] The eighth processing module is used to fuse potential bottlenecks based on multimodal attention;

[0092] The ninth processing module is used to output the final prediction and confidence assessment results based on the fusion results, and complete the operation identification of the drilling simulation.

[0093] Furthermore, the expression for the loss function of the drilling simulation operation identification system is as follows:

[0094] ;

[0095] in, Represents the loss function for multiple components. express Experience weights, express Experience weights, express Experience weights, express Experience weights, Cross-entropy loss represents the classification of operation types. This represents the consistency loss of node embedding within a dynamic intelligent subgraph. This represents the cross-modal alignment loss between visual functional regions and text queries and knowledge graph nodes in images. This represents the regression loss from retrieving key drilling parameter values ​​from the screenshot. This represents the overall consistency loss between multimodal features and knowledge graph representations.

[0096] The beneficial effects of this invention are:

[0097] This invention, without requiring the client to disclose any internal core data or posing any risk of technical leakage, utilizes only legally collectible interface screenshots, image text recognition models to detect text, exportable time-series parameter logs, and student voice / text expressions to accurately identify student operational intentions, task stages, and erroneous behaviors in real time. This enables a true "AI drill instructor" that replaces human instructors, increasing training efficiency by 3 to 5 times, reducing instructor workload by more than 80%, and completely solving industry pain points.

[0098] This invention comprehensively utilizes four types of information: text, images (screenshots), student voice, and temporal parameters. It uses a multimodal attention mechanism for joint modeling, which, compared to traditional methods that rely on a single modality, can more accurately infer the user's actual operational behavior and improve the system's intelligence level.

[0099] This invention utilizes a pre-constructed snapshot-operation-hardware-parameter knowledge graph, enabling the system to align user input with known knowledge during the reasoning process, avoiding reliance on purely data-driven black-box predictions. When some modalities are missing or noise is present (such as image text recognition failure), the knowledge graph provides structured constraints, improving the robustness of reasoning.

[0100] This invention, by reading structured parameter data records from consecutive time points and introducing time-series modeling, can effectively capture the changing trends of system states (such as the evolution of parameter anomalies), thereby more accurately determining the user's current operational stage. Compared to static snapshot methods, this approach has significant advantages in anomaly detection and operational prediction.

[0101] When the model output is uncertain, this invention can proactively ask the user clarifying questions to guide them in supplementing necessary information. By employing a human-machine collaborative approach, it reduces the prediction error rate while simultaneously improving user experience and system usability. This invention is not only applicable to the teaching operation recognition of current systems but can also be extended to fields requiring multimodal information fusion and intelligent reasoning, such as aviation, drilling, and complex industrial equipment, demonstrating excellent scalability and application prospects. Attached Figure Description

[0102] Figure 1 This is a flowchart of the method of the present invention.

[0103] Figure 2 This is a flowchart of the present invention.

[0104] Figure 3 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0105] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0106] Example 1

[0107] like Figure 1 As shown, the present invention provides a drilling simulation operation identification method, the implementation method of which is as follows:

[0108] S1. Construct a knowledge graph containing snapshots, operations, hardware, and parameters, and generate node vector embeddings during the knowledge graph construction process. The snapshot layer represents the overall configuration state information of the drilling simulation system at the start of a predetermined simulation task or at a specific analysis time. Each snapshot node is associated with multiple operation nodes, each operation node is connected to all its associated hardware nodes, and each hardware node is associated with all its related parameter nodes. The implementation method is as follows:

[0109] S101. Construct a knowledge graph containing snapshot, operation, hardware, and parameters; S102. During the construction of the knowledge graph, collect text descriptions for each node, wherein the text descriptions include at least the node name and functional description; the nodes include snapshot nodes, operation nodes, hardware nodes, and parameter nodes; S103. Call a text encoder to encode the text description information to obtain the initial semantic embedding of each node; S104. Based on the initial semantic embedding and the node connection relationships in the knowledge graph, aggregate domain information for each node so that the generated node vector embedding simultaneously contains the semantic information of the node itself and the structural information of adjacent nodes;

[0110] S2. The multimodal encoder is pre-trained and adapted, and the implementation method is as follows:

[0111] S201. Use an object detection model to identify different functional areas in the drilling simulation operation interface to achieve region segmentation. S202. Based on the region segmentation results, use a pre-trained visual encoder to extract global features from the entire interface screenshot, generating an initial visual embedding representing the overall state of the interface. S203. Based on the region segmentation results, crop and encode the interface images corresponding to each functional area to obtain region-level visual features. Simultaneously, encode the extracted text content and its confidence level to generate corresponding text features. The region-level visual and text features are used in subsequent steps for multimodal candidate node generation and multimodal similarity calculation. S204. Use a temporal encoder to process the historical change patterns of parameters. The short-term mode captures rapid changes at the second level, the medium-term mode represents the direction, magnitude, and rate of parameter change within a preset time window, and the long-term mode analyzes the changes from the snapshot to the current drilling process, completing the pre-training and adaptation of the multimodal encoder. S202, S203, and S204 are parallel sub-processes. The object detection model is an existing YOLO series model.

[0112] S3. By generating three candidate node paths, the most relevant nodes to the question raised by the current user are obtained. In the process of generating the three candidate nodes, an adapted multimodal encoder is called to encode the text, visual and parameter information, and dynamic weights are used for allocation. The question raised by the current user is the question raised by the student user.

[0113] The three candidate node paths include: an exact matching path, used to process user-mentioned devices, parameters, or operations; a semantic retrieval path, used to process user natural language descriptions; and a visual-image text recognition (OCR) retrieval path, used to process interface information.

[0114] S4. Based on the most relevant nodes to the problem, expand the relevant nodes using a dynamic intelligent subgraph, and use a dynamic context strategy to dynamically adjust the node vector embeddings added to the intelligent subgraph in real time, using the node vector embeddings generated in S1 as initial values. The implementation method is as follows:

[0115] S401. Using the most relevant nodes to the problem as seed nodes, selectively expand using edge type and weight, prioritizing the expansion of the most relevant hardware nodes and parameter nodes from the operation nodes to generate a candidate context node set; S402. For the candidate context node set, calculate the node relevance probability by combining the temporal state information of the corresponding parameters, and select the most relevant node to add to the intelligent subgraph based on the node relevance probability. Stop the selection when the average value is lower than the threshold; S403. After obtaining the intelligent subgraph most relevant to the current query, select the node embedding belonging to the intelligent subgraph as the initial state based on the node vector embedding generated in S1, and use lightweight graph convolution to dynamically adjust the node vector embedding of each node in the intelligent subgraph so that the same node can obtain differentiated representations in different job contexts;

[0116] The dynamic adjustment is achieved as follows: In the first layer of graph convolution, for each node in the intelligent subgraph, the current node vector embedding is used as input. Each node performs a weighted aggregation of its own current embedding vector and the embedding vectors of all its neighboring nodes, enabling the node to perceive the directly related local context information. In the second layer of graph convolution, based on the node embedding vector updated by the first layer of graph convolution, each node performs a weighted aggregation of its own current embedding vector and the embedding vectors of all its neighboring nodes again, enabling the local context information to be propagated globally within a two-hop range in the intelligent subgraph, thus completing the dynamic adjustment. After the dynamic adjustment, the embedding vectors of each node in the intelligent subgraph have incorporated the context information.

[0117] S5. Based on dynamically adjusted node vector embeddings and text features and region-level visual features obtained by multimodal encoder, a multimodal similarity matrix is ​​constructed. The implementation method is as follows:

[0118] S501. Based on the dynamically adjusted node vector embedding in S4, the query text representing the current analysis target is encoded, the contextual information of each token is fused, and an enhanced representation is generated through a self-attention mechanism; the query text is the text information representing the current analysis target, which can come from user input, system-preset task descriptions, or be generated by parsing interface text; S502. Cosine similarity calculation is performed between the token embedding and the knowledge graph node embedding; S503. Based on the cosine similarity calculation result, node type weighting is added to enhance relevance, completing the text-knowledge graph similarity matrix. The calculation is as follows: S504. Based on the functional region type identified in S201, determine the weight initialization strategy for the text features and visual features corresponding to the region type; S505. Based on the weight initialization strategy, fuse the text embedding extracted by the image text recognition model and the visual embedding of the region generated by the target inspection model; S506. During fusion, dynamically adjust the initial weights according to the image sharpness and the confidence level of the image text recognition results to complete the calculation of the visual-knowledge graph similarity matrix. S507. By calculating the semantic association strength between the token and the visual region, the cross-modal correlation matrix is ​​completed. The computation; the image text recognition model and the object inspection model are based on existing CRNN models and existing YOLO series models.

[0119] S6. Based on the multimodal similarity matrix, a temporal bias matrix is ​​constructed by fusing temporal difference information. The implementation method is as follows:

[0120] S601. Collect and save parameter data generated during system operation; S602. Based on the collected parameter data, calculate the multi-timescale temporal differential features of the collected parameters at the same timescale as the parameters processed by the temporal encoder in S204, obtaining a differential temporal vector, including short-term differential changes, medium-term differential changes, and long-term differential changes; S204 learns "how to see changes," while S602 calculates "how much has changed specifically"; S204 utilizes the historical change patterns of the parameters processed by the temporal encoder for modeling multi-timescale temporal features and encoder adaptation; S602 calculates parameter changes using multiple timescales, performing differential calculations on the actually collected parameter data within the timescale and coding framework determined in S204 to obtain temporal differential features for online inference; S603. Calculate the activation intensity vector based on the differential temporal vector; S604. Generate a mask matrix according to the preset mapping table from fields to knowledge graph parameter node IDs. S605, Based on activation intensity vector and mask matrix The temporal activation vectors of the parameter nodes are calculated. The temporal difference information is used as a dynamic feature to be embedded in the parameter node; S606, based on the text-knowledge graph similarity matrix Visual-Knowledge Graph Similarity Matrix and temporal activation vectors Construct a cross-modal timing bias matrix;

[0121] S7. Based on the temporal bias matrix, multimodal attention is calculated, and the implementation method is as follows:

[0122] S701, Constructing Attention Bias S702, based on temporal bias matrix and attention bias Multimodal attention is calculated;

[0123] S8. Based on multimodal attention, potential bottlenecks are fused, and the implementation method is as follows:

[0124] S801. A fixed-size learnable query vector is used as the information bottleneck to limit the intermediate representation space for cross-modal information fusion. The fixed-size learnable query vector in S801 serves as the query end in the cross-attention mechanism, and in S802, it is used to selectively extract key information from four types of features: text, visual, temporal, and knowledge graph. S802. The query vector corresponding to the information bottleneck is used as the query end in the cross-attention mechanism to extract key information relevant to the current analysis task from text, visual, temporal, and knowledge graph features. S803. Based on the extracted key information, a single confidence score is calculated for each modality, and the single confidence score is evaluated based on query explicitness, image text recognition quality / image quality, change significance, and retrieval matching degree. S804. Based on the single confidence scores of each modality, a weighted fusion mechanism is used to dynamically allocate weights within the potential representation space corresponding to the information bottleneck to generate the final representation, thus completing the fusion of the potential bottleneck.

[0125] S9. Based on the fusion results, output the final prediction and confidence assessment results to complete the operation identification of the drilling simulation. The implementation method is as follows:

[0126] S901. Using the fusion result obtained in S8 as the shared feature input, design a multi-task prediction head, where the multi-task includes operation classification, confidence prediction, and anomaly prediction; S902. Based on the fusion result and the output of the multi-task prediction head, use a multi-level method to quantify the uncertainty of the identification result; S903. Based on the fusion result, when the uncertainty exceeds the threshold, generate a clarification question for the user and analyze the source of uncertainty; when the uncertainty does not exceed the threshold, output the drilling simulation operation identification result.

[0127] In this embodiment, the knowledge graph contains four types of nodes, presenting a four-layer structure. The snapshot layer represents the overall configuration state information of the drilling simulation system at the start of a predetermined simulation task or at a specific analysis moment. Within a snapshot, a predetermined operation is completed under specific settings. Each snapshot node is associated with multiple operation nodes, such as tripping the drill string, shutting off the pump, and adjusting the drilling pressure. Each operation node is connected to all related hardware nodes, which include all hardware such as the rotary table, hook, and drill string. Each piece of hardware is associated with all its related parameter nodes, such as rotation speed and pressure. The edge from snapshot to operation represents an "inclusion" relationship, the edge from operation to hardware represents an "involvement" relationship, and the edge from hardware to parameter represents a "monitoring" relationship. The complete model architecture diagram is shown below.Figure 2 As shown.

[0128] In this embodiment, a four-layer knowledge graph structure and its relationships for a closed drilling system are defined, namely, snapshot node → operation node (inclusion relationship), operation node → hardware device node (involvement relationship), hardware device node → parameter node (monitoring relationship), and a semantic modeling and retrieval mechanism based on this structure.

[0129] In this embodiment, the pre-training and adaptation process of the multimodal encoder is as follows: Based on the Qwen 3B model, this invention uses drilling-related technical documents and teaching materials for training, enabling the model to better understand technical terminology. Considering the special nature of the drilling control interface that the visual encoder needs to handle, which typically contains a large number of data curves, dashboards, etc., this invention uses a combination of object detection model (existing YOLO series models) and image text recognition model (existing CRNN model) for processing. First, the object detection model is used to identify different functional areas in the interface, including curve areas, parameter display areas, and dashboards, to achieve region segmentation. Then, the pre-trained visual encoder is used to extract global features from the interface screenshot to generate initial visual embeddings. For the detected regions, the images are cropped and encoded separately to obtain region-specific visual features. For text information, the image text recognition model is used to extract the text content and confidence level in the interface, and then encoded using the pre-trained Qwen model.

[0130] A time-series encoder is used to process historical variation patterns of parameters. Different parameters exhibit significantly different variation characteristics; for example, pressure parameters may change drastically, while temperature parameters may change relatively smoothly. This invention employs multi-scale time-series modeling: a short-term model captures rapid changes at the second level, a medium-term model captures trends at the minute level, and a long-term model analyzes macroscopic changes throughout the entire drilling process from the snapshot to the present.

[0131] In this embodiment, the three candidate nodes are generated as follows:

[0132] The precise path matching mainly handles devices, parameters, or operations directly mentioned by the user. However, it takes into account issues such as synonyms, abbreviations, and dialect expressions. For example, a user might directly say "pump pressure," and the system needs to understand that this refers to "mud pump pressure." Here, the Qwen model (a model with common meanings and specific meanings) trained on domain data is used to accomplish this.

[0133] Semantic retrieval processes users' natural language descriptions. A user might say, "Drilling feels very slow." This description doesn't directly name the equipment, but it implies a connection to factors like drill pressure, rotational speed, and drill string assembly. Semantic retrieval requires understanding these implicit relationships. Through query expansion techniques, the original query is expanded into multiple related technical terms, which are then retrieved separately.

[0134] The visual-image text recognition (OCR) retrieval path processes interface information. The information in the interface screenshots often best reflects the current system status. The text information extracted by the image text recognition model (existing CRNN model) includes parameter values, alarm information, and device status, while visual information includes curve trends, color changes, and graphic symbols. The fusion of these two types of information needs to consider their reliability differences. The image text recognition model (existing CRNN model) is accurate with clear text but prone to errors with blurry or slanted text; visual features are sensitive to overall layout and trends but not precise enough for specific numerical values. A three-way parallel candidate node generation mechanism (precise matching path + semantic retrieval path + visual-image text recognition OCR retrieval path) combined with a small neural network adaptively outputs fusion weights based on the quality indicators of each path.

[0135] Dynamic weight allocation is adopted: when the query is specific, the weight of exact matching is increased; when the description is ambiguous, the weight of semantic retrieval is increased; when the interface is abnormal, the weight of visual retrieval is increased. The weights are adjusted through a small neural network. The inputs are various quality indicators (correctness / coverage of exact matching, similarity distribution of semantic retrieval, confidence / feature stability of the image text recognition model (existing CRNN model) for visual retrieval), and the output is normalized weights to achieve adaptive fusion.

[0136] In this embodiment, the dynamic intelligent subgraph unfolding and dynamic context strategy are as follows: Based on edge type and weight selective k-hop (K-hop) unfolding, hardware nodes and parameter nodes with high relevance are preferentially expanded from operation nodes to avoid irrelevant branches; the node relevance probability is calculated according to the temporal parameter state, and relevant nodes are given priority. A greedy algorithm is used to select highly relevant nodes to be added to the intelligent subgraph, and the process stops when the average value is lower than the threshold. After obtaining the intelligent subgraph that is highly relevant to the current query, a lightweight graph convolutional network is further used to dynamically adjust the embedding vector of each node in the subgraph in real time, so that the same node can obtain differentiated representations under different operational contexts (different snapshots, geological conditions, operation stages). Specifically, the dynamic adjustment process includes the following two layers of graph convolutional operations:

[0137] First layer graph convolution: For each node in the intelligent subgraph, the current node vector embedding is used as input. Each node performs a weighted aggregation of its own current embedding vector and the embedding vectors of all its direct neighboring nodes (1-hop neighbors) (preferably, the product of a normalized adjacency matrix and a learnable linear transformation matrix is ​​used), so that the node can quickly perceive the local context information directly related to it.

[0138] The second layer of graph convolution: Based on the node embedding updated by the first layer of graph convolution, the same aggregation operation is performed again, so that information can be propagated globally within a two-hop range in the subgraph, and the context information can be fully penetrated. The dynamic subgraph selective k-hop unfolding algorithm based on the greedy-diversity criterion is used, and two layers of lightweight graph convolutional network are applied to the unfolded intelligent subgraph to achieve context-adaptive node embedding adjustment.

[0139] Since the aforementioned steps have controlled the subgraph size to a small range (usually less than 300 nodes) through a greedy-diversity algorithm and only perform two layers of graph convolution operations, the computational cost of the entire dynamic adjustment process is extremely small and can be completed in milliseconds, meeting the requirements of real-time inference.

[0140] After the two layers of lightweight graph convolution processing described above, the embedding vectors of each node in the intelligent subgraph have been fully integrated with contextual information such as the current snapshot type, operation stage, and real-time parameter status. For example, the final embedding vector of the "drill pressure" parameter node in the context of "shale layer sliding drilling" will be significantly different from its embedding vector in the "start drilling" stage, thereby greatly improving the accuracy of subsequent similarity calculation and operation recognition.

[0141] In this embodiment, the multimodal similarity matrix is ​​constructed using a multi-bias injection mechanism consisting of text-knowledge graph similarity, visual-knowledge graph similarity, and temporal difference bias matrix, which guides cross-modal attention calculation, as detailed below:

[0142] Implementing a text-knowledge graph similarity matrix During computation, a context-aware approach is employed. The query text is encoded using the Qwen model (a model for general understanding of the meaning of queries), and contextual information of each token is integrated to generate an enhanced representation through a self-attention mechanism. During matching, cosine similarity is calculated between the token embeddings and the knowledge graph node embeddings, with node type weighting added to enhance relevance. Temperature parameters are dynamically set, making the distribution sharper when the query is explicit and smoother when it is ambiguous, automatically adjusting based on the query entity.

[0143] Implementing a visual-knowledge graph similarity matrix At the same time, visual features extracted by the image text recognition model (existing CRNN model) and the object detection model (existing YOLO series models) are integrated: the image text recognition model (existing CRNN model) extracts text embeddings, and the object detection model (existing YOLO series models) generates region visual embeddings. Based on the readability differences between text information and visual appearance in different functional areas, reasonable initial fusion weights are set for text features and visual features, and dynamically adjusted during fusion based on image clarity and the confidence level of the image text recognition model. Specifically: the data display area primarily uses text features with visual features as a secondary factor; the curve graph area primarily uses visual features with text features as a secondary factor; and the alarm area uses a balanced fusion of both.

[0144] Achieving cross-modal correlation matrix The system calculates the semantic association strength between text tokens and visual regions. For example, if a user says "This stress value is incorrect," the system needs to understand which specific numerical value displayed on the interface refers to that "stress value." By using a knowledge graph as an intermediary, the system can establish a connection between the concept of stress value in the text and the area on the interface that displays stress data. To ensure sparsity, a threshold is set; values ​​below the threshold are set to 0 to reduce noise.

[0145] In this embodiment, the fusion of time-series differential information is as follows: When implementing time-series differential fusion, system data is collected and saved periodically at specific stages of the user's operation, and parameter changes are analyzed and calculated using multi-time-scale analysis:

[0146] Short-term changes: Capture instantaneous operational effects; medium-term changes: Reflecting progress trends; long-term changes: This reflects the macro strategy, while the above all pertain to the current state of the local parameters. This indicates the system's parameter status when the current user asks a question. This indicates the parameter status information stored in the database in the previous stage. This represents the parameter status information stored in the database during the first three stages. This represents the original parameter information of the snapshot. The difference time series vector is concatenated and input into the MLP neural network to obtain an activation intensity vector:

[0147] ;

[0148] To ensure that activation only occurs on the correct parameter nodes, the system generates a mask matrix based on the parameter fields actually appearing in the current structured parameter data record file, using a pre-defined "field to knowledge graph parameter node ID" mapping table (e.g., the system uses a pre-defined "JSON field to knowledge graph parameter node ID" mapping table based on the parameter fields actually appearing in the current JSON file). The final temporal activation vector is obtained by multiplying the activation vector element-wise with the mask matrix. This allows the difference information to be embedded as a dynamic feature, precisely corresponding to the parameter node. Subsequently, temporal activation vectors are used... Construct the timing bias matrix:

[0149] ;

[0150] in, Represents the timing bias matrix. Represents the text-knowledge graph similarity matrix. Represents the visual-knowledge graph similarity matrix. This represents a combination of parameter nodes, timing nodes, and timing activation vectors. d Constructed diagonal matrix, This represents a temporal activation vector that corresponds one-to-one with a parameter node in the knowledge graph, used to characterize the dynamic importance of each parameter node under changes across multiple time scales. This represents an MLP neural network. Indicates short-term changes. Indicates medium-term changes. Indicates long-term change. This indicates the system's parameter status when the current user asks a question. This indicates the parameter status information stored in the database in the previous stage. This represents the parameter status information stored in the database during the first three stages. This indicates the original parameter information of the snapshot. This represents the set of knowledge graph parameter nodes related to the current task. express transpose, This indicates transpose.

[0151] This matrix enables the text and visual regions to have higher attention weights on the parameter as the parameter changes more abnormally, significantly enhancing the model's sensitivity to abnormal trends and its ability to understand context.

[0152] In this embodiment, the knowledge graph guides attention as follows: When implementing the attention mechanism, a multi-bias coordination mechanism is used to inject semantic associations. First, attention biases are constructed. :

[0153] ;

[0154] Through logarithmic transformation and a learnable small linear layer Suppressing excessive correlation and normalizing to ensure even distribution and avoid strong correlation dominance; cross-modal attention uses text as the query and visual information as the key / value pair.

[0155] ;

[0156] in, This represents multimodal attention. This represents the query vector obtained by linear transformation of text features. This represents the key vector obtained by linear transformation of visual features. Indicates transpose. This represents the feature dimensions of the query vector and the key vector. express Learnable weight parameters, express Learnable weight parameters, These represent the word index in the text feature sequence and the region index in the visual feature set at the functional region level, respectively. Represents the timing bias matrix. Indicates position code, Indicates a small linear layer. This represents a nonlinear compression function. Represents the cross-modal correlation matrix. Represents the natural logarithm operation. This represents the modified linear unit activation function.

[0157] In this embodiment, the potential bottleneck fusion uses a fixed number (32-128) of learnable query vectors as the information bottleneck. Key information is dynamically extracted and weighted from multimodal features and knowledge graph subgraphs through cross-attention, as follows:

[0158] When achieving potential bottleneck fusion, a fixed-size learnable query vector is used as the information bottleneck, with initial values ​​based on data cluster centers representing typical operating patterns. Key information is extracted from text, vision, time series, and knowledge graphs through a cross-attention mechanism. A single confidence score is calculated for each modality, evaluated based on query explicitness, image text recognition quality / image quality, change significance, and retrieval matching degree. Fusion dynamically allocates weights through a basic weighting mechanism, prioritizing complementary information and reducing redundancy to directly generate the final representation.

[0159] In this embodiment, the final prediction and confidence assessment are achieved by a multi-task prediction head that simultaneously outputs operation classification, anomaly detection, and comprehensive confidence, as well as an uncertainty assessment mechanism based on three levels of quantification: cognitive uncertainty, accidental uncertainty, and task uncertainty, and a strategy that triggers the generation of clarification questions to the user when the uncertainty exceeds a threshold, as detailed below:

[0160] To achieve the final prediction, a multi-task prediction head is designed, covering operation classification, confidence prediction, and anomaly prediction. A shared feature extractor is used to ensure consistency, and the prediction head is independently optimized to adapt to task characteristics. Uncertainty is quantified using a multi-level approach: cognitive uncertainty is estimated through random deactivation Dropout or ensemble learning; random uncertainty is determined through data quality assessment; task uncertainty is derived through statistical analysis of historical data; and the overall confidence level is used to determine whether to intervene manually or request information.

[0161] Example 2

[0162] like Figure 3 As shown, the present invention provides a drilling simulation operation identification system for executing the drilling simulation operation identification method described in Embodiment 1, comprising:

[0163] The first processing module is used to construct a knowledge graph containing snapshots, operations, hardware, and parameters, and to generate node vector embeddings during the knowledge graph construction process. The snapshot layer is used to represent the overall configuration status information of the drilling simulation system at the start of a predetermined simulation task or at a specific analysis time. Each snapshot node is associated with multiple operation nodes, each operation node is connected to all hardware nodes related to it, and each hardware node is associated with all its related parameter nodes.

[0164] The second processing module is used for pre-training and adaptation of the multimodal encoder;

[0165] The third processing module is used to obtain the most relevant node to the question raised by the current user by generating three candidate node paths. In the process of generating three candidate nodes, an adapted multimodal encoder is called to encode the text, visual and parameter information and dynamic weights are used for allocation.

[0166] The fourth processing module, based on the relevant nodes most relevant to the problem, expands the relevant nodes using a dynamic intelligent subgraph and uses a dynamic context strategy to dynamically adjust the node vector embeddings added to the intelligent subgraph in real time, using the generated node vector embeddings as initial values.

[0167] The fifth processing module is used to construct a multimodal similarity matrix based on dynamically adjusted node vector embeddings and text and visual features obtained by multimodal encoder.

[0168] The sixth processing module is used to construct a temporal bias matrix based on the multimodal similarity matrix by fusing temporal difference information;

[0169] The seventh processing module is used to calculate multimodal attention based on the temporal bias matrix;

[0170] The eighth processing module is used to fuse potential bottlenecks based on multimodal attention;

[0171] The ninth processing module is used to output the final prediction and confidence assessment results based on the fusion results, and complete the operation identification of the drilling simulation.

[0172] In this embodiment, the training loss and optimization strategy are multi-component loss functions:

[0173] ;

[0174] in, Represents the loss function for multiple components. express Experience weights, express Experience weights, express Experience weights, express The empirical weights typically range from 0.1 to 2. The cross-entropy loss, representing the operation type classification, is used as the main task loss. The consistency loss for dynamic intelligent subgraph node embedding is represented by a triple-based contrastive loss, ensuring that relevant nodes are represented closer together and irrelevant nodes are represented further apart. The cross-modal alignment loss between visual functional regions and text queries and knowledge graph nodes in images is represented by a contrastive learning approach. This represents the regression loss from retrieving key drilling parameter values ​​from the screenshot. The overall consistency loss between multimodal features and knowledge graph representation is represented by a multi-path distributed alignment loss.

[0175] like Figure 2 As shown, this invention aims at operation recognition, comprehensively utilizing user input, system front-end interface information, and time-series data (system parameter information saved at each time interval) to achieve external recognition of user operations and system status in a drilling simulation system. The system first receives user input information. User input can be fuzzy, such as descriptive statements like "the drilling speed feels very slow," or precise, where the user directly mentions equipment, parameters, or operational behaviors. When the user input is fuzzy, the system performs semantic retrieval, retrieving semantically related implicit node information from a knowledge graph; when the user input is precise, the system directly parses the explicitly mentioned equipment, parameters, or operational content from the user input.

[0176] While receiving user input, the system acquires a screenshot of the current drilling simulation system's front-end interface (i.e., the "acquiring system front-end screenshot" in the figure) to reflect the system's real-time operating status. The system front-end screenshot includes visual content such as parameter display areas, curve display areas, instrument panel areas, and status prompts. Based on the user input parsing results and the system front-end screenshot information, the system performs a three-way candidate node path generation process (i.e., the three-way candidate node generation (obtaining relevant nodes) in the figure), thereby obtaining a set of candidate nodes related to the current user input and system status.

[0177] After obtaining candidate nodes, the system performs dynamic intelligent subgraph expansion processing. First, starting with the candidate node in the knowledge graph, a first convolution operation selects the N most relevant neighbor nodes of the pre-candidate node. Then, starting from the neighbor nodes selected in the previous round, a second convolution operation selects the N most relevant neighbor nodes for the current task (i.e., selecting the N most relevant neighbor nodes again in the graph), thus constructing a dynamic intelligent subgraph highly relevant to the current recognition task. Based on the intelligent subgraph, the system constructs a text-knowledge graph subgraph similarity matrix (i.e., a text-knowledge graph similarity matrix). ) and the visual-knowledge graph subgraph similarity matrix (i.e., the visual-knowledge graph similarity matrix) Simultaneously, by combining the correspondence between textual information and front-end interface visual information, a text-visual similarity matrix (i.e., a cross-modal correlation matrix) is constructed. ).

[0178] On the other hand, the system retrieves system parameter information saved at each time point from the time-series data database, and calculates short-term, medium-term, and long-term changes in the parameters based on the time-series data. Short-term changes reflect rapid changes between adjacent time points, medium-term changes reflect trend changes within multiple time windows, and long-term changes reflect the overall changes in the parameters from the initial state of the snapshot to the current moment. Based on the results of short-term, medium-term, and long-term changes, the system constructs a time-series bias matrix to characterize the impact of parameter changes on the current operation recognition process.

[0179] Building upon this foundation, the system performs multimodal attention calculations on the text-knowledge graph subgraph similarity matrix, the visual-knowledge graph subgraph similarity matrix, the text-visual similarity matrix, and the temporal bias matrix. This multimodal attention mechanism highlights the text, visual, and knowledge graph information most relevant to the current operation. Subsequently, based on the multimodal attention calculation results, the system performs potential bottleneck fusion processing. This involves compressing and integrating multimodal information through a fixed-size potential information bottleneck to generate a unified high-level feature representation, which is then used to produce the model's internal output.

[0180] The system assesses the confidence level of the model's internal output. When the recognition result is below the preset confidence threshold, the system returns to the front-end input or information acquisition stage for further analysis or interaction. When the recognition result meets the confidence requirements, the system outputs the final drilling simulation operation recognition result, thus completing the entire operation recognition process.

[0181] In summary, this invention achieves the following objectives by combining external signals such as visual information (interface screenshots), text output from image text recognition models, and temporal parameters with the semantic structure of a knowledge graph: accurately identifying user operation types and system states when internal system data is inaccessible; realizing semantic modeling between job snapshots, operations, devices, and parameters through structured associations in a knowledge graph; improving the robustness and interpretability of recognition in closed environments by integrating visual, textual, and temporal multimodal features; and enhancing the model's adaptability and stability to multi-source data through dynamic weights and temporal difference mechanisms.

[0182] In this embodiment, an alternative to the knowledge graph is to replace the four-layer heterogeneous knowledge graph with an ontology-based rule reasoning system (based on ontology rules) or a relational database + trigger system. This can also achieve semantic association and query between snapshot, operation, hardware and parameters, but sacrifices some automatic learning capabilities in exchange for higher interpretability.

[0183] Alternatives to three-way candidate generation and adaptive fusion include: replacing "three-way parallelism + dynamic weighting by a small neural network" with rule-based priority routing (e.g., if the interface has an alarm image → prioritize the visual path; if the user's voice contains explicit parameter names → prioritize the precise matching path); or using a Bayesian network / decision tree to perform probabilistic fusion based on the confidence of each path; or directly using a single path + post-processing voting mechanism.

[0184] Alternatives to dynamic subgraph unfolding: Instead of the greedy-diversity algorithm, we can use attention sparsification (Top-K neighbors) after a fixed k-hop (k-hop) full unfolding; or use full-graph FAISS (Facebook AI similarity search library) vector retrieval for nearest neighbor nodes (sacrificing some structural information for speed).

[0185] Alternatives to multi-biased cross-modal attention: standard Transformer encoder with hand-designed inter-modal gating mechanism; or directly embedding knowledge graph nodes and concatenating them with multimodal features before feeding them into an MLP neural network.

[0186] Alternatives to fixed-size learnable bottleneck query vectors include: traditional global average pooling + multi-layer MLP neural networks; using learnable class prototype vectors (Prototype Learning); using RNN / LSTM for temporal compression of multimodal sequences; and directly using the [CLS] token of a large model as a fusion representation.

[0187] Complete alternative technical solutions: pure visual large models (such as GPT-4V, LLaVA-Next) directly perform end-to-end inference on interface screenshots and voice commands; traditional expert systems based on template matching + image and text recognition rule engines + temporal state machines; and reinforcement learning agents learn the mapping relationship between external signals and operations through trial and error.

Claims

1. A method for identifying drilling simulation operations, characterized in that, Includes the following steps: S1. Construct a knowledge graph containing snapshots, operations, hardware, and parameters, and generate node vector embeddings during the knowledge graph construction process. The snapshot layer is used to represent the overall configuration status information of the drilling simulation system at the start of a predetermined simulation task or at a fixed analysis time. Each snapshot node is associated with multiple operation nodes, each operation node is connected to all hardware nodes related to it, and each hardware node is associated with all its related parameter nodes. S2. Perform pre-training and adaptation processing on the multimodal encoder; S3. By generating three candidate node paths, the most relevant nodes related to the current user's question are obtained. In the process of generating three candidate nodes, an adapted multimodal encoder is called to encode the text, visual and parameter information, and dynamic weights are used for allocation. S4. Based on the relevant nodes most relevant to the problem, expand the relevant nodes using a dynamic intelligent subgraph, and use a dynamic context strategy to dynamically adjust the node vector embeddings added to the intelligent subgraph in real time, using the node vector embeddings generated in S1 as the initial values. S5. Construct a multimodal similarity matrix based on dynamically adjusted node vector embeddings and text features and region-level visual features obtained by multimodal encoder. S6. Based on the multimodal similarity matrix, a temporal bias matrix is ​​constructed by fusing temporal difference information; S7. Calculate multimodal attention based on the temporal bias matrix; S8, based on multimodal attention, integrates potential bottlenecks; S8 includes the following steps: S801. A fixed-size learnable query vector is used as an information bottleneck to limit the intermediate representation space for cross-modal information fusion. S802. Using the query vector corresponding to the information bottleneck as the query end in the cross-attention mechanism, key information related to the current analysis task is extracted from text, vision, time series and knowledge graph. S803. Based on the extracted key information, calculate the single confidence score for each modality, and evaluate the single confidence score according to query explicitness, image text recognition quality / image quality, change significance, and retrieval matching degree respectively. S804. Based on the single confidence level of each modality, the weights are dynamically allocated in the potential representation space corresponding to the information bottleneck through a weighted fusion mechanism to generate the final representation and complete the fusion of potential bottlenecks. S9. Based on the fusion results, output the final prediction and confidence assessment results to complete the operation identification of the drilling simulation.

2. The drilling simulation operation identification method according to claim 1, characterized in that, S1 includes the following steps: S101. Construct a knowledge graph that includes snapshots, operations, hardware, and parameters; S102. During the construction of the knowledge graph, a text description is collected for each node. The text description includes at least the node name and functional description. The node includes snapshot nodes, operation nodes, hardware nodes, and parameter nodes. S103. Call the text encoder to encode the text description information to obtain the initial semantic embedding of each node; S104. Based on the initial semantic embedding and the node connection relationship in the knowledge graph, the domain information of each node is aggregated so that the generated node vector embedding contains both the semantic information of the node itself and the structural information of the adjacent nodes.

3. The drilling simulation operation identification method according to claim 1, characterized in that, S2 includes the following steps: S201. Identify different functional areas in the drilling simulation operation interface to achieve area segmentation processing; S202. Based on the region segmentation results, a pre-trained visual encoder is used to extract global features from the entire interface screenshot to generate an initial visual embedding that represents the overall state of the interface. S203. Based on the region segmentation results, the interface images corresponding to each functional region are cropped and encoded separately to obtain region-level visual features. At the same time, the text content extracted from the interface and its confidence level are encoded to generate corresponding text features. S204. The historical change patterns of parameters are processed using a time-series encoder. The short-term mode captures rapid changes at the second level, the medium-term mode is used to characterize the direction, magnitude and rate of parameter change within a preset time window, and the long-term mode analyzes the changes from the snapshot to the current drilling process, thus completing the pre-training and adaptation of the multimodal encoder.

4. The drilling simulation operation identification method according to claim 1, characterized in that, The three candidate node paths in S3 include: Precise path matching is used to process user-mentioned devices, parameters, or operations; Semantic retrieval path, used to process users' natural language descriptions; The visual-image text recognition (OCR) retrieval path is used to process interface information.

5. The drilling simulation operation identification method according to claim 1, characterized in that, S4 includes the following steps: S401. Take the most relevant nodes to the problem as seed nodes, and selectively expand them using edge type and weight. Prioritize expanding the most relevant hardware nodes and parameter nodes from the operation nodes to generate a set of candidate context nodes. S402. For the candidate context node set, calculate the node correlation probability by combining the temporal state information of the corresponding parameters, and select the node with the highest correlation based on the node correlation probability to add to the smart subgraph. Stop the selection when the average correlation probability of the selected nodes is lower than the threshold. S403. After obtaining the smart subgraph most relevant to the current query, based on the node vector embedding generated in S1, select the node embedding belonging to the smart subgraph as the initial state, and use lightweight graph convolution to dynamically adjust the node vector embedding in the smart subgraph so that the same node can obtain differentiated representations in different job contexts.

6. The drilling simulation operation identification method according to claim 5, characterized in that, The dynamic adjustment includes the following steps: In the first layer of graph convolution, for each node in the intelligent subgraph, its current node vector embedding is used as input. Each node performs weighted aggregation processing on its own current embedding vector and the embedding vectors of all its neighboring nodes, so that the node can perceive the directly related local context information. In the second layer of graph convolution, based on the node embedding vector updated by the first layer of graph convolution, each node performs a weighted aggregation process on its own current embedding vector and the embedding vectors of all its neighboring nodes, so that the local context information can be propagated globally within a two-hop range in the intelligent subgraph and dynamically adjusted. After the dynamic adjustment, the embedding vectors of each node in the intelligent subgraph have been integrated with the context information.

7. The drilling simulation operation identification method according to claim 3, characterized in that, S5 includes the following steps: S501: Based on the node vector embedding dynamically adjusted by S4, the query text used to represent the current analysis target is encoded, the context information of each word token is fused, and an enhanced representation is generated through a self-attention mechanism; S502. Calculate the cosine similarity between the word token embedding and the knowledge graph node embedding. S503. Based on the cosine similarity calculation results, node type weighting is added to enhance relevance, completing the text-knowledge graph similarity matrix. Calculation; S504. Based on the functional area type identified in S201, determine the weight initialization strategy for the text features and visual features of the area type. S505. Based on the weight initialization strategy, the text embedding and the visual embedding of the corresponding region are weighted and fused. S506. During fusion, the initial weights are dynamically adjusted based on image clarity and the confidence level of image-text recognition results to complete the fusion of the visual-knowledge graph similarity matrix. Calculation; S507. By calculating the semantic association strength between word tokens and visual regions, complete the cross-modal relevance matrix analysis. The calculation.

8. The drilling simulation operation identification method according to claim 3, characterized in that, S6 includes the following steps: S601. Collect and save parameter data generated during system operation; S602. Based on the collected parameter data, at the same time scale as the parameters processed by the S204 time encoder, calculate the multi-time-scale temporal differential features of the collected parameters to obtain the differential time-series vector, which includes short-term differential changes, medium-term differential changes and long-term differential changes. S603. Based on the differential time-series vector, the activation intensity vector is calculated; S604. Generate a mask matrix based on the preset mapping table from fields to knowledge graph parameter node IDs. ; S605, Based on activation intensity vector and mask matrix The temporal activation vectors of the parameter nodes are calculated. This allows for the use of temporal difference information as dynamic features embedded in parameter nodes. S606, Based on text-knowledge graph similarity matrix Visual-Knowledge Graph Similarity Matrix and temporal activation vectors Construct a cross-modal timing bias matrix.

9. The drilling simulation operation identification method according to claim 8, characterized in that, The expression for the timing bias matrix is ​​as follows: ； ； ； ； ； in, Represents the timing bias matrix. Represents the text-knowledge graph similarity matrix. Represents the visual-knowledge graph similarity matrix. This represents a combination of parameter nodes, timing nodes, and timing activation vectors. d Constructed diagonal matrix, This represents a temporal activation vector that corresponds one-to-one with a parameter node in the knowledge graph. This represents an MLP neural network. Indicates short-term changes. Indicates medium-term changes. Indicates long-term change. This indicates the system's parameter status when the current user asks a question. This indicates the parameter status information stored in the database in the previous stage. This represents the parameter status information stored in the database during the first three stages. This indicates the original parameter information of the snapshot. This represents the set of knowledge graph parameter nodes related to the current task. express transpose, This indicates transpose.

10. The drilling simulation operation identification method according to claim 3, characterized in that, S7 includes the following steps: S701, Constructing Attention Bias ; S702, based on temporal bias matrix and attention bias Multimodal attention is calculated.

11. The drilling simulation operation identification method according to claim 10, characterized in that, The expression for the multimodal attention is as follows: ； ； in, This represents multimodal attention. This represents the query vector obtained by linear transformation of text features. This represents the key vector obtained by linear transformation of visual features. Indicates transpose. This represents the feature dimensions of the query vector and the key vector. express Learnable weight parameters, express Learnable weight parameters, These represent the word index in the text feature sequence and the region index in the visual feature set at the functional region level, respectively. Represents the timing bias matrix. Indicates position code, Indicates a small linear layer. This represents a nonlinear compression function. Represents the cross-modal correlation matrix. Represents the natural logarithm operation. This represents the modified linear unit activation function.

12. The drilling simulation operation identification method according to claim 1, characterized in that, S9 includes the following steps: S901. Using the fusion result obtained in S8 as the input of shared features, design a multi-task prediction head, where the multi-task includes operation classification, confidence prediction and anomaly prediction. S902. Based on the fusion results and the output of the multi-task prediction head, a multi-level method is used to quantify the uncertainty of the recognition results; S903. Based on the fusion results, if the uncertainty exceeds the threshold, a clarification question is generated for the user, and the source of uncertainty is analyzed. If the uncertainty does not exceed the threshold, the drilling simulation operation identification result is output.

13. A drilling simulation operation identification system, used to execute the drilling simulation operation identification method according to any one of claims 1-12, characterized in that, include: The first processing module is used to construct a knowledge graph containing snapshots, operations, hardware, and parameters, and to generate node vector embeddings during the knowledge graph construction process. The snapshot layer is used to represent the overall configuration status information of the drilling simulation system at the start of a predetermined simulation task or at a fixed analysis time. Each snapshot node is associated with multiple operation nodes, each operation node is connected to all hardware nodes related to it, and each hardware node is associated with all its related parameter nodes. The second processing module is used for pre-training and adaptation of the multimodal encoder; The third processing module is used to obtain the most relevant node to the current user's question by generating three candidate node paths. In the process of generating three candidate nodes, an adapted multimodal encoder is called to encode the text, visual and parameter information and dynamic weights are used for allocation. The fourth processing module is used to expand the relevant nodes based on the most relevant nodes to the problem using a dynamic intelligent subgraph, and to dynamically adjust the node vector embeddings added to the intelligent subgraph in real time using a dynamic context strategy, with the generated node vector embeddings as the initial values. The fifth processing module is used to construct a multimodal similarity matrix based on dynamically adjusted node vector embeddings and text and visual features obtained by multimodal encoder. The sixth processing module is used to construct a temporal bias matrix based on the multimodal similarity matrix by fusing temporal difference information; The seventh processing module is used to calculate multimodal attention based on the temporal bias matrix; The eighth processing module is used to fuse potential bottlenecks based on multimodal attention; The ninth processing module is used to output the final prediction and confidence assessment results based on the fusion results, and complete the operation identification of the drilling simulation.

14. The drilling simulation operation identification system according to claim 13, characterized in that, The expression for the loss function of the drilling simulation operation identification system is as follows: ； in, Represents the loss function for multiple components. express Experience weights, express Experience weights, express Experience weights, express Experience weights, Cross-entropy loss represents the classification of operation types. This represents the consistency loss of node embedding within a dynamic intelligent subgraph. This represents the cross-modal alignment loss between visual functional regions and text queries and knowledge graph nodes in images. This represents the regression loss from retrieving key drilling parameter values ​​from the screenshot. This represents the overall consistency loss between multimodal features and knowledge graph representations.

Images