Dynamic guidance method and device based on job deviation risk classification

Abstract

The application provides a dynamic guidance method and device based on job deviation risk grading, and relates to the technical field of intelligent operation and maintenance of mines. The method comprises the following steps: obtaining a structured standard job process model, analyzing sequential constraint relationships and conditional constraint relationships; obtaining job state data in real time through multi-source perception fusion in the execution task; identifying job deviation types and deviation risk levels based on job states, process nodes, job actions and constraint relationships; dynamically matching guidance strategies based on the current node, the deviation type and the risk level; and guiding the deviation correction of the job personnel through multi-modal fusion output of the mixed reality terminal. The application takes the structured process model as a dynamic benchmark, realizes automatic deviation identification and risk grading in combination with multi-source perception, matches differentiated guidance strategies and outputs in multiple modes, overcomes the defects of fixed process guidance and disconnection from the scene, weak non-differential prompt control, and significantly improves the standardization and safety of high-risk operations.

Description

Technical Field

[0001] This application relates to the field of intelligent operation and maintenance technology in mines, and in particular to a dynamic guidance method and device based on operational deviation risk classification. Background Technology

[0002] In high-risk operations such as mine equipment inspection and maintenance, the compliant execution of Standard Operating Procedures (SOPs) is crucial to ensuring personnel safety and stable equipment operation. Currently, intelligent guidance systems such as augmented reality have been deployed in industrial operation and maintenance scenarios, assisting operators in standardized operations through visual guidance.

[0003] However, related technologies generally adopt a fixed-process push mode, which can only display information one-way based on preset static text, resulting in significant drawbacks. Such systems cannot perceive the actual execution status of on-site workers and lack the ability to identify operational deviations such as omissions in procedures or disordered sequences in real time, leading to a serious disconnect between guidance information and the actual work situation. Static, undifferentiated, and single-modal guidance methods can no longer meet the stringent requirements of dynamic perception and precise control in high-risk work scenarios. Summary of the Invention

[0004] This application aims to at least partially address one of the technical problems in the related art.

[0005] Therefore, the first aspect of this application proposes a dynamic guidance method based on the risk classification of operational deviations, comprising the following steps:

[0006] Obtain a structured standard operating procedure model corresponding to the current inspection and maintenance task. The structured standard operating procedure model includes at least a set of process nodes, a set of work actions, and a set of safety constraints. Based on the structured standard operating procedure model, parse out the sequential constraint relationship used to define the execution logic between process nodes and the condition constraint relationship used to define the prerequisite state that the node execution must meet. During the process of the operator performing the current maintenance task according to the structured standard operating procedure model, the operation status data is acquired in real time through multi-source sensing fusion. Based on the job status data, the set of process nodes, the set of job actions, the sequential constraints, and the conditional constraints, the job deviation type and deviation risk level are identified in real time. Based on the current process node, the type of job deviation, and the level of deviation risk, the corresponding guidance strategy is dynamically matched using a pre-set guidance strategy matching function. The guidance strategy is executed through a mixed reality terminal in a multimodal fusion output mode to correct and guide the workers.

[0007] In some embodiments of this application, the real-time acquisition of work status data through multi-source perception fusion includes: acquiring the work status data through at least one of the visual acquisition module, motion capture module, terminal operation log, and device sensors of the mixed reality terminal; wherein, the work status data includes: the operation behavior data of the operator, equipment status data, and on-site environmental data.

[0008] In some embodiments of this application, the job deviation type includes at least one of the following: Process node omission deviation is used to indicate that a process node in the set of process nodes was not executed or the execution result did not meet the standard. Job sequence mismatch deviation is used to indicate that process nodes are not executed according to the specified sequence constraints; Non-standard operation deviation is used to indicate that the characteristics of the operator's operation are not matched with the action requirements in the set of operation actions; The safety confirmation missing deviation indicates that the preliminary safety confirmation step in the conditional constraint relationship has not been completed.

[0009] In some embodiments of this application, the non-standard deviation of the operation action is identified by: acquiring the operation video stream of the operator through the visual acquisition module; extracting the operation action features of the operator using a lightweight target detection algorithm and human key point capture technology; and comparing the operation action features with the action requirements in the set of operation actions to identify non-standard actions.

[0010] In some embodiments of this application, the deviation risk level includes general deviation, significant deviation, and high-risk deviation; wherein, the high-risk deviation includes at least one of the following: electrical violation, missing interlocking device, and inadequate key safety confirmation.

[0011] In some embodiments of this application, the guidance strategy includes: When the deviation risk level is a general deviation, a prompting guidance strategy is implemented, and the operator is informed through voice or text prompts on the mixed reality terminal. When the type of work deviation is a process node omission deviation, and the deviation risk level is a general deviation or a major deviation, a remedial guidance strategy is implemented to guide the operator to return to the missed node to perform the remedial operation. When the type of work deviation is work sequence mismatch and the risk level of deviation is general deviation or major deviation, a rollback guidance strategy is implemented to force the process to roll back to the compliance node and guide it to be re-executed. When the work deviation type is a safety confirmation missing deviation, a forced guidance strategy is executed to lock the operation interface of the current maintenance task until the safety confirmation is completed. When the deviation risk level is high-risk, a mandatory intervention suspension guidance strategy is implemented to immediately stop the current operation, trigger an alarm, and send a warning message to the ground control platform.

[0012] In some embodiments of this application, the execution of the guidance strategy through a mixed reality terminal in a multimodal fusion output manner includes: presenting virtual annotations, operation guidance animations, and risk area highlights through a mixed reality visual overlay method using the mixed reality terminal; providing full-process voice broadcast prompts through voice interaction; and activating sound and light alarms and / or the tactile vibration module of the mixed reality terminal to provide enhanced reminders when the forced intervention termination guidance strategy is triggered.

[0013] In some embodiments of this application, the method further includes: after completing the current maintenance task, generating a maintenance report containing the operation deviation type, the deviation risk level, and the corresponding guidance strategy; and archiving and storing the maintenance report.

[0014] A second aspect of this application provides a dynamic guidance device based on operational deviation risk classification, characterized in that it includes: The acquisition module is used to acquire the structured standard operating procedure model corresponding to the current inspection and maintenance task. The structured standard operating procedure model includes at least a set of process nodes, a set of operation actions, and a set of safety constraints. Based on the structured standard operating procedure model, the module parses out the sequential constraint relationship for defining the execution logic between process nodes and the condition constraint relationship for defining the prerequisite state that the node execution must meet. The status awareness module is used to acquire real-time work status data through multi-source perception fusion during the process of operators performing the current maintenance task according to the structured standard operating procedure model. The deviation identification module is used to identify the type and risk level of job deviation in real time based on the job status data, the set of process nodes, the set of job actions, the sequence constraint relationship, and the condition constraint relationship. The strategy matching module is used to dynamically match the corresponding guidance strategy based on the current process node, the operation deviation type, and the deviation risk level, combined with a pre-set guidance strategy matching function. The strategy execution module is used to execute the guidance strategy through a mixed reality terminal in a multimodal fusion output mode to guide the operator to correct deviations.

[0015] A third aspect of this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method described in the first aspect above.

[0016] This application provides a dynamic guidance method based on risk classification of operational deviations. It constructs a structured standard operating procedure model and extracts sequential and conditional constraints from it, using this as the operational benchmark. Combined with multi-source sensing fusion techniques, it acquires operational status data in real time, enabling automatic identification and type determination of operational deviations. This completely changes the passive guidance mode of existing technologies, which rely on preset fixed procedures and cannot perceive on-site operational conditions, ensuring close linkage between guidance content and actual operational status. By classifying identified deviations into risk levels, including identifying high-risk deviations, it overcomes the shortcomings of existing methods that provide indiscriminate and uniform prompts. It can accurately capture and promptly locate high-risk operations that seriously threaten personal and equipment safety. Based on the current process node, deviation type, and risk level, it dynamically generates and executes differentiated guidance strategies, outputting them via a mixed reality terminal in a multimodal fusion manner. This not only achieves a reasonable match between guidance intensity and risk level, avoiding excessive interference from low-risk prompts on normal operations, but also ensures efficient transmission and powerful intervention of correction information under harsh working conditions such as high noise, low light, and hands-free operation, preventing accident risks at the source and significantly improving the standardization and safety of maintenance and repair operations.

[0017] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0018] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a dynamic guidance method based on job deviation risk classification provided in this application embodiment; Figure 2 A schematic diagram of a structured standard operating procedure model provided in an embodiment of this application; Figure 3 A schematic diagram of a technical architecture based on job deviation identification provided in this application embodiment; Figure 4 A schematic diagram illustrating the execution of a guidance strategy by a mixed reality terminal, as provided in an embodiment of this application; Figure 5 A schematic diagram of a dynamic guidance system based on job deviation risk classification provided in an embodiment of this application; Figure 6This is a schematic diagram of a dynamic guidance device based on the risk classification of operational deviations, provided as an embodiment of this application. Detailed Implementation

[0019] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0020] Specifically, the following describes a dynamic guidance method and apparatus based on job deviation risk classification according to embodiments of this application with reference to the accompanying drawings.

[0021] Figure 1 This is a flowchart illustrating a dynamic guidance method based on job deviation risk classification, provided as an embodiment of this application. Figure 1 As shown, this dynamic guidance method based on operational deviation risk classification may include the following steps: Step 101: Obtain the structured standard operating procedure model corresponding to the current inspection and maintenance task. The structured standard operating procedure model includes at least a set of process nodes, a set of work actions, and a set of safety constraints. Based on the structured standard operating procedure model, parse out the sequential constraint relationship used to define the execution logic between process nodes and the condition constraint relationship used to define the prerequisite state that the node execution must meet.

[0022] Optionally, in some embodiments of this application, the structured standard operating procedure model may further include a set of technical standards, a set of related information, and a set of emergency response measures, constructing a unified semantic expression model to obtain the following structured standard operating procedure model: P={N,A,S,C,H,M} Wherein, P is the structured standard operating procedure model for the current maintenance task; N is the set of process nodes, which is the smallest execution unit of the maintenance task, including attributes such as node number, node name, preconditions, and postconditions; A is the set of work actions, corresponding to the specific operation content and action requirements of a single process node, such as closing the circuit breaker, de-energizing, and sequential scanning; S is the set of technical standards, which defines the quality requirements, parameter thresholds, and acceptance criteria that the node operation must meet; C is the set of safety constraints, which includes the hazard sources, risk levels, red-line operations, and safety confirmation requirements corresponding to the node; H is the set of associated information, which covers the equipment 3D model, operation animation, historical fault cases, and spare parts information of the corresponding node; and M is the set of emergency response, which clarifies the emergency response procedures and measures when abnormalities occur in the node operation.

[0023] Based on this, the three types of implicit constraints in the SOP are analyzed using a structured standard operating procedure (SOP) model, and a semantic association network of the process is constructed, such as... Figure 2 As shown, the first constraint relationship describes the sequential execution logic and prerequisite dependencies between process nodes, serving as the core benchmark for identifying out-of-order deviations. The second constraint relationship defines the prerequisite conditions that a node's execution must meet, such as equipment status, environmental status, and safety confirmations, providing the core basis for identifying violations. Finally, the risk constraint relationship establishes a semantic association between hazard sources, red-line operations, safety measures, and emergency response, supporting risk classification and guidance strategy matching.

[0024] By using the above modeling method, traditional paper-based SOPs are transformed into a structured semantic network containing nodes, actions, standards, constraints, and related information, enabling digital and structured management of SOPs. This provides an accurate benchmark model for subsequent dynamic process guidance and operational deviation identification, solving the core pain point of the disconnect between SOPs and on-site operations.

[0025] Step 102: During the process of the operator performing the current maintenance task according to the structured standard operating procedure model, the operation status data is obtained in real time through multi-source sensing fusion.

[0026] In some embodiments of this application, work status data can be collected through at least one of the following: a visual acquisition module, a motion capture module, a terminal operation log, and device sensors of a mixed reality terminal. The work status data includes: operator behavior data, equipment status data, and on-site environmental data.

[0027] Step 103: Based on the job status data and the set of process nodes, the set of job actions, the sequential constraints, and the conditional constraints, identify the job deviation type and deviation risk level in real time.

[0028] In some embodiments of this application, the job deviation type may include at least one of the following: Process node omission deviation indicates that a process node (such as an inspection, operation, or safety confirmation node) in the process node set has not been executed, or the execution result has not met the standard, or the execution result has not met the technical standard. Job sequence mismatch deviation is used to indicate that process nodes are not executed in the correct order, thus disrupting the prerequisite dependencies of the processes. Non-standard operation deviation refers to a mismatch between the operator's operation characteristics and the action requirements in the set of work actions. Non-standard operation deviation can be identified in the following ways: acquiring the operator's operation video stream through a visual acquisition module; extracting the operator's operation characteristics using a lightweight target detection algorithm (YOLO target detection algorithm) and human key point capture technology; and comparing the operation characteristics with the action requirements in the set of work actions to identify non-standard actions.

[0029] The safety confirmation deficiency deviation indicates that the preliminary safety confirmation step in the conditional constraint relationship was not completed, or that subsequent operations were carried out without the implementation of safety measures.

[0030] In some embodiments of this application, the deviation risk level may include general deviation, significant deviation, and high-risk deviation; High-risk deviations include at least one of the following: electrical violations, missing interlocking devices, or inadequate confirmation of critical safety requirements.

[0031] Step 104: Based on the current process node, job deviation type, and deviation risk level, dynamically match the corresponding guidance strategy using a pre-set guidance strategy matching function.

[0032] Optionally, a three-dimensional guidance strategy matching mechanism can be constructed, consisting of process node, risk level, and deviation type, as shown in the following expression:

[0033] in, The output guidance strategy. This is the current process node. For the identified operational deviations, The deviation risk level is... It is a deviation type. This is a function for matching the guiding strategy.

[0034] Figure 3 This is a schematic diagram of a technical architecture based on job deviation identification provided in an embodiment of this application. Figure 3 As shown, the boot strategy may include: When the deviation risk level is a general deviation, a prompting guidance strategy is implemented. The operator is informed of the operational problem and correction requirements through voice or AR interface text prompts on the mixed reality terminal, without affecting the progress of the operation process. When the work deviation type is a process node omission deviation, and the deviation risk level is a general deviation or a major deviation, a rework guidance strategy is implemented to guide the operator to return to the missed node to rework the operation. Only after the confirmation is completed can the subsequent process be carried out to ensure the integrity of the work. When the type of work deviation is work sequence mismatch and the risk level of deviation is general deviation or major deviation, a rollback guidance strategy is implemented to force the process to roll back to the compliance node and guide it to be re-executed. When the work deviation type is a safety confirmation missing deviation, a mandatory guidance strategy is implemented to lock the operation interface of the current maintenance task until the safety confirmation is completed. Before the safety confirmation is completed and the measures are implemented, subsequent work is prohibited to strengthen safety control. When the deviation risk level is high-risk, a mandatory intervention suspension guidance strategy is implemented to immediately stop the current operation, trigger an alarm, and send a warning message to the ground control platform, triggering on-site confirmation by the team leader and remote expert review to prevent major safety accidents.

[0035] Step 105: Through the mixed reality terminal, the guidance strategy is executed in a multimodal fusion output mode to guide the operators to correct their deviations.

[0036] Optionally, the mixed reality terminal can be intrinsically safe MR smart glasses for mining applications.

[0037] Figure 4 This is a schematic diagram illustrating a mixed reality terminal execution guidance strategy provided in an embodiment of this application. Figure 4 As shown, in some embodiments, based on the entire lifecycle of maintenance operations, three types of guidance modes are designed to achieve full-process coverage: The first type is pre-operational preventative guidance, which pushes the core risk points, key process requirements, and emergency response measures for the maintenance to the operators before the start of the task; before the execution of a single process node, it pushes the operating steps, technical standards, and safety tips for that node to avoid common operational errors in advance. The second type is real-time correction guidance during the operation, which matches differentiated guidance and correction strategies to different types and risk levels of deviations identified during the operation to achieve timely correction of deviations. The last type is post-operation review and improvement guidance, which automatically generates deviation statistics, violation operation records, and process time analysis for this maintenance after the operation is completed, and pushes corresponding training content to achieve continuous improvement of the operators' operational capabilities.

[0038] Through the mixed reality terminal, virtual annotations, operation guidance animations, and risk area highlights are presented in a visually overlay manner, making them intuitive and easy to understand. Voice interaction is used to provide full-process voice prompts, freeing up the hands. When a forced intervention-type guidance strategy is triggered, sound and light alarms and / or the tactile vibration module of the mixed reality terminal are activated to provide enhanced reminders, ensuring that operators can detect the situation in a timely manner and improving the timeliness of guidance.

[0039] In some embodiments of this application, after the current maintenance task is completed, a maintenance report containing the type of work deviation, the level of deviation risk, and the corresponding guidance strategy can be generated and archived and stored.

[0040] By implementing the embodiments of this disclosure, a structured standard operating procedure model is constructed, and sequential and conditional constraints are extracted from it. This model serves as the operational benchmark. Combined with multi-source sensing fusion methods, real-time operational status data is acquired, enabling automatic identification and type determination of operational deviations. This completely changes the passive guidance mode of existing technologies, which preset fixed procedures and cannot perceive on-site operational conditions, ensuring close linkage between guidance content and actual operational status. By classifying the identified deviations into risk levels, including identifying high-risk deviations, the drawbacks of existing methods that provide indiscriminate and uniform prompts are overcome. This allows for accurate capture and timely location of high-risk operations that seriously threaten personal and equipment safety. Based on the current process node, deviation type, and risk level, differentiated guidance strategies are dynamically generated and executed. Outputted via a mixed reality terminal in a multimodal fusion manner, this not only achieves a reasonable match between guidance intensity and risk level, avoiding excessive interference from low-risk prompts on normal operations, but also ensures efficient transmission and powerful intervention of correction information under harsh working conditions such as high noise, low light, and hands-free operation. This prevents accident risks from the source and significantly improves the standardization and safety of maintenance and repair operations.

[0041] Figure 5 This is a schematic diagram of a dynamic guidance system based on job deviation risk classification, provided as an embodiment of this application. Figure 5 As shown, the dynamic guidance system based on operational deviation risk classification can include a data layer, a logic layer, and an interactive application layer. This dynamic guidance system based on operational deviation risk classification can use a hybrid reality terminal (such as intrinsically safe MR / AR smart glasses for mining) as its carrier.

[0042] The data layer, serving as the underlying data support for the dynamic guidance system, is used for the unified storage, standardized processing, and efficient retrieval of data related to the entire inspection and maintenance process. It provides reliable data assurance for upper-level logical decision-making and interactive applications, and may include six core database types: 1) Structured database of inspection and maintenance SOPs: Stores core data such as process nodes, operating standards, and safety constraints for various mining equipment inspection and maintenance operations, realizes digital management and precise retrieval of SOPs, and provides clear process benchmarks for intelligent guidance; 2) Equipment lifecycle database: Stores equipment model parameters, historical maintenance records, failure cases, spare parts information, etc., providing comprehensive data support for maintenance decisions and fault diagnosis; 3) Fault Knowledge Graph Database: Stores semantic association data of device fault phenomena, fault causes, handling measures, and related processes, enabling rapid fault location and targeted handling guidance; 4) On-site environmental perception database: Stores downhole worker behavior data, real-time equipment operating parameters, and environmental monitoring data (including dust concentration, humidity, light intensity, and noise intensity), providing real-time input data for work status perception, deviation identification, and multimodal interaction adaptation; 5) Interaction rule base: Stores multimodal interaction identification rules, guidance strategy matching rules, deviation identification rules, and interaction adaptation logic rules to standardize the behavior and decision-making logic of the entire interaction system; 6) Basic Resource Library: Basic resources such as 3D models of storage devices, MR (Mixed Reality) interactive materials, voice command sets, gesture recognition models, and materials required for XR virtual-real fusion registration, to ensure the smooth implementation of various interactive applications.

[0043] The logic layer serves as the core decision-making hub of the entire interactive system. Developed using Python, its core computing module enables efficient data exchange with the data layer and interactive application layer through standardized interfaces. The core comprises six major functional modules, balancing technological advancement with engineering practicality: 1) SOP Structured Parsing Module: Completes standardized preprocessing, core element extraction, and constraint relationship parsing of maintenance and repair SOP texts, transforming static process texts into calculable and callable dynamic process models, providing accurate process benchmarks for intelligent guidance; 2) Operation Status Perception Module: Integrates visual acquisition, motion capture, terminal operation data, and equipment sensor data to analyze the operation status, operation characteristics, and changes in the on-site environment of the operators in real time, providing real-time and accurate input data for deviation identification, guidance decision-making, and multimodal interaction adaptation; 3) Deviation identification and risk classification module: Based on SOP benchmarks and real-time operation status, it automatically identifies, locates and classifies typical operation deviations. Combining the characteristics of the hazard source and the severity of the consequences, it intelligently determines the deviation risk level, providing a reliable basis for matching differentiated guidance strategies. 4) Dynamic guidance strategy matching module: Based on the work process node, deviation type, risk level, and personnel experience level, it matches the corresponding guidance modality, guidance content and intervention intensity, and outputs personalized and scenario-based intelligent guidance strategies to achieve accurate correction of work deviations; 5) Multimodal interaction fusion module: It completes the fusion and parsing of multi-channel interaction commands such as voice, gesture, eye movement, and head posture, as well as intent recognition and conflict resolution. At the same time, it realizes the collaborative output of multimodal guidance information such as AR vision, posture, voice, sound and light, and touch. The core integrates multimodal natural interaction adaptation logic to realize dynamic switching of interaction methods. 6) Remote collaborative interaction module: Enables real-time transmission of downhole field images, overlay presentation of remote expert MR virtual annotations, and two-way voice and video conferencing, supporting multi-role collaborative maintenance across space and efficiently solving on-site technical problems.

[0044] The interactive application layer is the direct carrier of human-computer interaction. It is designed for three core users in coal mine maintenance and repair operations (underground workers, surface control personnel, and remote experts), and corresponding interactive terminals and functional modules are designed to achieve accurate output of guidance information and efficient collection of interactive commands. 1) Underground operation terminal: The core carrier is the intrinsically safe MR smart glasses for mining, supplemented by the explosion-proof handheld terminal for mining. The core functions include visual guidance of MR process, real-time alarm of operation deviation, non-contact natural interaction, on-site data collection, and remote collaborative conferencing. It is adapted to underground mobile operation scenarios and is the core interactive terminal of the entire system. 2) Ground control terminal: A web-based control platform is developed based on a B / S architecture. The core functions include dispatching maintenance tasks, real-time monitoring of the operation process, intelligent risk warning, statistical analysis of maintenance data, and historical operation backtracking, providing ground management personnel with a full-process control and interactive interface. 3) Expert Support Terminal: Based on PC and mobile terminals, a remote collaboration module is developed. The core functions include real-time viewing of on-site images, AR virtual annotation, remote technical guidance, and push of maintenance solutions, providing professional remote expert interactive support for on-site operations.

[0045] The dynamic guidance system based on operational deviation risk classification provided in this embodiment relies on a mixed reality terminal as the hardware carrier and a structured inspection and maintenance standard operating procedure (SOP) as the data foundation to realize intelligent identification of operational deviations, risk classification, and dynamic guidance closed-loop control. The overall architecture is lightweight, highly compatible, and has strong anti-interference capabilities, making it suitable for complex working conditions in coal mines.

[0046] Figure 6 This is a schematic diagram of a dynamic guidance device based on operational deviation risk classification, provided as an embodiment of this application. Figure 6 As shown, the dynamic guidance device based on the risk classification of operational deviations may include: an acquisition module 601, a status perception module 602, a deviation identification module 603, a strategy matching module 604, and a strategy execution module 605.

[0047] The acquisition module 601 is used to acquire the structured standard operating procedure model corresponding to the current inspection and maintenance task. The structured standard operating procedure model includes at least a set of process nodes, a set of operation actions, and a set of safety constraints. Based on the structured standard operating procedure model, the module 601 parses out the sequential constraint relationship used to define the execution logic between process nodes and the condition constraint relationship used to define the prerequisite state that the node execution must meet. The status awareness module 602 is used to acquire real-time work status data through multi-source sensing fusion during the process of operators performing current maintenance tasks according to the structured standard operating procedure model. The deviation identification module 603 is used to identify the type and risk level of work deviation in real time based on work status data and process node set, work action set, sequential constraint relationship and condition constraint relationship. The strategy matching module 604 is used to dynamically match the corresponding guidance strategy based on the current process node, job deviation type and deviation risk level, combined with the pre-set guidance strategy matching function. The strategy execution module 605 is used to execute guidance strategies through a mixed reality terminal in a multimodal fusion output mode to guide operators to correct deviations.

[0048] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0049] To implement the above embodiments, this application also proposes an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0050] To implement the above embodiments, this application also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0051] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0052] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0053] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0054] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0055] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0056] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0057] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0058] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A dynamic guidance method based on operational deviation risk classification, characterized in that, Includes the following steps: Obtain a structured standard operating procedure model corresponding to the current inspection and maintenance task. The structured standard operating procedure model includes at least a set of process nodes, a set of work actions, and a set of safety constraints. Based on the structured standard operating procedure model, parse out the sequential constraint relationship used to define the execution logic between process nodes and the condition constraint relationship used to define the prerequisite state that the node execution must meet. During the process of the operator performing the current maintenance task according to the structured standard operating procedure model, the operation status data is acquired in real time through multi-source sensing fusion. Based on the job status data, the set of process nodes, the set of job actions, the sequential constraints, and the conditional constraints, the job deviation type and deviation risk level are identified in real time. Based on the current process node, the type of job deviation, and the level of deviation risk, the corresponding guidance strategy is dynamically matched using a pre-set guidance strategy matching function. The guidance strategy is executed through a mixed reality terminal in a multimodal fusion output mode to correct and guide the workers.

2. The method according to claim 1, characterized in that, The method of acquiring real-time operation status data through multi-source sensing fusion includes: The operation status data is collected through at least one of the visual acquisition module, motion capture module, terminal operation log, and device sensors of the mixed reality terminal; The operation status data includes: the operation behavior data of the operators, equipment status data, and on-site environmental data.

3. The method according to claim 2, characterized in that, The types of operational deviations include at least one of the following: Process node omission deviation is used to indicate that a process node in the set of process nodes was not executed or the execution result did not meet the standard. Job sequence mismatch deviation is used to indicate that process nodes are not executed according to the specified sequence constraints; Non-standard operation deviation is used to indicate that the characteristics of the operator's operation are not matched with the action requirements in the set of operation actions; The safety confirmation missing deviation indicates that the preliminary safety confirmation step in the conditional constraint relationship has not been completed.

4. The method according to claim 3, characterized in that, The non-standard deviations in the operational actions are identified in the following ways: The operation video stream of the operator is acquired through the visual acquisition module; The operation motion features of the workers are extracted using lightweight target detection algorithms and human key point capture technology; The operation action features are compared with the action requirements in the set of work actions to identify non-standard actions.

5. The method according to claim 3, characterized in that, The deviation risk levels include general deviations, significant deviations, and high-risk deviations; The high-risk deviations include at least one of the following: electrical violations, missing interlocking devices, and inadequate confirmation of critical safety requirements.

6. The method according to claim 5, characterized in that, The guidance strategy includes: When the deviation risk level is a general deviation, a prompting guidance strategy is implemented, and the operator is informed through voice or text prompts on the mixed reality terminal. When the type of work deviation is a process node omission deviation, and the deviation risk level is a general deviation or a major deviation, a remedial guidance strategy is implemented to guide the operator to return to the missed node to perform the remedial operation. When the type of work deviation is work sequence mismatch and the risk level of deviation is general deviation or major deviation, a rollback guidance strategy is implemented to force the process to roll back to the compliance node and guide it to be re-executed. When the work deviation type is a safety confirmation missing deviation, a forced guidance strategy is executed to lock the operation interface of the current maintenance task until the safety confirmation is completed. When the deviation risk level is high-risk, a mandatory intervention suspension guidance strategy is implemented to immediately stop the current operation, trigger an alarm, and send a warning message to the ground control platform.

7. The method according to claim 6, characterized in that, The execution of the guidance strategy via a mixed reality terminal in a multimodal fusion output manner includes: Through mixed reality terminals, virtual annotations, operation guidance animations, and risk area highlights are presented in a mixed reality visual overlay manner; The system provides full voice prompts via voice interaction. When the forced intervention suspension guidance strategy is triggered, an audible and visual alarm and / or the tactile vibration module of the mixed reality terminal is activated to provide enhanced reminders.

8. The method according to claim 1, characterized in that, Also includes: After completing the current maintenance task, a maintenance report is generated that includes the type of work deviation, the level of deviation risk, and the corresponding guidance strategy. The maintenance report shall be archived and stored.

9. A dynamic guidance device based on operational deviation risk classification, characterized in that, include: The acquisition module is used to acquire the structured standard operating procedure model corresponding to the current inspection and maintenance task. The structured standard operating procedure model includes at least a set of process nodes, a set of operation actions, and a set of safety constraints. Based on the structured standard operating procedure model, the module parses out the sequential constraint relationship for defining the execution logic between process nodes and the condition constraint relationship for defining the prerequisite state that the node execution must meet. The status awareness module is used to acquire real-time work status data through multi-source perception fusion during the process of operators performing the current maintenance task according to the structured standard operating procedure model. The deviation identification module is used to identify the type and risk level of job deviation in real time based on the job status data, the set of process nodes, the set of job actions, the sequence constraint relationship, and the condition constraint relationship. The strategy matching module is used to dynamically match the corresponding guidance strategy based on the current process node, the operation deviation type, and the deviation risk level, combined with a pre-set guidance strategy matching function. The strategy execution module is used to execute the guidance strategy through a mixed reality terminal in a multimodal fusion output mode to guide the operator to correct deviations.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-8.

Images