Training aid and method

Abstract

This invention relates to the field of 3D training and discloses a training aid and method, including a display positioning module for achieving hybrid virtual-real interaction and spatial positioning, collecting operational and environmental data; a model fusion module for data interaction with the display positioning module, for constructing a device model and achieving virtual-real overlay; and an intelligent guidance module connected to both the display positioning module and the model fusion module for providing training guidance, fault injection, and assessment. The display positioning module transmits the collected data to the model fusion module and the intelligent guidance module. The model fusion module presents the virtual-real fusion scene through the display positioning module. The intelligent guidance module generates guidance or warning instructions based on the data processing results and outputs them through the display positioning module, accurately recording operational problems, making training assessments more objective and standardized, and providing data support for subsequent personalized training, significantly improving the efficiency and professionalism of industrial equipment maintenance training.

Description

Technical Field

[0001] This invention relates to the field of 3D training, and in particular to a training aid and training method. Background Technology

[0002] In response to the core pain points in practical training of main equipment (pressure vessels, steam generators, main pumps, etc.) in nuclear power plants, such as highly complex equipment structures, high operational risks, long training cycles, high costs, and low standardization, traditional physical simulation training models are limited by inherent defects such as bulky and immobile physical equipment, easily damaged precision parts, and low operational error tolerance. This leads to problems such as personnel safety hazards, rising equipment maintenance costs, and difficulty in implementing standardized teaching during the training process.

[0003] Traditional industrial equipment maintenance training relies on physical simulation devices, which suffers from problems such as high operational risks due to complex equipment structures, escalating training costs due to the easy wear and tear of precision components, and difficulties in implementing standardized teaching. Meanwhile, existing digital VR training methods have shortcomings such as insufficient 3D visualization, limited human-computer interaction, a disconnect between virtual and real scenes, poor immersion, and a lack of dynamic guidance. This results in a significant deviation between training effectiveness and real-world conditions. Current training methods cannot accurately integrate virtual models with the real environment and physical tools, making it difficult for operators to obtain a near-realistic operating experience. Furthermore, the lack of real-time intelligent guidance and dynamic fault injection capabilities during operation hinders the effective training of operators' ability to handle complex and sudden faults. Traditional training is mostly conducted in a single-person, single-machine mode, making it difficult to simulate the team collaboration scenarios of real nuclear power plant maintenance. It lacks the technical support for multi-user collaborative operations, and instructors cannot remotely intervene for guidance and assessment. The authenticity and practicality of the training scenario are limited, and assessment relies heavily on experience-based judgment, lacking a standardized, data-driven quantitative evaluation system. It is impossible to accurately record operators' operational trajectories, errors, and skill weaknesses, making it difficult to trace and quantify training effectiveness. Subsequent personalized advanced training lacks data support. Summary of the Invention

[0004] Therefore, the technical problem to be solved by this invention is that in existing industrial equipment maintenance training, physical simulation methods have problems such as high operational risks, easy equipment damage, high training costs and low standardization. Digital training methods have defects such as the separation of virtual and real scenarios, single interaction methods and lack of dynamic intelligent guidance, making it difficult to reproduce the real operation experience. At the same time, existing training cannot realize multi-user collaborative operation and remote instructor guidance. Assessment and evaluation rely on experience-based judgment, lack a standardized quantitative system, and cannot accurately record operational problems, resulting in a large deviation between training effect and actual working conditions, making it difficult to meet the professional training needs of nuclear power plant maintenance.

[0005] The above-mentioned technical problems are solved by the following technical solution: This invention proposes a training auxiliary device, which includes a display positioning module for realizing hybrid virtual-real interaction and spatial positioning, collecting operation and environmental data; a model fusion module for data interaction with the display positioning module, for constructing a device model and realizing virtual-real overlay; and an intelligent guidance module connected to the display positioning module and the model fusion module respectively, for providing training guidance. The display positioning module transmits the collected data to the model fusion module and the intelligent guidance module. The model fusion module presents the virtual-real fusion scene through the display positioning module. The intelligent guidance module generates guidance or warning instructions based on the data processing results and outputs them through the display positioning module.

[0006] In a preferred embodiment of the training auxiliary device of the present invention, it further includes a collaborative training module, which synchronizes data with the display positioning module, the model fusion module and the intelligent guidance module, and is used to receive interactive data from each module and realize status synchronization between multiple devices, supporting multi-user collaborative maintenance training.

[0007] In a preferred embodiment of the training aid device of the present invention: the display positioning module includes a head-mounted display device and a spatial positioning unit. The head-mounted display device is used to receive and present virtual-real fusion scene data, guidance or warning instructions, and integrates a multimodal interaction unit to realize interaction with the virtual model. The spatial positioning unit uses an extended multi-source sensor data fusion algorithm to fuse multi-source sensor data, realize spatial positioning, and output positioning data to the model fusion module.

[0008] In a preferred embodiment of the training aid device of the present invention: the multimodal interaction unit includes a gesture recognition module, an eye tracking module, and a voice control module. The gesture recognition module recognizes the operator's maintenance operation gestures and converts them into operation data. The eye tracking module triggers information prompts based on the operator's gaze point. The voice control module receives voice commands and retrieves relevant maintenance information.

[0009] In a preferred embodiment of the training aid device of the present invention: the model fusion module includes a model building unit and a virtual-real fusion unit. The model building unit builds a three-dimensional model of the device and optimizes its performance. The virtual-real fusion unit receives positioning data from the spatial positioning unit to achieve coordinate matching and precise alignment of the virtual model and the physical device.

[0010] In a preferred embodiment of the training auxiliary equipment of the present invention: the intelligent guidance module can also perform fault injection and assessment, which includes a knowledge graph unit and an assessment unit. The knowledge graph unit integrates industrial equipment maintenance-related data to construct a structured knowledge graph, providing a basis for the detection and judgment of operational behavior. The assessment unit uses a multi-objective scoring model based on the operational data to quantitatively assess the maintenance training operation and generate assessment data.

[0011] In a preferred embodiment of the training aid device of the present invention: the collaborative training module includes a state synchronization unit and a space sharing unit. The state synchronization unit realizes low-latency synchronization of virtual scene states among multiple devices based on a network framework. The space sharing unit realizes spatial coordinate system alignment of all participants through a distributed spatial anchor point system.

[0012] To address the aforementioned technical problems, the present invention also provides the following technical solution: a training method, comprising training auxiliary equipment; and, constructing a three-dimensional model of industrial equipment and optimizing its performance, deploying MR hardware equipment, collecting training site environmental data and completing the virtual-real fusion calibration of the virtual model with the physical equipment and the real environment, integrating relevant data on industrial equipment maintenance, constructing a structured knowledge graph for maintenance training and setting quantitative assessment rules, presenting a virtual-real fusion maintenance training scenario through MR equipment, collecting operator operation data and performing real-time detection and judgment of operation behavior based on the knowledge graph, automatically outputting guidance or warning information, quantitatively assessing maintenance training operations based on operation data according to quantitative assessment rules, and generating and outputting an industrial equipment maintenance training assessment report.

[0013] In a preferred embodiment of the training aid device of the present invention: an extended multi-source sensor data fusion algorithm is used to fuse multi-source sensor data to achieve spatial positioning, and the virtual model and the physical device are precisely aligned based on the positioning data. At the same time, the multimodal interaction triggering logic of the MR hardware device is optimized.

[0014] In a preferred embodiment of the training aid device of the present invention: when an operator is detected to have violated operating rules, a fault scenario simulation is automatically triggered based on the fault association rules of the knowledge graph, and corresponding safety warning information is output simultaneously. A multi-objective scoring model is adopted to quantitatively assess the operation from at least two dimensions: operation completion, operation accuracy, and operation standardization.

[0015] The beneficial effects of the training aids and methods of this invention are as follows: Through the coordinated operation of display positioning, model fusion, and intelligent guidance modules, relying on multimodal interaction, spatial positioning algorithms, and virtual-real fusion technology, precise integration of virtual and physical scenarios is achieved, allowing trainees to obtain a realistic operational experience and effectively avoiding the wear and safety risks of operating physical equipment. Simultaneously, the knowledge graph unit provides intelligent dynamic guidance for training, automatically triggering warnings and fault simulations when violations occur, improving trainees' fault-handling capabilities. The collaborative training module enables low-latency synchronous operation for multiple users, recreating team maintenance scenarios, allowing instructors to remotely intervene and provide guidance. The assessment unit uses a multi-objective scoring model to quantitatively evaluate training operations, accurately recording operational problems, making training assessments more objective and standardized, and providing data support for subsequent personalized training, significantly improving the efficiency and professionalism of industrial equipment maintenance training. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. Obviously, the drawings described below only relate to some embodiments of the present invention and are not intended to limit the present invention. Wherein: Figure 1 A schematic diagram of the overall system interaction of the training aid equipment is shown; Figure 2 The diagram shows the network architecture for the MR system deployment in the training aids. Figure 3 The flowchart for the development and production of the MR system in training aids is shown; Figure 4 The diagram shows the operation flow chart of the MR system in the training auxiliary equipment, with evaporator maintenance as the training objective; Figure 5 The flowchart for maintenance training in the training auxiliary equipment is shown; Figure 6 A flowchart of the maintenance and assessment process for training auxiliary equipment is shown; Figure 7 A flowchart of multi-user synchronous collaboration in training aids is shown.

[0017] Figure 8 A flowchart illustrating the binding process of physical tools in training aids is shown. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0019] The terminology used in this invention is that which is currently widely used in the art in consideration of the function of the invention; however, these terms may vary according to the intent of those skilled in the art, precedent, or new technology in the art. Furthermore, specific terms may be chosen by the applicant, and in such cases, their detailed meanings will be described in the detailed description of the invention. Therefore, the terms used in this specification should not be construed as simple names, but rather based on their meanings and the overall description of the invention.

[0020] Reference Figures 1-8 This embodiment provides a training aid device, including a display positioning module 1, used to realize hybrid virtual-real interaction and spatial positioning, and to collect operation and environmental data. Specifically, the display positioning module 1, as the core of human-computer interaction and data collection, can adopt various immersive technologies such as VR, AR, and MR. No specific limitation is made here, but it is recommended to use an industrial-grade MR hardware base as the basic carrier. It mainly realizes hybrid virtual-real interaction and spatial positioning functions, and can collect the trainee's operation data and the environmental data of the training venue in real time. The operation data includes the trainee's hand movements, voice commands, and gaze focus position, etc. The environmental data includes the spatial coordinates of the training venue, obstacle distribution, lighting conditions, etc. No specific limitation is made here. The display positioning module 1 transmits all the collected data to the model fusion module 2 and the intelligent guidance module 3 in real time through the local area network, providing a data foundation for subsequent virtual-real fusion and intelligent guidance.

[0021] Model fusion module 2 interacts with display positioning module 1 to construct equipment models and achieve virtual-real overlay. Model fusion module 2 establishes a stable network data interaction connection with display positioning module 1. Its core function is to construct a three-dimensional equipment model of industrial equipment and achieve virtual-real overlay of virtual models with physical equipment and real environment. Model fusion module 2 receives environmental data transmitted by display positioning module 1, combines it with the constructed equipment model to complete the rendering of virtual-real fusion scene, and then feeds back the rendered virtual-real fusion scene data to display positioning module 1 in real time, so that display positioning module 1 can complete the scene presentation.

[0022] The intelligent guidance module 3 is connected to the display and positioning module 1 and the model fusion module 2 respectively. It is used to provide training guidance, fault injection, and assessment. The display and positioning module 1 transmits the collected data to the model fusion module 2 and the intelligent guidance module 3. The model fusion module 2 presents the virtual-real fusion scene through the display and positioning module 1. The intelligent guidance module 3 generates guidance or warning instructions based on the data processing results and outputs them through the display and positioning module 1. The intelligent guidance module 3 establishes a wireless network data connection with the display and positioning module 1 and the model fusion module 2 respectively. Its core functions are to provide trainees with maintenance training guidance, dynamic fault injection, and training operation assessment. The intelligent guidance module 3 processes and analyzes the received data in real time. It judges whether the trainee's operation behavior conforms to the maintenance specifications through the built-in maintenance rule algorithm. Based on the judgment result, it generates corresponding training guidance instructions or operation warning instructions and transmits the instructions to the display and positioning module 1. The display and positioning module 1 outputs the instructions to the trainee, realizing intelligent dynamic guidance in the training process.

[0023] Furthermore, it also includes a collaborative training module 4. This module 4 achieves data synchronization with the display and positioning module 1, model fusion module 2, and intelligent guidance module 3. It receives interactive data from each module and synchronizes the status of multiple devices, supporting multi-user collaborative maintenance training. The addition of collaborative training module 4, suitable for multi-user team maintenance training scenarios, is built using a cloud server and the Netcode network framework. Collaborative training module 4, display and positioning module 1, model fusion module 2, and intelligent guidance module 3 can all achieve high-speed data synchronization via a 5G local area network. It can receive interactive data from each module in real time, including multi-user operation data collected by display and positioning module 1, virtual-real fusion scene data from model fusion module 2, and guidance and assessment data from intelligent guidance module 3. The core function of collaborative training module 4 is to achieve status synchronization between multiple training devices. Interactive data from each module is synchronized to all devices participating in the training, allowing multiple users to be in the same virtual-real integrated training scenario. This supports collaborative training for industrial equipment maintenance by multiple users, recreating a realistic team maintenance operation scenario. Since most existing technologies develop training equipment primarily for individual training, this leads to limitations. Furthermore, individual wearable products are mostly used for personal training and do not provide a detailed solution for multi-person training scenarios. This solution addresses the issue of equipment scarcity in the virtual-real integration process by allowing managers to assign different tasks to each trainee through the intelligent guidance module 3. It also addresses the problem of insufficient actual equipment in nuclear power plants, where equipment is expensive and there may not be enough of a particular product for multiple people to work together. Therefore, different tasks can be performed separately, and training content can be exchanged with others while completing the task to ensure comprehensive training.

[0024] Furthermore, the display positioning module 1 includes a head-mounted display device 11 and a spatial positioning unit 12. The head-mounted display device 11 is used to receive and present virtual-real fusion scene data, guidance or warning instructions, and integrates a multimodal interaction unit to realize interaction with the virtual model. The head-mounted display device 11 can be any model that meets the above requirements and can cooperate with each module. Specifically, it should support full-color perspective display and support external multimodal interaction modules, such as gesture, eye tracking, voice and other hardware interfaces or SDKs. The model can be Microsoft HoloLens2, which is not specifically limited here. It is a terminal device worn directly by the trainee and serves as the core of information presentation. It can receive virtual-real fusion scene data transmitted by the model fusion module 2 and guidance or warning instructions transmitted by the intelligent guidance module 3 in real time, and present them to the trainee synchronously through the full-color perspective display interface. At the same time, the head-mounted display device 11 integrates a multimodal interaction unit, which is the interaction carrier between the trainee and the virtual model. It can realize the natural interaction between the trainee and the virtual model of industrial equipment, allowing the trainee to complete the maintenance training actions through intuitive operation.

[0025] The spatial positioning unit 12 can perform real-time positioning and uses an extended multi-source sensor data fusion algorithm to fuse multi-source sensor data. Specifically, it can use various fusion algorithms such as EKF, particle filtering, and complementary filtering. In this solution, the Kalman filtering algorithm is selected to achieve spatial positioning and output positioning data to the model fusion module 2. The spatial positioning unit 12 is the positioning core of the display positioning module 1. It consists of a spatial positioning camera array and an IMU inertial measurement unit built into the head-mounted display device 11. The spatial camera needs to meet the requirements of three-dimensional point cloud output, be compatible with SLAM algorithm and PnP calculation, and be able to fuse with IMU data in real time. OptiTrackPrime13W can be used, but no specific limitation is made here. It uses the extended Kalman filtering (EKF) algorithm as the core positioning algorithm. This algorithm is written in C++ and embedded in the underlying firmware of the head-mounted display device 11. The spatial positioning unit 12 achieves millimeter-level spatial positioning of trainees, physical equipment, and maintenance tools through the fused multi-source sensor data and outputs the generated accurate positioning data to the model fusion module 2 in real time, providing the core positioning basis for the accurate alignment of virtual and real data in the model fusion module 2.

[0026] Specifically, the calculation principle of Kalman filtering (EKF) is as follows: State vector: It includes position, attitude (quaternion), velocity, gyroscope bias, and accelerometer bias.

[0027] in: x: System state vector, representing all current states of the MR spatial positioning system; T : Transpose symbol, converts a row vector into a column vector, used in matrix operations; : Position vector, representing three-dimensional coordinates (X / Y / Z) in space; Attitude quaternion, representing the rotation / orientation of the device in space; : Velocity vector, representing the velocity of spatial motion (X / Y / Z directions); Fixed error of the gyroscope sensor; Fixed error of the accelerometer sensor.

[0028] Prediction (IMU driven): ,in: F Its Jacobian matrix; : Representing the current moment Representing the previous moment, specifically the prior system state estimate for the current moment, which is predicted based on the data from the previous moment, and the state at time t is predicted using time t-1. The optimal state of the system after correction at the previous moment; The current IMU outputs angular velocity and acceleration measurement input data; The prior covariance matrix at the current moment represents the magnitude of the error in the predicted state; : The current state transition Jacobian matrix, the core linearization term of EKF; : The covariance matrix corresponding to the state at the previous time step; Process noise covariance, IMU measurement noise.

[0029] Update (Visual Observation): When visual observation is obtained (e.g., SLAM pose calculation, PnP pose calculation), calculate Kalman gain .

[0030] in: Kalman gain, balancing the weights of prediction and observation, EKF core; : The transpose of the Jacobian matrix of the observation matrix, a linearized mapping of visual observation; : Observation noise covariance, noise in visual / SLAM localization.

[0031] Update state and covariance: h represents the observation model, which maps the state to the observation space, and H is its Jacobian matrix.

[0032] It is important to note that in the Jacobian matrix: Discrete time step (e.g.) (where t is the previous time and t is the current time). : State vector; Observations (such as pose data solved by visual SLAM / PnP); Kalman gain (the core computational cost of the filtering algorithm, used to balance the weights of prediction and observation); State covariance matrix (representing the estimation error of the state vector); : Identity matrix (general mathematical symbol); Posterior state estimation, the optimal state after fusion of observations; : Observations, pose data solved by visual SLAM / PnP; : Posterior covariance matrix, final state uncertainty.

[0033] Furthermore, the key algorithmic steps of the EKF (Electronic Kalman Filter) are as follows: This algorithm is the foundation for accurate and stable superposition of virtual and real data. It integrates visual data from the inertial measurement unit (IMU) built into the MR headset, the spatial camera array, and the recognition information of entity markers, and inputs raw data from multiple sensors (image frames, acceleration, angular velocity), preset spatial anchor point coordinates, and entity marker feature library.

[0034] Furthermore, during processing, visual SLAM initialization involves feature extraction and matching of the scanned point cloud data to construct an initial environment map. Extended Kalman Filter (EKF) fusion uses the pose output by visual SLAM as the observation value and fuses it with the pose predicted by IMU to smooth the trajectory and effectively suppress jitter caused by rapid movement and the cumulative error of pure visual methods. Secondly, marker-assisted relocalization is performed when the system detects visual markers on the physical device model. The known 3D coordinates of the markers and their projections in the image are solved using PnP (Perspective-n-Point) to provide an absolute pose reference, correct the drift in the fusion algorithm, and achieve "dynamic reference plane calibration". The final output is a high-precision, low-latency six-DOF pose of the device itself, as well as a stable transformation matrix of the virtual model relative to the real world.

[0035] Furthermore, the multimodal interaction unit includes a gesture recognition module 111, an eye-tracking module 112, and a voice control module 113. The gesture recognition module 111 recognizes the operator's maintenance operation gestures and converts them into operation data. The gesture recognition module 111 is a depth camera and gesture recognition algorithm module built into the head-mounted display device 11. The built-in skeletal tracking algorithm can accurately recognize various nuclear power plant maintenance operation gestures of the trainee, such as grasping, rotating, disassembling, and aligning, and convert the recognized gesture actions into standardized operation data (transmitted in three-dimensional coordinate data format) and transmit it to the core processing unit of the display positioning module 1 to realize the function of the trainee operating the virtual model through gestures. For example, the gesture recognition trigger threshold can be set to a finger bending angle ≥30° and a wrist rotation angular velocity >30 degrees / second.

[0036] The eye-tracking module 112 triggers prompts based on the operator's gaze point. The eye-tracking module 112 is an eye tracker built into the head-mounted display device 11, which can be linked with the display interface of the head-mounted display device. It supports custom trigger conditions and can use Tobii4C (built-in). No specific limitation is made here. It can capture the trainee's gaze focus position in real time and has a certain sampling frequency. Based on the trainee's gaze point, when the trainee gazes at a certain part of the virtual model for a duration of ≥1s, the eye-tracking module 112 can automatically trigger the relevant information such as the structure, function, and maintenance specifications of that part. This information is presented to the trainee in 3D text form through the display interface of the head-mounted display device 11, helping the trainee focus on the details of complex structures and improving the relevance of the training.

[0037] The voice control module 113 receives voice commands and retrieves relevant maintenance information. The voice control module 113 is a far-field voice recognition module built into the head-mounted display device 11. Trainees can quickly retrieve relevant information such as maintenance operation procedures, equipment drawings, and troubleshooting methods for industrial equipment through voice commands. After receiving the trainee's voice commands, the voice control module 113 recognizes and parses the commands, quickly matches and retrieves the corresponding information, and presents it through the display interface of the head-mounted display device 11, greatly improving the convenience of information retrieval during training.

[0038] Furthermore, the model fusion module 2 includes a model building unit 21 and a virtual-real fusion unit 22. The model building unit 21 builds a 3D model of the equipment and optimizes its performance. The model building unit 21 can be composed of 3D modeling software such as 3ds Max 2024 and ZBrush 2024, as well as the lightweight model plugin MayaLT. No specific limitations are made here. Its core function is to build a 3D model of industrial equipment and optimize its performance. Based on the CAD design drawings of industrial equipment, it can use 3ds Max 2024 to perform polygon topology reconstruction and use ZBrush 2024 to sculpt the surface texture of the equipment to build a high-fidelity 3D model of industrial equipment, covering all key components of equipment such as pressure vessels, steam generators, and main pumps. At the same time, the model building unit 21 can perform PBR material baking to generate metallicity, roughness, and ambient occlusion (AO) maps, and optimize the performance of the built 3D model. While ensuring the realism of the model, it reduces the operating load of the model and ensures that the model runs smoothly during training.

[0039] The virtual-real fusion unit 22 receives positioning data from the spatial positioning unit 12, realizing coordinate matching and precise alignment of the virtual model and the physical device. The virtual-real fusion unit 22 is developed based on the MRTK3.0 mixed reality toolkit and establishes a 100Hz data connection with the spatial positioning unit 12 of the display positioning module 1. It can receive the precise positioning data output by the spatial positioning unit 12 in real time. Its core function is to realize coordinate matching and precise alignment of the six degrees of freedom between the virtual model and the physical device. Based on the received positioning data, the three-dimensional virtual model constructed by the model building unit 21 is precisely matched with the physical device and the real environment in the training venue through a multi-marker fusion positioning algorithm, realizing seamless superposition of the virtual model and the physical device, allowing trainees to obtain a realistic virtual-real fusion experience.

[0040] Furthermore, the intelligent guidance module 3 includes a knowledge graph unit 31 and an assessment unit 33. The knowledge graph unit 31 integrates relevant data on industrial equipment maintenance to construct a structured knowledge graph, providing a basis for the detection and judgment of operational behavior. The assessment unit 33 uses a multi-objective scoring model based on operational data to quantitatively assess maintenance training operations and generate assessment data.

[0041] Specifically, Knowledge Graph Unit 31's core function is to integrate industrial equipment maintenance-related data to construct a structured knowledge graph. This integrated maintenance-related data includes all maintenance-related materials such as industrial equipment maintenance manuals, historical fault databases, maintenance process regulations, and operating procedures. Through a data cleaning algorithm, the raw data is preprocessed, and after entity relation extraction and knowledge fusion, the scattered maintenance data is integrated into a structured knowledge graph with equipment components, maintenance procedures, fault phenomena, and operating procedures as nodes. This provides core data for the intelligent guidance module 3 to detect and judge the trainee's operational behavior. According to the assessment unit 33, which is written in C language and embedded in the simulation engine, its core function is to quantify the trainee's maintenance training operation and generate assessment data. Based on the trainee's operation data transmitted by the display positioning module 1, it adopts a multi-objective scoring model as the core assessment model to comprehensively quantify the trainee's operation process and operation results. The assessment unit 33 calculates the trainee's operation score according to the assessment model, generates standardized assessment data including the completion status of operation steps, operation error points, assessment scores, etc., and can store it to provide a data foundation for the generation of subsequent assessment reports.

[0042] Furthermore, the collaborative training module 4 includes a state synchronization unit 41 and a space sharing unit 42. The state synchronization unit 41 realizes low-latency synchronization of virtual scene states between multiple devices based on a network framework. It is developed based on the Netcode network framework, and its core function is to realize low-latency synchronization of virtual scene states between multiple devices. It can synchronize all state data in the virtual-real integrated training scene, such as the operation state of the virtual model, the operation behavior of the trainee, and the triggering state of the fault scene, to all devices participating in the training, ensuring that the virtual scene states seen by multiple users are highly consistent and avoiding the problem of scene desynchronization.

[0043] The space sharing unit 42 achieves spatial coordinate system alignment for all participants through a distributed spatial anchor point system. The core function of the space sharing unit 42 is to achieve spatial coordinate system alignment for all training participants. This service creates unified spatial anchor points for all participating devices, for example, with an anchor point positioning error of <5mm, ensuring that the spatial coordinate systems of all training participants remain consistent. This ensures that when multiple users conduct collaborative maintenance operations in the same virtual-real fusion scenario, the operation positions are accurately matched, thereby achieving a collaborative training effect where what you see is the same scene.

[0044] Furthermore, constructing and optimizing the 3D model of the industrial equipment is crucial. This can be achieved using professional 3D modeling tools such as 3ds Max 2024 and ZBrush 2024. A high-fidelity 3D model is built based on the CAD design drawings of the industrial equipment, covering all key components and structures, including pressure vessels, steam generators, and main pumps. Substance Painter is used for PBR material baking technology to optimize the model's materials, generating metallicity, roughness, and AO maps. This ensures the model runs smoothly in the simulation engine at a frame rate of at least 60fps, guaranteeing smooth operation during training.

[0045] Deploy MR hardware devices, collect training venue environmental data, and complete the virtual-real fusion calibration of the virtual model with physical devices and the real environment. Deploy MR hardware devices such as head-mounted displays and spatial positioning cameras of training aids to the training venue. Arrange the required number of spatial positioning cameras according to the size of the venue. Use the spatial positioning unit 12 of the display positioning module 1 to perform a comprehensive scan of the training venue, collect environmental data such as spatial coordinates and obstacle distribution, and generate a 3D point cloud map. At the same time, based on the collected environmental data, call spatial anchor points through the MRTK toolkit to complete the virtual-real fusion calibration of the virtual model with the physical devices and the real environment in the training venue, ensuring that the virtual model and physical devices are accurately superimposed.

[0046] By integrating relevant data on industrial equipment maintenance, a structured knowledge graph for maintenance training is constructed, and quantitative assessment rules are set. This involves integrating relevant data such as industrial equipment maintenance manuals, historical fault databases, and maintenance process specifications through data cleaning algorithms. Based on a graph database, a structured knowledge graph for maintenance training is constructed. At the same time, standardized quantitative assessment rules are set according to training requirements, clarifying assessment dimensions, scoring standards for each dimension, and weight allocation. Assessment data is stored in real time in Excel format.

[0047] The MR device presents a virtual-real fusion maintenance training scenario, collects operator operation data, and performs real-time detection and judgment of operation behavior based on a knowledge graph, automatically outputting guidance or warning information. The rendered virtual-real fusion maintenance training scenario is presented to the trainee through the head-mounted display device of the display positioning module 1. The trainee performs maintenance operations in this scenario. The display positioning module 1 collects the trainee's operation data in real time at a frequency of 100Hz. The intelligent guidance module 3 uses a rule algorithm based on the constructed knowledge graph to perform real-time detection and judgment of operation behavior. If the operation conforms to the specifications, the head-mounted display device 11 outputs the corresponding training guidance information; if the operation violates the regulations, the corresponding operation warning information is output.

[0048] Based on operational data, the maintenance training operations are quantitatively assessed according to quantitative assessment rules, generating and outputting an industrial equipment maintenance training assessment report. The assessment and evaluation unit 33 of the intelligent guidance module 3, based on the collected trainee operational data, conducts a comprehensive quantitative assessment of the trainee's maintenance training operations according to the set quantitative assessment rules. Based on the assessment results, a standardized industrial equipment maintenance training assessment report is generated. The report includes the trainee's completion status of operational steps, analysis of operational errors, assessment scores for each dimension, total score, and targeted improvement suggestions. The assessment report can be output in PDF format to the instructor's terminal and the trainee's terminal head-mounted display device 11. No specific limitations are placed on the output method here, as long as the output effect is achieved.

[0049] Furthermore, an extended Kalman filter (EKF) algorithm is employed to fuse multi-source sensor data for spatial positioning. Based on the positioning data, precise alignment of the degrees of freedom between the virtual model and the physical device is achieved. Simultaneously, the multimodal interaction triggering logic of the MR hardware device is optimized. After deploying the MR hardware device, the EKF algorithm is used to fuse multi-source sensor data to achieve high-precision spatial positioning. This multi-source sensor data includes inertial measurement data from the IMU built into the head-mounted display device, visual image data from the spatial positioning camera of the spatial positioning unit 12, and recognition data from the physical device markers. The algorithm is written in C++ and embedded in the underlying firmware of the head-mounted display device 11. Based on the precise positioning data generated after fusion processing (positioning error <3m), the system achieves high accuracy. (m) To achieve precise alignment of the virtual model and the physical device across six degrees of freedom, ensuring accurate matching in all dimensions such as translation along the X / Y / Z axes and rotation around the X / Y / Z axes, the MRTK toolkit was used to optimize the multimodal interaction triggering logic of the MR hardware device based on the operational characteristics of industrial equipment maintenance. The triggering thresholds for gesture recognition were adjusted to a finger bending angle ≥30°, eye tracking to a gaze duration ≥1s, and voice control to a wake-up word volume ≥60dB. This reduced the error rate and improved the fluency and accuracy of human-computer interaction. The specific content of the interaction logic is not limited here; the gesture recognition and other specific values ​​are provided for reference only.

[0050] Furthermore, when an operator's violation is detected, the fault association rules based on the knowledge graph automatically trigger a fault scenario simulation and simultaneously output the corresponding safety warning information. The intelligent guidance module 3 presets fault association rules based on the constructed knowledge graph. These rules are the association logic between the violation, the corresponding fault phenomenon, and the fault handling method. The rules are written and embedded in the graph database. When the operation data collected by the display positioning module 1 is detected and judged to be a violation, the intelligent guidance module 3 automatically triggers the corresponding fault scenario simulation based on the fault association rules of the knowledge graph and through the Unity 2022.3 physics engine. In the virtual-real fusion training scenario, the equipment fault phenomenon that the violation may cause, such as high-pressure injection or component jamming, is reproduced. At the same time, the corresponding safety warning information 3D text + voice prompts and fault handling prompts are output to the trainee through the head-mounted display device 11 Hololens2 to guide the trainee to carry out fault handling operations.

[0051] A multi-objective scoring model is adopted, which quantifies the assessment from at least two dimensions: operation completion, operation accuracy, and operation standardization. The assessment unit 33 adopts a multi-objective scoring model, which selects at least two dimensions from operation completion, operation accuracy, and operation standardization for quantitative assessment. The weight of each dimension can be customized by the instructor via computer, and the total weight is 1. Operation completion is the ratio of the number of necessary maintenance steps completed by the trainee to the total number of necessary maintenance steps. Operation accuracy is the deviation rate (normalized calculation) between the actual value and the standard value of the trainee's key operation. Operation standardization is the ratio of the number of times the trainee performs standardized operations to the total number of operations. Based on the set weights of each dimension, the trainee's comprehensive assessment score is calculated. Combined with data such as operation error points and time consumption statistics, a detailed maintenance training assessment report is generated and output to the instructor's and trainee's terminals.

[0052] Specifically, refer to Figure 2 This document describes the network architecture for the MR system in training aids. In this architecture, the application server and database server provide hardware support for the intelligent guidance module 3 and the collaborative training module 4. The database server stores structured maintenance data from the knowledge graph unit 31, quantitative assessment data from the assessment unit 33, and operational interaction data from all training processes. A third-party server enables the retrieval of external maintenance resources and data synchronization, serving as supplementary support for training data. A switch acts as the core network connection component, enabling high-speed wired data interaction between the application server, database server, third-party server, and administrator and instructor terminals, ensuring stable data transmission. A wireless router serves as the head unit for the display and positioning module 1. The head-mounted display device 11 provides a 5G wireless LAN connection, enabling wireless data synchronization between the head-mounted display device 11 and various servers, administrator terminals, and instructor terminals. This meets the cross-device data interaction requirements of the display positioning module 1, model fusion module 2, intelligent guidance module 3, and collaborative training module 4. The administrator terminal can perform background configuration, permission management, and data maintenance for the entire training system. The instructor terminal supports remote intervention by instructors in the training process, allowing them to view trainees' operation data in real time, issue guidance instructions, and configure assessment rules. The MR glasses, which are the head-mounted display device 11 in the display positioning module 1, are the core operation and information receiving terminal for trainees. Multiple MR glasses are connected to the overall architecture through a wireless network, enabling multi-user collaborative training device networking.

[0053] Furthermore, refer to Figure 3This document outlines the development flowchart for the MR system in training auxiliary equipment. The core development steps of the model fusion module 2—building the equipment model and achieving virtual-real fusion—are as follows: 1) Main equipment 3D modeling: This is completed by the model building unit 21 of the model fusion module 2. Based on industrial equipment CAD design drawings, a high-fidelity 3D model of the main nuclear power plant equipment is built using professional 3D modeling tools. 2) SLAM environment reconstruction: This is performed by the spatial positioning unit 12 of the display positioning module 1. The training site is scanned using a spatial positioning camera array and SLAM algorithm to collect environmental data and generate a 3D point cloud map, providing a realistic environmental foundation for virtual-real fusion. 3) MR virtual-real fusion development: This is completed by the virtual-real fusion unit 22 of the model fusion module 2. Based on the MRTK 3.0 mixed reality toolkit, it is implemented using the positioning data from the spatial positioning unit 12. The development of virtual model interaction involves precise alignment of coordinates and degrees of freedom with the real environment, and the development of multimodal interaction units for the head-mounted display device 11 of the display and positioning module 1. This includes optimizing the triggering logic and interactive response effects of the gesture recognition module 111, eye-tracking module 112, and voice control module 113. Virtual disassembly and assembly development is also included, combining the maintenance process of main equipment in nuclear power plants to develop the disassembly and assembly interaction logic of the virtual model, matching the actual maintenance operation steps and action requirements. The MR system is then deployed to the training site, completing hardware debugging, virtual-real fusion calibration, and network setup. Finally, the development of teaching, training, and assessment modules is completed by the intelligent guidance module 3. This module constructs maintenance training guidance logic based on the knowledge graph unit 31, and sets quantitative assessment rules in conjunction with the assessment and evaluation unit 33, developing functional modules for training guidance and assessment.

[0054] Reference Figure 4This document presents a flowchart illustrating the operational process of using the MR system in training equipment, specifically focusing on evaporator maintenance. This flowchart outlines the core operational steps for trainees conducting single-device maintenance training with this equipment. The process begins with the MR device system startup, completing the hardware startup and software data synchronization of the display positioning module 1, model fusion module 2, and intelligent guidance module 3. After the system self-check, it enters the training preparation state. Next, the trainee selects to begin evaporator maintenance training. The instructor or trainee independently selects the evaporator maintenance training topic on the head-mounted display device 11. The system retrieves the corresponding 3D virtual model and maintenance knowledge graph. Subsequently, the trainee wears the MR head-mounted display (i.e., head-mounted display device 11), and the head-mounted display device 11 automatically connects with the spatial positioning unit 12. Next, positioning calibration is achieved. Then, the system completes the virtual-real fusion of the MR system. The virtual-real fusion unit 22 of the model fusion module 2 accurately overlays the virtual model of the evaporator with the physical equipment or real environment of the training site. The virtual-real fusion maintenance scenario is presented to the trainees through the head-mounted display device 11. The trainees perform disassembly and assembly operations in the scenario through gesture interaction. The gesture recognition module 111 of the head-mounted display device 11 recognizes the maintenance operation gestures and converts them into operation data, realizing the disassembly and assembly interaction with the virtual model of the evaporator. During the entire operation, the system completes automatic assessment and evaluation by the intelligent guidance module 3. Based on the collected operation data, the system performs real-time quantitative assessment from the dimensions of operation completion, operation accuracy, and operation standardization through a multi-objective scoring model.

[0055] Furthermore, such as Figure 5 This is a flowchart of the maintenance training process in the training auxiliary equipment. This process is the core logic of the intelligent guidance module 3 providing real-time dynamic training guidance to trainees. First, the MR virtual maintenance begins. The system retrieves the virtual and real fusion scene corresponding to the maintenance topic, and the trainee enters the maintenance operation process. The first step is to determine whether the tool selection is accurate. If not, the intelligent guidance module 3 prompts the wrong tool selection through the head-mounted display device 11 and provides the correct tool option. If correct, proceed to the next step. The second step is to determine whether the previous disassembly and assembly action has finished playing. If not, the system waits for the previous animation to finish playing, and the disassembly and assembly button becomes clickable. If correct, proceed to the next step. The third step is to determine whether the steps are accurate. If not, the intelligent guidance module 3 prompts the wrong step and plays the correct step animation. If correct, the trainee selects the disassembly and assembly button, and then the system pops up a tool selection screen. The correct tool is selected, and the disassembly and assembly animation plays. After completion, the system automatically proceeds to the next step until the entire maintenance training process is completed. Throughout the process, the display and positioning module 1 collects operation data at a frequency of 100Hz. The intelligent guidance module 3 realizes real-time detection and guidance of operation behavior based on the structured data of the knowledge graph unit 31.

[0056] Furthermore, such as Figure 6The diagram shows the maintenance assessment process in the training auxiliary equipment. This process is the core execution step for the assessment and evaluation unit 33 of the intelligent guidance module 3 to complete the quantitative assessment. It relies on the database server to store and retrieve assessment data. First, the instructor completes the system configuration on the instructor terminal through the equipment maintenance configuration module, which includes 1. disassembly and assembly sequence configuration, 2. step score configuration, 3. assessment time configuration, and 4. interactive animation configuration. After the configuration is completed, the data is synchronized to the database server, and then the assessment begins. The system assessment timer starts, the trainee begins the maintenance operation and selects the parts. After the trainee clicks the disassembly and assembly button, the system judges whether the operation is accurate. If not, it judges that the step is not completed; if so, it judges that the step is completed and plays the disassembly and assembly animation. At the same time, the assessment and evaluation unit 33 completes the step scoring. After all assessment steps are completed, the assessment and evaluation unit 33 performs score statistics based on the score results of each step and synchronizes the assessment score record to the database server. Finally, the trainee submits the exam, and the system automatically generates and outputs a standardized industrial equipment maintenance training assessment report.

[0057] Further reference Figure 7 This is a flowchart of multi-user synchronous collaboration in training auxiliary equipment. This process is the core synchronization logic for multi-user collaborative maintenance training implemented by the collaborative training module 4. It is developed based on the UnityNetcode network framework and is completed by the state synchronization unit 41 and the space sharing unit 42 of the collaborative training module 4. Specifically, one trainer's device is used as the host, and the devices of the other trainers (student A and student B) are used as slaves. All devices are aligned in spatial coordinate system through the space sharing unit 42 of the collaborative training module 4. When any trainer performs a maintenance operation (such as student A performing the operation of sending bolt rotation angle), the operation data is transmitted to the host in real time. The host processes the operation data through the UnityNetcode network framework and synchronizes the state to all slaves. After receiving the synchronization data, each slave updates the virtual bolt posture, so that the operation state of the virtual model in the head-mounted display device 11 of all trainers is highly consistent. This ensures that when multiple users carry out collaborative maintenance operations in the same virtual-real fusion scene, the operation position and the device state are accurately matched, achieving a collaborative training effect where what you see is the same scene.

[0058] Furthermore, such as Figure 8The diagram shows the process of binding physical tools in training aids. This process is the core logic of the display positioning module 1 to accurately bind physical maintenance tools with virtual models, achieving real-time synchronization between physical tool operations and virtual model actions. Specifically, a marker ball is first mounted on the physical maintenance tool. When the trainee moves the physical tool, the marker ball moves. The head-mounted display device 11 (HoloLens) of the display positioning module 1 captures the spatial position and motion trajectory of the marker ball through its built-in camera, generating visual sensing data. This data is transmitted to the Unity engine for processing. The Unity engine, combined with the positioning data from the spatial positioning unit 12, calculates the tool pose in real time. Finally, based on the calculated pose data, it drives the corresponding virtual tool in the virtual model to move synchronously, achieving accurate matching of the poses of the physical and virtual tools. This allows the trainee's physical tool operations to be directly mapped to the virtual model, enhancing the realistic operational experience of maintenance training.

[0059] Finally, it should be noted that the methods and devices described in detail above are merely embodiments, and those skilled in the art can modify these embodiments in different ways as long as they do not depart from the scope of the present invention.

Claims

1. A training aid device, characterized in that: include, The display positioning module (1) is used to realize hybrid virtual-real interaction and spatial positioning, and to collect operation and environmental data; The model fusion module (2) interacts with the display positioning module (1) to construct a device model and achieve virtual-real overlay; The intelligent guidance module (3) is connected to the display positioning module (1) and the model fusion module (2) respectively, and is used to provide training guidance; The display positioning module (1) transmits the collected data to the model fusion module (2) and the intelligent guidance module (3). The model fusion module (2) presents the virtual-real fusion scene through the display positioning module (1). The intelligent guidance module (3) generates guidance or warning instructions based on the data processing results and outputs them through the display positioning module (1).

2. The training aid equipment according to claim 1, characterized in that: It also includes a collaborative training module (4), which achieves data synchronization with the display positioning module (1), the model fusion module (2) and the intelligent guidance module (3), and is used to receive the interactive data of each module and realize the status synchronization between multiple devices, supporting multi-user collaborative maintenance training.

3. The training aid equipment according to claim 2, characterized in that: The display positioning module (1) includes a head-mounted display device (11) and a spatial positioning unit (12). The head-mounted display device (11) is used to receive and present virtual-real fusion scene data, guidance or warning instructions, and integrate a multimodal interaction unit to realize interaction with the virtual model. The spatial positioning unit (12) uses an extended multi-source sensor data fusion algorithm to fuse multi-source sensor data, realize spatial positioning, and output positioning data to the model fusion module (2).

4. The training aid equipment according to claim 3, characterized in that: The multimodal interaction unit includes a gesture recognition module (111), an eye tracking module (112), and a voice control module (113). The gesture recognition module (111) recognizes the operator's maintenance operation gestures and converts them into operation data. The eye tracking module (112) triggers information prompts based on the operator's gaze point. The voice control module (113) receives voice commands and retrieves relevant maintenance information.

5. The training aid device according to any one of claims 2 to 4, characterized in that: The model fusion module (2) includes a model building unit (21) and a virtual-real fusion unit (22). The model building unit (21) builds a three-dimensional model of the device and optimizes its performance. The virtual-real fusion unit (22) receives positioning data from the spatial positioning unit (12) to achieve coordinate matching and precise alignment of the virtual model and the physical device.

6. The training aid equipment according to claim 5, characterized in that: The intelligent guidance module (3) can also perform fault injection and assessment, including a knowledge graph unit (31) and an assessment unit (33). The knowledge graph unit (31) integrates industrial equipment maintenance-related data to construct a structured knowledge graph, providing a basis for the detection and judgment of operational behavior. The assessment unit (33) uses a multi-objective scoring model based on operational data to quantitatively assess maintenance training operations and generate assessment data.

7. The training aid equipment according to claim 6, characterized in that: The collaborative training module (4) includes a state synchronization unit (41) and a space sharing unit (42). The state synchronization unit (41) realizes low-latency synchronization of virtual scene states among multiple devices based on a network framework. The space sharing unit (42) realizes spatial coordinate system alignment of all participants through a distributed spatial anchor point system.

8. A training method, characterized in that: Includes the training aids described in any one of claims 1 to 7; as well as, Build 3D models of industrial equipment and optimize model performance; Deploy MR hardware devices, collect training venue environmental data, and complete the virtual-real fusion calibration of virtual models with physical devices and real environments; Integrate relevant data on industrial equipment maintenance, construct a structured knowledge graph for maintenance training, and set quantitative assessment rules; The MR device presents a virtual-real integrated maintenance training scenario, collects the operator's operation data, and detects and judges the operation behavior in real time based on the knowledge graph, and automatically outputs guidance or warning information. Based on operational data, the maintenance training operations are quantitatively assessed according to quantitative assessment rules, and an industrial equipment maintenance training assessment report is generated and output.

9. The training method according to claim 8, characterized in that: An extended multi-source sensor data fusion algorithm is used to fuse multi-source sensor data to achieve spatial positioning. Based on the positioning data, the virtual model and the physical device are accurately aligned in degrees of freedom. At the same time, the multimodal interaction triggering logic of MR hardware devices is optimized.

10. The training method according to claim 8 or 9, characterized in that: When an operator is detected to have violated regulations, the fault association rules based on the knowledge graph automatically trigger a fault scenario simulation and simultaneously output the corresponding safety warning information. A multi-objective scoring model is adopted to conduct quantitative assessment from at least two dimensions: operation completion, operation accuracy, and operation standardization.

Images