Mixed reality industrial simulation training system

Abstract

The invention provides a mixed reality industrial simulation training system which is composed of a space fusion perception unit, a multi-physics field simulation unit, a cooperative training management unit and a self-adaptive teaching engine. The space fusion sensing unit is composed of a real-time environment modeling and positioning subsystem and a physical interaction detection algorithm, and the real-time environment modeling and positioning subsystem dynamically constructs a three-dimensional space coordinate system of a real industrial scene through a visual sensor array and a radio frequency scanning device; generating a pose matrix of the virtual equipment model in the perspective display interface; the physical interaction detection algorithm realizes contact force feedback between virtual and real objects based on dynamic solution of collision volume, and when an operator triggers a virtual switch through a handheld controller, a real equipment control signal is synchronously driven; the multi-physics field simulation unit comprises a physical engine core module and a fault injection interface.

Description

Technical Field

[0001] The present invention belongs to the field of simulation training, and in particular relates to a mixed reality industrial simulation training system. Background Art

[0002] Currently, industrial simulation training systems face significant technical bottlenecks in complex industrial scenarios such as mining, machinery, electricity, construction, hazardous chemicals, power generation, and oil and gas. This severely restricts their widespread adoption in safety training for high-precision, high-risk operations such as manufacturing and equipment operation and maintenance. Existing systems often rely on single visual sensors or inertial measurement units for spatial perception, making it difficult to achieve high-precision 3D modeling of real-world industrial environments. Positioning errors often exceed 5cm in workshops with strong electromagnetic interference and multiple metal reflections, resulting in significant deviations between the virtual device model and the real scene. This makes it difficult for operators to accurately align virtual switches with physical control terminals, leading to an error rate of over 30% during training.

[0003] Physics simulation engines are generally limited to rigid-body dynamics simulations and lack the multi-physics coupling computational capabilities to account for the deformation of flexible materials and the dynamic properties of fluids. For example, when simulating oil pipeline maintenance, existing systems cannot accurately reflect the feedback effects of fluid pressure fluctuations on valve operation within the pipeline, resulting in a training effect that matches real-world operating conditions less than 40%. Collaborative training systems also face significant communication latency issues. Traditional TCP / IP protocol stacks experience transmission delays exceeding 200ms when synchronizing commands across devices, which can easily lead to timing errors when multiple trainees collaborate on hydraulic systems. Furthermore, these systems lack dynamic risk deduction mechanisms, making it impossible to predict the chain reactions triggered by misoperations (such as pressure surges caused by valve closures). Consequently, accident simulations fall short of industrial safety standards. Adaptive teaching modules generally utilize static knowledge graphs, failing to dynamically adjust training difficulty based on trainee physiological characteristics (such as abnormal eye movement patterns and muscle fatigue). The error rate in eye focus recognition exceeds 25%, leading to a mismatch between training tasks and trainee abilities and a reduction in skill acquisition efficiency by over 50%.

[0004] Privacy protection mechanisms suffer from serious flaws. Traditional data aggregation methods directly upload raw operational behavior data to the cloud, failing to decouple identity information from behavioral characteristics. In multi-base joint training scenarios, the risk index for sensitive operational data leakage exceeds the ISO 27001 security threshold. Existing fault simulation systems only support the injection of predefined abnormal patterns and are unable to analyze register state changes in industrial control systems (such as DCS signal jitter in PLCs) in real time. This leads to a disconnect between virtual device abnormal behavior and real-world control logic. The accuracy of mechanical fault simulations is less than 60%, hindering trainees from developing effective emergency decision-making capabilities.

[0005] Hardware-side sensor fusion solutions have inherent flaws. The time synchronization error between visual and radio frequency signals exceeds 10ms. During operator training for high-speed rotating equipment (such as turbines), the distortion rate of virtual-real collision detection results reaches 15%. Force feedback delays prevent operators from perceiving torque changes in virtual tools, reducing muscle memory formation efficiency by 70%. Furthermore, existing systems suffer from poor cross-platform compatibility and a lack of standardized conversion modules between different manufacturers' industrial protocols (such as PROFINET and EtherCAT) and MR device interfaces. This increases enterprise deployment costs by more than threefold, and the system's scalability is limited, making it difficult to adapt to the rapid iteration training requirements of smart manufacturing production lines.

[0006] These technical defects collectively result in the existing industrial training systems being unable to meet the requirements of high-end industrial scenarios in terms of accuracy, authenticity, security, and intelligence. There is an urgent need to achieve a systematic upgrade through core technology breakthroughs such as multimodal perception fusion, multi-physics field simulation engines, and low-latency collaborative architectures. Summary of the Invention

[0007] The present invention proposes a mixed reality industrial simulation training system. This technical solution solves the technical problems in the field of industrial training, such as unnatural interaction between virtual and real environments, insufficient authenticity of physical simulation, difficulty in multi-person collaborative training, lack of personalized teaching, and weak cross-domain data privacy protection, and realizes immersive training in high-fidelity industrial equipment operation and emergency response.

[0008] The technical solution of the present invention is implemented as follows: a mixed reality industrial simulation training system, consisting of a spatial fusion perception unit, a multi-physics field simulation unit, a collaborative training management unit, and an adaptive teaching engine;

[0009] The spatial fusion perception unit consists of a real-time environment modeling and positioning subsystem and a physical interaction detection algorithm. The real-time environment modeling and positioning subsystem uses a visual sensor array and a radio frequency scanning device to dynamically construct a three-dimensional spatial coordinate system for the real industrial scene and generate a pose matrix for the virtual device model in a perspective display interface. The physical interaction detection algorithm implements contact force feedback between virtual and real objects based on the dynamic solution of collision volumes. When the operator triggers a virtual switch using a handheld controller, it synchronously drives the real device control signal.

[0010] The multi-physics simulation unit includes a physics engine core module and a fault injection interface. The physics engine core module generates a continuous physical simulation of the equipment's operating state by solving the rigid body motion equations, flexible body deformation gradients, and the fluid Navier-Stokes equations. The fault injection interface receives a unidirectional data stream from the industrial control device, analyzes the PLC register state changes, maps them to the virtual model, and dynamically activates a preset sequence of mechanical abnormal behavior.

[0011] The collaborative training management unit is equipped with a low-latency communication protocol stack and a risk deduction decision tree. This low-latency communication protocol stack establishes a synchronization channel for operation instructions between multiple terminals, encoding trainee action data into spatiotemporal event markers. The risk deduction decision tree uses a temporal feature extraction network to predict operation chain reactions and generates three-dimensional visual warning signs based on cost function evaluation.

[0012] The adaptive teaching engine integrates a knowledge topology analyzer and a privacy-preserving data aggregator. The knowledge topology analyzer parses the multi-hop association rules between device nodes, and dynamically reorganizes the training task topology structure by combining the eye movement focus hotspots and electromyography signal strength in the trainees' physiological monitoring data stream. The privacy-preserving data aggregator adopts a feature decoupling strategy to separate operational behavior data and identity information, realizing encrypted migration and fusion optimization of cross-domain model parameters.

[0013] Existing VR training systems rely on purely virtual environments, resulting in spatial matching errors between real industrial equipment and virtual elements often exceeding the centimeter level, making it impossible to achieve the sub-millimeter virtual-real fusion required for precision operation training. The fundamental technical difficulty lies in the insufficient temporal and spatial synchronization accuracy of multimodal environmental perception data. Traditional visual SLAM algorithms are susceptible to positioning drift in complex industrial scenarios due to metal reflections and electromagnetic interference. The physical simulation of traditional virtual training only supports rigid body discrete collision detection, and lacks the ability to jointly simulate continuous physical processes such as flexible deformation and fluid dynamics, making it difficult to simulate real fault forms such as hydraulic leakage and mechanical fatigue.

[0014] The technical bottleneck lies in the contradiction between the computational complexity and real-time performance of multi-physics field coupling solutions. Existing physics engines find it difficult to simultaneously solve the Navier-Stokes equations and nonlinear finite element models within an 8ms delay. Existing collaborative training systems mostly use a local area network transmission architecture. The synchronization delay of multi-terminal operation instructions in a wide area network exceeds 100ms, resulting in inconsistent virtual device states when multiple people collaborate. The core obstacle lies in the inefficiency of the spatiotemporal event marking encoding mechanism. The traditional UDP protocol cannot complete the verification and reordering of action data packets with timestamps within 30ms. Traditional virtual training uses a fixed script teaching model and cannot dynamically adjust training content according to the physiological characteristics of trainees. The key constraint is the lack of a correlation analysis model between device knowledge topology and biosignals. The gaze hotspot data obtained by the existing eye tracking system at a sampling rate of 120Hz is difficult to map in real time to the fault diagnosis path of complex device structures.

[0015] Existing data sharing solutions often rely on centralized storage, leading to the risk of leaking sensitive enterprise operational data. Technical barriers lie in the traceability of gradient updates within federated learning frameworks, and the injection of noise into traditional differential privacy mechanisms can reduce model accuracy by over 15%. This system achieves submillimeter spatial matching through a multimodal perception engine, overcoming the challenges of visual-RF data fusion algorithms in metallic environments. A multibody dynamics coupling solver was developed to overcome the bottleneck of real-time multi-physics simulation. A spatiotemporal event labeling encoding protocol was designed to reduce wide area network latency to under 30ms. A knowledge topology analyzer was constructed to dynamically map physiological signals to device nodes. Finally, an innovative privacy-preserving data aggregator was developed to enable secure cross-domain sharing while maintaining model performance.

[0016] As a preferred embodiment, the perspective display interface of the spatial fusion perception unit adopts a multi-layer waveguide optical structure, in which the first waveguide layer is configured with a diffraction grating array to couple the light field information of the virtual device model to the visual focal plane, thereby achieving depth perception matching between the virtual object and the real scene; the second waveguide layer integrates a dynamic dimming unit, which adjusts the transmittance of each pixel through a microfluidic liquid crystal unit, so that the virtual warning sign maintains visual significance under different ambient lighting conditions.

[0017] As a preferred embodiment, the handheld controller has a built-in six-degree-of-freedom tracking module, and a pressure-sensitive array is arranged on the surface of its end effector. When the operator triggers the electric switch, the controller sends the contact surface deformation gradient data to the physical interaction detection algorithm, triggering the generation of a force feedback waveform for virtual-real linkage. After being optimized by a convolutional neural network, this waveform drives the linear actuator to produce a step-by-step damping change.

[0018] As a preferred embodiment, the physical engine core module of the multi-physics field simulation unit includes a hybrid solver architecture, in which the rigid body motion equation solver adopts the constrained Jacobian matrix decomposition method to establish the joint constraint topology of the equipment motion chain; the flexible body deformation gradient calculation introduces a nonlinear finite element model, and simulates the plastic deformation caused by metal fatigue by dynamically updating the unit stiffness matrix; the fluid simulation module constructs an adaptive octree space division grid, solves the free surface flow based on the particle-grid hybrid method, and couples it with the temperature field control equation to calculate the phase change effect; the fault injection interface configures a protocol conversion middleware, parses the PLC register status code into the abnormal parameter offset of the equipment kinematic chain, and triggers the progressive fault simulation of gearbox tooth breakage and hydraulic valve sticking.

[0019] As a preferred embodiment, the workflow of the privacy-preserving data aggregator is as follows:

[0020] In the feature decoupling stage, the original operation data is separated into the device operation feature vector and the identity identification latent space through the variational autoencoder;

[0021] During the encryption migration phase, a homomorphic encryption algorithm is used to blind the feature vectors, and a proxy re-encryption gateway is used to achieve directional transmission of cross-domain model parameters.

[0022] In the federated optimization phase, a weight distribution mechanism based on contribution entropy is designed to dynamically adjust the aggregation weight according to the information entropy value updated by each participant. At the same time, an adversarial generative network is used to compensate for the model performance degradation caused by data distribution offset.

[0023] As a preferred embodiment, the workflow of the real-time environment modeling and positioning subsystem is as follows:

[0024] During the initialization phase, a sparse point cloud map is constructed through multi-view visual feature matching, and the penetrating detection data of millimeter-wave radar is integrated to complete the 3D structure of the obscured area.

[0025] During the dynamic tracking phase, a tightly coupled visual-inertial odometry system is used to calculate the six-degree-of-freedom pose of the head-mounted display device, while a semantic segmentation network is used to identify the boundaries of the interactive area of ​​the industrial equipment.

[0026] During the calibration maintenance phase, when it is detected that the virtual object projection residual exceeds the threshold, the checkerboard calibration mode is automatically activated, the camera-radar extrinsic parameter matrix is ​​re-optimized using the pre-deployed fiducial marker array, and the accumulated pose error is eliminated through the optimization algorithm.

[0027] After adopting the above technical solution, the beneficial effects of the present invention are as follows: the system significantly improves the authenticity and effectiveness of industrial simulation training. The spatial fusion perception unit improves the virtual-reality matching accuracy to sub-millimeter level through visual-radio frequency hybrid positioning, enabling operators to accurately control virtual-reality integrated industrial equipment, and improving operational accuracy compared to traditional VR training.

[0028] The multi-physics simulation unit supports joint simulation of rigid-flexible flow and multi-physics fields, realistically simulating the stress distribution changes and lubricant leakage diffusion process during gearbox tooth breakage, enhancing fault reproduction fidelity. The collaborative training management unit's low-latency communication protocol reduces command synchronization error to less than 5ms during multi-person collaborative training, enabling cross-regional teams to complete challenging collaborative tasks such as precision assembly. The risk deduction decision tree predicts operational errors in advance through temporal feature extraction, and the dynamic projection of 3D visual warning icons accelerates trainees' emergency response. The adaptive teaching engine's knowledge topology analyzer automatically reinforces weak links based on eye movement hotspots, improving skill acquisition efficiency and shortening training cycles. The privacy-preserving data aggregator utilizes a feature decoupling strategy to achieve model performance retention within a federated learning framework while ensuring that operational data is untraceable, addressing the security challenges of industrial data sharing. The system as a whole reduces trainees' operational errors and improves the accuracy of high-risk incident response. It also supports the monthly independent expansion of new industrial fault models, providing an evolving and highly secure MR training platform for the development of intelligent manufacturing talent. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 This is a block diagram of the spatial fusion perception unit system of the present invention;

[0031] Figure 2 This is a system block diagram of the multi-physics field simulation unit of the present invention;

[0032] Figure 3 This is a system block diagram of the collaborative training management unit of the present invention;

[0033] Figure 4 This is a block diagram of the adaptive teaching engine system of the present invention;

[0034] Figure 5 This is a system block diagram of the spatial fusion perception unit of the present invention. DETAILED DESCRIPTION

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0036] Example:

[0037] like Figure 1 As shown, a mixed reality industrial simulation training system consists of a spatial fusion perception unit, a multi-physics field simulation unit, a collaborative training management unit, and an adaptive teaching engine;

[0038] The spatial fusion perception unit consists of a real-time environment modeling and positioning subsystem and a physical interaction detection algorithm. The real-time environment modeling and positioning subsystem uses a visual sensor array and a radio frequency scanning device to dynamically construct a three-dimensional spatial coordinate system for the real industrial scene and generate a pose matrix for the virtual device model in a perspective display interface. The physical interaction detection algorithm implements contact force feedback between virtual and real objects based on the dynamic solution of collision volumes. When the operator triggers a virtual switch using a handheld controller, it synchronously drives the real device control signal.

[0039] The multi-physics simulation unit includes a physics engine core module and a fault injection interface. The physics engine core module generates a continuous physical simulation of the equipment's operating state by solving the rigid body motion equations, flexible body deformation gradients, and the fluid Navier-Stokes equations. The fault injection interface receives a unidirectional data stream from the industrial control device, analyzes the PLC register state changes, maps them to the virtual model, and dynamically activates a preset sequence of mechanical abnormal behavior.

[0040] The collaborative training management unit is equipped with a low-latency communication protocol stack and a risk deduction decision tree. This low-latency communication protocol stack establishes a synchronization channel for operation instructions between multiple terminals, encoding trainee action data into spatiotemporal event markers. The risk deduction decision tree uses a temporal feature extraction network to predict operation chain reactions and generates three-dimensional visual warning signs based on cost function evaluation.

[0041] The adaptive teaching engine integrates a knowledge topology analyzer and a privacy-preserving data aggregator. The knowledge topology analyzer parses the multi-hop association rules between device nodes, and dynamically reorganizes the training task topology structure by combining the eye movement focus hotspots and electromyography signal strength in the trainees' physiological monitoring data stream. The privacy-preserving data aggregator adopts a feature decoupling strategy to separate operational behavior data and identity information, realizing encrypted migration and fusion optimization of cross-domain model parameters.

[0042] This mixed-reality industrial simulation training system demonstrates its full-chain technology integration capabilities in high-end equipment operation and maintenance training scenarios. Through its core architecture of multi-dimensional perception, multi-physics coupling, and intelligent teaching, it creates an immersive training environment that deeply blends virtual and real life. During system startup, the spatial fusion perception unit uses a distributed visual array and radio frequency scanning device to create a high-precision three-dimensional spatial model of the real industrial scene. Based on real-time point cloud data and feature matching algorithms, it generates a spatial pose matrix for virtual equipment components, achieving millimeter-level alignment between virtual and real space. A perspective display interface dynamically renders the virtual equipment model, and a physical interaction detection algorithm is used to calculate the contact and collision volume between the operator and the virtual object in real time. Tactile feedback accurately simulates mechanical properties such as assembly resistance and torque transmission, ensuring consistent operation with real-world conditions. When trainees trigger a virtual equipment switch using a handheld controller, the system simultaneously drives real-world industrial control signals, achieving closed-loop control that integrates virtual and real life. For example, during turbine assembly and disassembly training, the rotation angle of the virtual wrench is dynamically mapped to the tightening torque of the real bolt. Abnormal operation triggers the physics engine to calculate component deformation in real time and generate visual feedback of progressive damage.

[0043] The multi-physics simulation unit utilizes a multi-domain coupled solver, integrating rigid-body dynamics, flexible-body deformation, and fluid dynamics models to accurately replicate the operating states of complex industrial equipment. A fault injection interface uses industrial protocols to analyze the register states of field control devices in real time, dynamically activating pre-set mechanical fault sequences. For example, in hydraulic system training, when simulating a stuck proportional valve, the system simultaneously calculates pipeline pressure fluctuations and drives a real pressure gauge to display abnormal values. Simultaneously, it generates a virtual oil leak diffusion effect based on fluid equations. The collaborative training management unit utilizes a low-latency communication protocol to synchronize multi-terminal operation instructions in time and space, encoding student actions as time-stamped spatiotemporal event markers to ensure temporal consistency across cross-role collaborative operations. The risk deduction decision tree utilizes a deep temporal network to analyze operation chain reactions and, combined with cost function evaluation, generates a three-dimensional holographic warning icon. When a violation of a possible cascading failure is detected, an audible and visual alarm is immediately triggered, the simulation process is frozen, and an accident evolution animation is simultaneously pushed to all terminals.

[0044] The adaptive teaching engine analyzes the multi-level association rules of the equipment system based on the knowledge topology graph and captures the trainee's cognitive load and operational status in real time through multimodal physiological sensor data: the eye tracking module analyzes the distribution of visual attention and identifies gaze deviation behavior at key operational nodes; the surface electromyography sensor monitors the activation pattern of the hand muscle groups and evaluates the stability of operational force control. Based on this, the system dynamically reconstructs the training task topology. For example, if it detects that a trainee has neglected to wear protective equipment when operating in a high-temperature area, it automatically inserts a safety standard reinforcement training module. The privacy-preserving data aggregator uses feature decoupling technology to separate operational behavior characteristics from personal identity information. Through the federated learning framework, it realizes the encrypted migration and aggregate optimization of cross-domain model parameters, ensuring compliance with data sovereignty regulations in multi-national and multi-base joint training, while continuously improving the physical simulation accuracy of virtual equipment.

[0045] In a typical aircraft engine maintenance scenario, the system constructs a virtual assembly environment featuring compressor blades and combustor components. Trainees use tactile gloves to sense the micron-level tolerances of blade grooves, and a visual interface dynamically displays the compliance zones within the assembly path. During rotor dynamic balancing, the multiphysics engine calculates the rotor's mass distribution and critical speed characteristics in real time, presenting vibration modal cloud maps through the head-mounted display (HMD). A fault injection interface simulates spectral anomalies caused by bearing wear, requiring trainees to diagnose the source of the fault using a virtual spectrum analyzer. In collaborative training mode, operational data from the mechanical assembly and electrical inspection teams is shared in real time via a low-latency channel. Any incorrect operation by any trainee triggers a three-dimensional alarm and initiates emergency response plan simulations. After training, the system generates a multi-dimensional evaluation report based on operational accuracy, process compliance, and risk management effectiveness. Using a reinforcement learning algorithm, the system dynamically optimizes the fault complexity and teaching cadence of subsequent courses, creating a competency-based, self-evolving training system. This solution overcomes technical bottlenecks in traditional training systems, such as fidelity of virtual-reality interaction, temporal consistency of multi-role collaboration, and personalized teaching adaptability, providing a safe, controllable, efficient, and compliant skills transfer solution for high-end equipment manufacturing.

[0046] The underground coal mining machine operation training system is used as the scenario for the application of technical solutions:

[0047] 1. Application of spatial fusion perception unit

[0048] Environmental modeling: An explosion-proof visual sensor array (including an infrared camera) is used to scan the underground fully mechanized mining face, dynamically construct the three-dimensional spatial coordinate system of the hydraulic support and scraper conveyor (accuracy ±2cm), and generate a virtual coal mining machine model in the head-mounted display (pose matrix update frequency 90Hz).

[0049] Physical interaction: When students use handheld controllers to simulate adjusting the height of the coal mining drum, the collision volume detection algorithm calculates the contact force between the pick and the coal seam in real time (based on the Mohr-Coulomb model). The tactile gloves provide feedback on the vibration intensity (adjustable from 0 to 10N). At the same time, the OPC UA protocol drives a real PLC to simulate changes in the coal mining machine's traction speed.

[0050] 2. Multi-physics simulation unit operation

[0051] Normal operating conditions: Real-time physics engine analysis: rigid body dynamics (coal mining machine wheel and guide rail friction coefficient μ = 0.15); flexible body deformation (compressive bending of the hydraulic support guard plate δmax ≤ 5mm); fluid simulation (spray dust suppression system water mist diffusion radius R = 3m); fault injection: When the teacher triggers the "drum motor overload" fault code: analysis of the abnormal value of PLC register 0x3E8 address (> 150% rated current); activation of the virtual model abnormal sequence: motor smoke particle effect → temperature alarm color change (red → yellow → red) → automatic shutdown protection action (delay 800ms)

[0052] 3. Implementation of collaborative training management unit

[0053] Multi-machine collaboration: Five trainees each operated a coal mining machine, a support worker, and an inspector. A low-latency protocol stack (latency <15ms) synchronized support jack travel data (encoded as a [timestamp, device ID, displacement] triplet). When the risk decision tree detected "support not promptly supported → roof subsidence > threshold," a red pulsed light band alert was projected on the AR interface. Accident simulation: A time-series network identified the trainee's misoperation chain: dust removal spray not activated → methane concentration virtually rose to 1.5% → triggering a Level 3 alarm (3D explosion shock wave animation).

[0054] 4. Adaptive teaching engine optimization

[0055] Personalized training: A knowledge topology analyzer establishes coal mining process association rules. The main frequency of the cutting motor power vibration signal shifts to 85Hz, requiring inspection of the reduction gears. Specialized training modules are automatically inserted based on trainee eye movement data (gaze duration of the dust removal valve < 2 seconds). Data security: A feature decoupler separates operational data (e.g., "support operation response time = 1.2 seconds") from identity information. Federated learning aggregates fault handling experience from multiple mining areas (model parameters are encrypted with AES-256).

[0056] Application Description of the Spatial Fusion Perception Unit's Perspective Display Interface: During high-temperature, high-pressure chemical equipment inspection training, operators wearing head-mounted displays (HMDs) enter a real workshop. The diffraction grating array in the first waveguide layer accurately projects a virtual pressure gauge model onto the flange connection of the real pipeline, with a visual focal plane depth matching error of less than 0.3mm, ensuring spatial perspective consistency between the virtual dial pointer and the actual valve position. When ambient light suddenly increases to 80,000 lux due to the opening of the workshop skylight, the microfluidic liquid crystal unit in the second waveguide layer increases the transmittance of the virtual "High Temperature Danger Zone" warning sign to 78% within 15ms. Through local pixel brightness compensation, the flashing red sign maintains a visual saliency of above 0.9 even in strong light. The operator can clearly identify the virtual thermal radiation contour of the steam pipeline leak area. At the same time, the mechanical pointer reading of the real pressure gauge and the virtual temperature indication precisely overlap in the spatial dimension, achieving a seamless presentation of virtual and real alarm information.

[0057] Application description of the handheld controller force feedback mechanism: In the virtual assembly and adjustment training of precision machine tools, when the trainee uses the handheld controller to tighten the virtual spindle bearing, the pressure-sensitive array of the end effector detects the deformation gradient distribution of the thumb pressing area in real time. When the contact pressure reaches the 8N threshold, the controller sends the deformation tensor data containing 24 pressure sensing units to the physical interaction detection algorithm. The algorithm analyzes the stress distribution pattern of the contact surface through the pre-trained 3D convolutional neural network and generates a force feedback waveform with torque-sensing characteristics. The linear actuator generates a stepped damping change based on the waveform characteristics, providing smooth resistance (2N·m) in the initial rotation stage. When the virtual thread pair enters the critical pre-tightening state, the damping force increases stepwise to 5N·m and is accompanied by 3Hz micro-vibration, simulating the working characteristics of the overload protection mechanism of a real wrench. This allows the trainees to accurately grasp the critical value of the assembly torque through tactile perception and avoid the "over-tightening" error in virtual training.

[0058] Application description of the hybrid solver of the multi-physics simulation unit: In the fault diagnosis training of wind turbines, the core module of the physics engine simultaneously solves the rigid body motion of the blades and the flexible deformation of the gearbox: the constrained Jacobian matrix decomposition method establishes the motion constraint equations of each component of the planetary gear train. When simulating the main shaft bearing wear failure, the nonlinear finite element model dynamically updates the stiffness matrix of the gear contact surface and calculates the vibration spectrum changes caused by the plastic deformation of the tooth surface; the fluid simulation module uses an adaptive octree grid to divide the air flow field around the blades, and the particle-grid hybrid method captures the turbulent separation effect, and couples it with the gearbox oil temperature field to calculate the impact of changes in lubricating oil viscosity on heat dissipation efficiency; the fault injection interface analyzes the bearing temperature anomaly code output by the PLC in real time, converts it into a progressive offset parameter of the gear meshing clearance, and simulates the entire process from initial pitting to tooth root fracture within 23 seconds. The oil leakage particles exhibit non-Newtonian fluid characteristics under the action of temperature gradient. The training system simultaneously generates multi-modal fault characterizations including abnormal vibration and noise, oil pressure drop, and temperature alarm.

[0059] Application Description of Privacy-Preserving Data Aggregator: In cross-automaker joint training, a variational autoencoder decomposes operation data into "gear shift operation timing features" (128-dimensional vectors) and "engineer identity latent codes" (32-dimensional vectors). After the feature vectors are homomorphically encrypted with Paillier, they are transmitted to the OEM's federated server via a proxy re-encryption gateway. During the federated optimization phase, the system calculates the information entropy of the gradients uploaded by each 4S store and assigns an aggregation weight of 0.3 or above to participants with an entropy value higher than 2.4. At the same time, the adversarial generative network learns the data distribution differences between each store and generates compensatory synthetic gradients. When the "dual-clutch transmission misoperation" feature uploaded by a 4S store is identified as a new attack mode, the system completes a global model update within 24 hours while protecting the store's data privacy. This automatically increases the deduction weight of the fault scenario in the virtual training systems of all participants, and makes it impossible to trace the data source backwards, thus achieving a balance between knowledge sharing and privacy protection.

[0060] Application Description of the Real-Time Environment Modeling and Positioning Subsystem: During the initialization phase of a nuclear power valve maintenance training scenario, multi-perspective visual feature matching identifies SIFT feature points on the reactor pressure vessel flange surface, constructing a sparse point cloud containing 382 key points. Millimeter-wave radar penetrates the insulation layer to detect obscured bolt holes and complete the 3D structural model. During the dynamic tracking phase, a tightly coupled visual-inertial odometry calculates the headset pose at a frequency of 200Hz. As the operator leans over to inspect the underlying pipeline, a semantic segmentation network identifies the operable area boundaries of the real valve in real time, limiting the spatial range of the virtual disassembly and assembly tools to the ±15° safe zone for injection pipeline maintenance. After four hours of continuous operation, the system detected a 1.2mm residual between the virtual wrench projection and the real valve position and automatically activated calibration mode. The fiducial marker array on the checkerboard calibration plate provides 32 feature points. The camera-radar extrinsic parameter matrix is ​​corrected using a Lie group optimization algorithm, reducing the cumulative pose error from 0.7mm to 0.1mm, ensuring the long-term operational accuracy of the training system in high-radiation environments.

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A mixed reality industrial simulation training system, characterized by: It consists of a spatial fusion perception unit, a multi-physics field simulation unit, a collaborative training management unit, and an adaptive teaching engine; The spatial fusion perception unit consists of a real-time environment modeling and positioning subsystem and a physical interaction detection algorithm. The real-time environment modeling and positioning subsystem uses a visual sensor array and a radio frequency scanning device to dynamically construct a three-dimensional spatial coordinate system for the real industrial scene and generate a pose matrix for the virtual device model in a perspective display interface. The physical interaction detection algorithm implements contact force feedback between virtual and real objects based on the dynamic solution of collision volumes. When the operator triggers a virtual switch using a handheld controller, it synchronously drives the real device control signal. The multi-physics simulation unit includes a physics engine core module and a fault injection interface. The physics engine core module generates a continuous physical simulation of the equipment's operating state by solving the rigid body motion equations, flexible body deformation gradients, and the fluid Navier-Stokes equations. The fault injection interface receives a unidirectional data stream from the industrial control device, analyzes the PLC register state changes, maps them to the virtual model, and dynamically activates a preset sequence of mechanical abnormal behavior. The collaborative training management unit is equipped with a low-latency communication protocol stack and a risk deduction decision tree. This low-latency communication protocol stack establishes a synchronization channel for operation instructions between multiple terminals, encoding trainee action data into spatiotemporal event markers. The risk deduction decision tree uses a temporal feature extraction network to predict operation chain reactions and generates three-dimensional visual warning signs based on cost function evaluation. The adaptive teaching engine integrates a knowledge topology analyzer and a privacy-preserving data aggregator. The knowledge topology analyzer parses the multi-hop association rules between device nodes, and dynamically reorganizes the training task topology structure by combining the eye movement focus hotspots and electromyography signal strength in the trainees' physiological monitoring data stream. The privacy-preserving data aggregator adopts a feature decoupling strategy to separate operational behavior data and identity information, realizing encrypted migration and fusion optimization of cross-domain model parameters.

2. The mixed reality industrial simulation training system according to claim 1, characterized in that: The perspective display interface of the spatial fusion perception unit adopts a multi-layer waveguide optical structure, in which the first waveguide layer is configured with a diffraction grating array to couple the light field information of the virtual device model to the visual focal plane, achieving depth perception matching between the virtual object and the real scene; the second waveguide layer integrates a dynamic dimming unit, which adjusts the transmittance of each pixel through a microfluidic liquid crystal unit, so that the virtual warning sign maintains visual significance under different ambient lighting conditions.

3. The mixed reality industrial simulation training system according to claim 1, characterized in that: The handheld controller has a built-in six-degree-of-freedom tracking module and a pressure-sensitive array on the surface of its end effector. When the operator triggers the electronically controlled switch, the controller sends contact surface deformation gradient data to the physical interaction detection algorithm, triggering the generation of a force feedback waveform for virtual-real linkage. This waveform, after being optimized by a convolutional neural network, drives the linear actuator to produce a stepped damping change.

4. The mixed reality industrial simulation training system according to claim 1, characterized in that: The core module of the physics engine of the multi-physics simulation unit includes a hybrid solver architecture, in which the rigid body motion equation solver uses the constrained Jacobian matrix decomposition method to establish the joint constraint topology of the device motion chain; the flexible body deformation gradient calculation introduces a nonlinear finite element model, and the plastic deformation caused by metal fatigue is simulated by dynamically updating the unit stiffness matrix; the fluid simulation module constructs an adaptive octree space partitioning grid, solves the free surface flow based on the particle-grid hybrid method, and couples it with the temperature field control equation to calculate the phase change effect; The fault injection interface is configured with a protocol conversion middleware, which parses the PLC register status code into abnormal parameter offsets of the equipment kinematic chain, triggering progressive fault simulations of gearbox tooth breakage and hydraulic valve sticking.

5. The mixed reality industrial simulation training system according to claim 1, characterized in that: The workflow of the privacy-preserving data aggregator is as follows: In the feature decoupling stage, the original operation data is separated into the device operation feature vector and the identity identification latent space through the variational autoencoder; During the encryption migration phase, a homomorphic encryption algorithm is used to blind the feature vectors, and a proxy re-encryption gateway is used to achieve directional transmission of cross-domain model parameters. In the federated optimization phase, a weight distribution mechanism based on contribution entropy is designed to dynamically adjust the aggregation weight according to the information entropy value updated by each participant. At the same time, an adversarial generative network is used to compensate for the model performance degradation caused by data distribution offset.

6. The mixed reality industrial simulation training system according to claim 1, characterized in that: The workflow of the real-time environment modeling and positioning subsystem is as follows: During the initialization phase, a sparse point cloud map is constructed through multi-view visual feature matching, and the penetrating detection data of millimeter-wave radar is integrated to complete the 3D structure of the obscured area. During the dynamic tracking phase, a tightly coupled visual-inertial odometry system is used to calculate the six-degree-of-freedom pose of the head-mounted display device, while a semantic segmentation network is used to identify the boundaries of the interactive area of ​​the industrial equipment. During the calibration maintenance phase, when it is detected that the virtual object projection residual exceeds the threshold, the checkerboard calibration mode is automatically activated, the camera-radar extrinsic parameter matrix is ​​re-optimized using the pre-deployed fiducial marker array, and the accumulated pose error is eliminated through the optimization algorithm.

Images