An industrial meta-universe-based virtual operation training method, system, device and medium

Abstract

This invention relates to the field of semiconductor manufacturing technology, specifically to a virtual operation training method, system, equipment, and medium based on an industrial metaverse. This method no longer relies solely on adjusting the difficulty of cognitive load, but instead maps trainee action deviations, cognitive load, and digital twin process state residuals as counterfactual intervention variables. It generates a fault chain on the equipment causal fault diagram that is strongly correlated with the current weak operation and conforms to process constraints. Through dual-clock synchronization, state prediction, event latching, and residual backfeeding, it generates a state consistency certificate for each simulation segment, ensuring that visual motion, process parameters, and tactile outputs originate from the same process state trajectory. The operation results are decomposed into four attributable indicators: action trajectory deviation, tactile force deviation, process state residual, and cognitive load fluctuation, avoiding evaluation of trainees solely based on final scores or expert trajectory similarity.

Description

Technical Field

[0001] This invention relates to the field of semiconductor manufacturing technology, and more specifically, to a virtual operation training method, system, device, and medium based on an industrial metaverse. Background Technology

[0002] With the rapid development of Industry 4.0 and smart manufacturing, industrial equipment is becoming increasingly complex, precise, and automated. Operators need to master complex equipment operating procedures, process parameters, and abnormal handling processes, leading to a continuous increase in training difficulty and cost.

[0003] Traditional industrial operations training faces the following challenges: 1. High safety risks: Training on real equipment may result in equipment damage, production accidents, or even personal injury or death due to operational errors. 2. High training costs: Using production equipment for training directly impacts production efficiency; training costs for high-precision equipment are extremely high. 3. Limited scenario coverage: Real-world training cannot simulate all failure scenarios, especially low-probability, high-risk abnormal operating conditions; 4. Difficulty in quantifying effectiveness: Traditional training lacks objective operational assessment methods, making it difficult to accurately assess the trainees' level of mastery; 5. Lack of personalization: Fixed training content and pace cannot adapt to the different cognitive abilities and learning speeds of trainees.

[0004] The existing system uses a fixed fault scenario library and cannot dynamically generate the most suitable training scenario based on the trainee's current cognitive state; the existing system only uses a single physics engine and cannot realistically reproduce the process characteristics of industrial equipment (such as dynamic changes in temperature, pressure, and flow); the existing system mainly relies on visual feedback and lacks tactile dimension operation evaluation; trainees cannot perceive key physical characteristics such as operating force and vibration, the evaluation dimension is single and cannot comprehensively reflect the operation quality. Summary of the Invention

[0005] This invention addresses the shortcomings of existing operator training systems, such as the lack of cognitive load-driven predictive scene generation, separation of physics engines and process simulation, and lack of tactile multimodal assessment. It proposes a virtual operator training method, system, equipment, and medium based on an industrial metaverse. Instead of adjusting difficulty solely based on cognitive load, it maps trainee movement deviations, cognitive load, and digital twin process state residuals as counterfactual intervention variables, generating a fault chain on the equipment causal fault diagram that is strongly correlated with the current weak operation and conforms to process constraints. Furthermore, through dual-clock synchronization, state prediction, event latching, and residual backfeeding, it generates a state consistency certificate for each simulation segment, ensuring that visual motion, process parameters, and tactile outputs originate from the same process state trajectory.

[0006] The specific implementation details of this invention are as follows: A virtual operation training method based on the industrial metaverse specifically includes the following steps: Step S1: Collect eye-tracking data, gesture operation data, physiological signal data, and tactile operation data of trainees in the Industrial Metaverse training scenario, and simultaneously collect the equipment motion status of the physics engine and the process status of the process simulation engine. Step S2: Calculate the cognitive load value of the trainee based on the eye movement data, gesture operation data and physiological signal data; and obtain the operation trajectory deviation based on the DTW distance between the standard operation trajectory and the trainee's actual operation trajectory, obtain the tactile force deviation based on the root mean square deviation between the standard force curve and the trainee's actual force curve, and obtain the operation hesitation duration based on the pause time before operation, which together constitute the operation deviation. Step S3: Calculate the process state residual based on the predicted state vector output by the process simulation engine under standard operation and the state vector triggered by the trainee's actual operation, and jointly attribute the operation deviation, cognitive load value and process state residual to the weak operation vector. Step S4: Map the weak operation vector to counterfactual intervention variables, perform counterfactual reasoning on the equipment causal fault graph, and pre-establish the safety envelope of each process parameter, that is, the value range of each process parameter within the normal operation and allowable deviation range; based on the safety envelope, generate candidate fault chains that satisfy the causal strength threshold, fault chain length threshold, safety envelope constraint and coverage similarity constraint. Step S5: Generate fault injection parameters based on the candidate fault chain, perform collaborative simulation through the physics engine, process simulation engine and haptic feedback engine, and generate a consistency certificate for the training segment based on dual clock synchronization, state prediction, event latching and residual backfeedback. Step S6: When the conformity certificate meets the preset threshold, calculate the attribution operation score based on action deviation, tactile force deviation, process state residual and cognitive load fluctuation, and update the counterfactual training scenario according to the fault chain coverage, weak operation coverage and safety envelope coverage.

[0007] To better realize the present invention, the eye movement data in step S1 further includes fixation point coordinates, pupil diameter, and blink frequency; the gesture operation data includes operation trajectory and operation hesitation duration; the physiological signal data includes heart rate variability; and the tactile operation data includes operation force, force change rate, and vibration response time.

[0008] To better realize the present invention, step S2 further includes the following steps: Step S21: Calculate the pupil dilation rate based on the current pupil diameter and baseline diameter; obtain the attentional distraction based on the standardized normalized value of the fixation point coordinates; and obtain the eye movement load based on the pupil dilation rate and attentional distraction. Step S22: Calculate the operational load based on DTW similarity and operational hesitation duration; Step S23: Calculate the HRV reduction rate based on the current heart rate variability and baseline heart rate variability to obtain the physiological load; Step S24: Obtain the cognitive load value based on the set load weights, eye movement load, operational load, and physiological load.

[0009] To better realize the present invention, step S3 further includes the following steps: Step S31: Compare the predicted state vector obtained from the process simulation engine with the simulation state vector triggered by the student's operation to obtain the process state residual; wherein the state vector includes temperature, pressure, flow rate, valve position, equipment vibration and safety interlock status; Step S32: Construct a weak operation vector based on the operation trajectory deviation, tactile force deviation, operation hesitation time, and process state residual.

[0010] To better realize the present invention, step S4 further includes the following steps: Step S41: Based on the valve opening, button triggering time, process parameter setting, operation sequence and interlock confirmation action, the trainable intervention value is obtained under the safety envelope constraint, and the weak operation vector is mapped to the counterfactual intervention variable do(X=x'). Step S42: Based on the counterfactual intervention variables, perform counterfactual reasoning on the constructed device causal failure graph to obtain candidate failure chains.

[0011] To better realize the present invention, step S5 further includes the following steps: Step S51: Generate fault injection time, duration, severity, observable signs and tactile signs based on candidate fault chains, and adjust information density according to cognitive load; Step S52: Perform physical clock synchronization and process clock synchronization according to the set fixed step size and the set adaptive step size; Step S53: Based on the set key events, feed back the process state residuals before and after the events into the physics engine and haptic feedback engine; Step S54: Generate a consistency certificate for the training segment. The consistency certificate includes timestamp error, state residual norm, event latch identifier, key event trigger record, haptic mapping version, and process safety envelope verification result. If any indicator in the consistency certificate exceeds the threshold, the scoring of the scene is paused and resynchronized.

[0012] To better realize the present invention, step S6 further includes the following steps: Step S61: When the conformity certificate meets the preset threshold, calculate the attribution operation score based on action deviation, tactile force deviation, process state residual and cognitive load fluctuation; Step S62: Update the counterfactual training scenario based on the fault chain coverage, weak operation coverage, and security envelope coverage. When the fault chain coverage, weak operation coverage, or security envelope coverage is lower than the preset coverage threshold, prioritize generating counterfactual fault scenarios for uncovered areas. When the coverage similarity is higher than the preset similarity threshold, suppress the generation of duplicate scenarios.

[0013] Based on the aforementioned virtual operation training method based on the industrial metaverse, and to better realize this invention, an immersive operation training system in an industrial metaverse environment is further proposed to execute the aforementioned virtual operation training method based on the industrial metaverse; it includes a multimodal acquisition unit, a real-time cognitive load assessment unit, a residual attribution unit, a counterfactual scenario generation unit, a digital twin collaborative simulation unit, and a coverage matrix training unit. The multimodal acquisition unit is used to collect eye-tracking data, gesture operation data, physiological signal data and tactile operation data of trainees in the industrial metaverse training scenario, and simultaneously collect the equipment motion state of the physics engine and the process state of the process simulation engine. The real-time cognitive load assessment unit is used to calculate the cognitive load value of the trainee based on the eye movement data, gesture operation data and physiological signal data, and to calculate the operation deviation based on the standard operation trajectory, standard force curve and the trainee's actual operation data. The residual attribution unit is used to calculate the process state residual based on the predicted state vector output by the process simulation engine under standard operation and the state vector triggered by the actual operation of the trainee, and to jointly attribute the operation deviation, cognitive load value and process state residual to the weak operation vector. The counterfactual scenario generation unit is used to map weak operation vectors into counterfactual intervention variables, perform counterfactual reasoning on the equipment causal fault graph, and pre-establish the safety envelope of each process parameter, that is, the value range of each process parameter within the normal operation and allowable deviation range; based on the safety envelope, generate candidate fault chains that satisfy the causal strength threshold, fault chain length threshold, safety envelope constraint and coverage similarity constraint. The digital twin co-simulation unit is used to generate fault injection parameters based on the candidate fault chain, perform co-simulation through the physics engine, process simulation engine and haptic feedback engine, and generate a consistency certificate for the training segment based on dual clock synchronization, state prediction, event latching and residual backfeedback. The coverage matrix training unit is used to calculate the attribution operation score based on action deviation, tactile force deviation, process state residual and cognitive load fluctuation when the consistency certificate meets the preset threshold, and update the counterfactual training scenario according to the fault chain coverage, weak operation coverage and safety envelope coverage.

[0014] Based on the aforementioned virtual operation training method based on an industrial metaverse, and to better realize this invention, an electronic device is further proposed, comprising a memory and a processor; the memory stores a computer program; when the computer program is executed on the processor, the aforementioned virtual operation training method based on an industrial metaverse is implemented.

[0015] Based on the aforementioned virtual operation training method based on an industrial metaverse, and to better realize this invention, a computer-readable storage medium is further proposed, wherein computer instructions are stored on the computer-readable storage medium; when the computer instructions are executed on the aforementioned electronic device, the aforementioned virtual operation training method based on an industrial metaverse is realized.

[0016] The present invention has the following beneficial effects: (1) This invention no longer relies solely on the difficulty of adjusting cognitive load, but instead maps trainee action deviation, cognitive load and digital twin process state residuals as counterfactual intervention variables, generating a fault chain on the equipment causal fault diagram that is strongly correlated with the current weak operation and conforms to process constraints, thereby realizing the generation of counterfactual fault scenarios driven by process state residuals.

[0017] (2) The present invention generates a state consistency certificate for each simulation segment through dual clock synchronization, state prediction, event latching and residual backfeeding, ensuring that visual motion, process parameters and tactile output come from the same process state trajectory.

[0018] (3) The present invention decomposes the operation results into four categories of attributable indicators: action trajectory deviation, tactile force deviation, process state residual and cognitive load fluctuation, thus avoiding the evaluation of trainees based solely on the final score or the similarity of the expert trajectory.

[0019] (4) The present invention updates the subsequent training objectives based on the fault chain coverage, weak operation coverage and safety envelope coverage, forming a closed loop of state acquisition, residual attribution, counterfactual generation, consistency simulation and coverage update. Attached Figure Description

[0020] Figure 1 A schematic diagram of the industrial metaverse training system architecture based on the counterfactual generation of process state residuals provided by the present invention.

[0021] Figure 2 The flowchart of the immersion operation training method provided by the present invention. Detailed Implementation

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments, and therefore should not be regarded as a limitation on the scope of protection. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "set up," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0024] Example 1: This embodiment proposes a virtual operation training method based on an industrial metaverse, which specifically includes the following steps: Step S1: Collect eye-tracking data, gesture operation data, physiological signal data, and tactile operation data of trainees in the Industrial Metaverse training scenario, and simultaneously collect the equipment motion status of the physics engine and the process status of the process simulation engine. Step S2: Calculate the cognitive load value of the trainee based on the eye movement data, gesture operation data and physiological signal data, and calculate the operation deviation based on the standard operation trajectory, standard force application curve and the trainee's actual operation data; The eye-tracking data in step S1 includes fixation point coordinates, pupil diameter, and blink frequency; the gesture operation data includes operation trajectory and operation hesitation duration; the physiological signal data includes heart rate variability; and the tactile operation data includes operation force, force change rate, and vibration response time.

[0025] Step S2 specifically includes the following steps: Step S21: Calculate the pupil dilation rate based on the current pupil diameter and baseline diameter; obtain the attentional distraction based on the standardized normalized value of the fixation point coordinates; and obtain the eye movement load based on the pupil dilation rate and attentional distraction. Step S22: Calculate the operational load based on DTW similarity and operational hesitation duration; Step S23: Calculate the HRV reduction rate based on the current heart rate variability and baseline heart rate variability to obtain the physiological load; Step S24: Obtain the cognitive load value based on the set load weights, eye movement load, operational load, and physiological load.

[0026] Step S3: Calculate the process state residual based on the predicted state vector output by the process simulation engine under standard operation and the state vector triggered by the trainee's actual operation, and jointly attribute the operation deviation, cognitive load value and process state residual to the weak operation vector. Step S3 specifically includes the following steps: Step S31: Compare the predicted state vector obtained from the process simulation engine with the simulation state vector triggered by the student's operation to obtain the process state residual; wherein the state vector includes temperature, pressure, flow rate, valve position, equipment vibration and safety interlock status; Step S32: Construct a weak operation vector based on the operation trajectory deviation, tactile force deviation, operation hesitation time, and process state residual.

[0027] Step S4: Map the weak operation vector to counterfactual intervention variables, perform counterfactual reasoning on the equipment causal fault graph, and pre-establish the safety envelope of each process parameter, that is, the value range of each process parameter within the normal operation and allowable deviation range; based on the safety envelope, generate candidate fault chains that satisfy the causal strength threshold, fault chain length threshold, safety envelope constraint and coverage similarity constraint. Step S4 specifically includes the following steps: Step S41: Based on the valve opening, button trigger time, process parameter setting, operation sequence and interlock confirmation action, and trainable intervention value after safety envelope constraint correction, map the weak operation vector into counterfactual intervention variables; Step S42: Based on the counterfactual intervention variables, perform counterfactual reasoning on the constructed device causal failure graph to obtain candidate failure chains.

[0028] Step S5: Generate fault injection parameters based on the candidate fault chain, perform collaborative simulation through the physics engine, process simulation engine and haptic feedback engine, and generate a consistency certificate for the training segment based on dual clock synchronization, state prediction, event latching and residual backfeedback. Step S5 specifically includes the following steps: Step S51: Generate fault injection time, duration, severity, observable signs and tactile signs based on candidate fault chains, and adjust information density according to cognitive load; Step S52: Perform physical clock synchronization and process clock synchronization according to the set fixed step size and the set adaptive step size; Step S53: Based on the set key events, feed back the process state residuals before and after the events into the physics engine and haptic feedback engine; Step S54: Generate a consistency certificate for the training segment. The consistency certificate includes timestamp error, state residual norm, event latch identifier, key event trigger record, haptic mapping version, and process safety envelope verification result. If any indicator in the consistency certificate exceeds the threshold, the scoring of the scene is paused and resynchronized.

[0029] Step S6: When the conformity certificate meets the preset threshold, calculate the attribution operation score based on action deviation, tactile force deviation, process state residual and cognitive load fluctuation, and update the counterfactual training scenario according to the fault chain coverage, weak operation coverage and safety envelope coverage.

[0030] Step S6 specifically includes the following steps: Step S61: When the conformity certificate meets the preset threshold, calculate the attribution operation score based on action deviation, tactile force deviation, process state residual and cognitive load fluctuation; Step S62: Update the counterfactual training scenario based on the fault chain coverage, weak operation coverage, and security envelope coverage. When the fault chain coverage, weak operation coverage, or security envelope coverage is lower than the preset coverage threshold, prioritize generating counterfactual fault scenarios for uncovered areas. When the coverage similarity is higher than the preset similarity threshold (e.g., 0.75), suppress the generation of duplicate scenarios.

[0031] Working Principle: This embodiment no longer relies solely on the difficulty of cognitive load adjustment, but instead maps trainee action deviation, cognitive load, and digital twin process state residuals as counterfactual intervention variables. It generates a fault chain on the equipment causal fault diagram that is strongly correlated with the current weak operation and conforms to process constraints. Through dual-clock synchronization, state prediction, event latching, and residual backfeeding, a state consistency certificate is generated for each simulation segment, ensuring that visual motion, process parameters, and tactile outputs originate from the same process state trajectory. The operation results are decomposed into four attributable indicators: action trajectory deviation, tactile force deviation, process state residuals, and cognitive load fluctuations, avoiding evaluation of trainees solely based on final scores or expert trajectory similarity. Subsequent training objectives are updated based on fault chain coverage, weak operation coverage, and safety envelope coverage, forming a closed loop of state acquisition, residual attribution, counterfactual generation, consistency simulation, and coverage update.

[0032] Example 2: This embodiment is based on the above embodiment 1, such as... Figure 1 , Figure 2 As shown, a specific embodiment will be described in detail.

[0033] Step S1: Multimodal data acquisition; Real-time collection of trainees' multimodal physiological and behavioral data: Eye-tracking data: fixation point coordinates, pupil diameter (sampling frequency 120Hz), blink frequency; Gesture operation data: operation trajectory (6DoF pose), operation hesitation duration; Physiological signal: Heart rate variability (HRV, sampling frequency 256Hz); Tactile manipulation data: manipulation force (0-15N), force change rate, vibration response time; Step S2: Real-time assessment of cognitive load; Constructing a cognitive load assessment model based on multimodal data: 1. Eye-tracking feature extraction: Pupil dilation rate = (Current pupil diameter - Baseline diameter) / Baseline diameter; Fixation point dispersion = Normalized standard deviation of fixation point coordinates (0-1); High workload is defined as: pupillary dilation > 20% or fixation point dispersion > 0.35; 2. Gesture operation feature extraction: DTW similarity: Dynamic time warping similarity to the standard operating trajectory; Operational hesitation duration: the pause time before operation (>2s indicates hesitation, >5s indicates confusion); 3. Physiological signal feature extraction: HRV reduction rate = (baseline HRV - current HRV) / baseline HRV High workload assessment: HRV reduction rate > 15% The methods for quantifying load in each dimension are as follows: (1) Eye movement load: When the pupillary dilation rate is ≤20% and the fixation point dispersion is ≤0.35, the eye movement load = 0.5×(pupil dilation rate / 20%) + 0.5×(fixation point dispersion / 0.35); when the pupillary dilation rate is >20% or the fixation point dispersion is >0.35, the eye movement load = 1.0; (2) Operational load: Operational load = 0.6 × (1 - DTW similarity) + 0.4 × min (operational hesitation time / 5, 1.0); (3) Physiological load: When the HRV reduction rate is ≤15%, the physiological load = HRV reduction rate / 15%; when the HRV reduction rate is >15%, the physiological load =1.0.

[0034] 4. Cognitive load fusion calculation: CL = w1 × eye movement load + w2 × operational load + w3 × physiological load; Optimal load range: 0.4-0.7; High load threshold: 0.8; Low load threshold: 0.3; Step S3: Attribution of process state residuals and generation of counterfactual failure scenarios; The equipment causal failure graph G is constructed during the initialization phase of the training system. Based on equipment failure mode and effects analysis (FMEA) data and historical failure case library, a Bayesian network or causal graph model is used to establish the causal dependencies between equipment components. In the graph, nodes represent the state of equipment components or process parameters, directed edges represent the direction of failure propagation, and edge weights represent the causal strength.

[0035] Based on the cognitive load assessment results, trainee operational deviations, and digital twin process state residuals, training scenarios are generated: 1. Calculation of process state residuals: The predicted state vector P(t) output by the process simulation engine is compared with the simulation state vector Q(t) triggered by the student operation to obtain the process state residual R(t) = Q(t) - P(t), where the state vector includes temperature, pressure, flow rate, valve position, equipment vibration and safety interlock status.

[0036] 2. Weak Operations - Residual Attribution: Based on operation trajectory deviation, tactile force application deviation, operation hesitation duration, and the dominant residual term of R(t), a weak operation vector W = {operation object, operation timing deviation, force application deviation, residual type, cognitive load level} is formed.

[0037] 3. Constructing counterfactual intervention variables: Map W to counterfactual intervention variables do(X=x'), where X includes valve opening degree, button trigger time, process parameter set value, operation sequence and interlock confirmation action; x' is a trainable intervention value corrected according to safety envelope constraints.

[0038] 4. Causal Fault Graph Reasoning: Perform counterfactual reasoning on the pre-constructed equipment causal fault graph G, and output candidate fault chains that meet the following conditions: the causal correlation strength with W is greater than 0.6; the fault chain length does not exceed 5 steps; each node in the chain satisfies the process safety envelope; and the coverage similarity with the trained fault chains is less than 0.75.

[0039] 5. Scene Parameter Generation: Based on the candidate fault chain, generate fault injection time, duration, severity, observable signs, and tactile signs, and define the information density of the training segment as I=(N_obs+N_tact+N_disturb) / N_base, where N_obs is the number of observable signs, N_tact is the number of tactile signs, N_disturb is the number of concurrent perturbations, and N_base is the number of baseline information units for the normal scene; adjust the information density according to the cognitive load CL: when CL>0.8, reduce the number of observable signs by I_new=max(I_min,I×0.6) without changing the nature of the fault; when CL<0.3, increase the number of concurrent minor perturbations by I_new=min(I_max,I×1.2).

[0040] Step S4: Physical-process-haptic co-simulation with conformity certificate; The complete simulation process from fault injection to fault handling corresponding to the counterfactual fault scenario generated in step S3 is defined as a training segment; a dual-clock synchronization mechanism is used to ensure data consistency among the physics engine, process simulation engine, and haptic feedback engine in the training segment. 1. Dual clock synchronization: Physical clock: fixed step size Δt_p = 10ms (Unity Physics); Process clock: Adaptive step size Δt_w = 100ms (COMSOL); Timestamp alignment error < 5ms; 2. State prediction and interpolation compensation: When Δt_w > Δt_p: Use a Kalman filter to predict the intermediate state (prediction error < 3%). When Δt_p > Δt_w: use cubic spline interpolation for compensation; 3. Event latching and residual backfeeding: Define key events: valve opening / closing, temperature change (>10°C / s), pressure change (>0.5 bar / s), interlock triggering, and tactile damping change; Critical events trigger forced synchronization, and the process state residual R(t) before and after the event is fed back into the physics engine and haptic feedback engine; the fed-back residual data serves as the basis for calculating the "state residual norm" index in the consistency certificate of this training segment; 4. Generate conformance certificate: For each training segment, generate a consistency certificate C = {timestamp error, state residual norm, event latch identifier, critical event trigger record, haptic mapping version, and process safety envelope verification result}; where the critical event trigger record includes the event occurrence time, event type, and the number of forced synchronizations triggered by the event, and the state residual norm is the average of the L2 norms of all backfeed residuals R(t) within the segment.

[0041] When any indicator in C exceeds the threshold, the system pauses the scoring of that scenario and resynchronizes to avoid evaluating trainees based on distorted simulation results. Step S5: Residual attribution multimodal operation evaluation; Integrating visual, tactile, and cognitive three-dimensional data to assess operational quality: 1. Haptic feedback generation: Force feedback gloves (HaptX) simulate operational forces (0-15N, resolution 0.1N); Vibration feedback simulates the vibration characteristics of the equipment (10-500Hz, response delay <20ms). Texture feedback simulates the surface characteristics of the device (roughness level 1-10); 2. Residual Attribution Assessment: Action dimensions: DTW similarity of operation trajectory, deviation of operation sequence, and deviation of operation timing; Tactile dimensions: deviation of operational force, rate of change of force, and response time of tactile signs; Process dimensions: peak value, duration, recovery time, and dominant residual term of R(t); Cognitive dimensions: spatiotemporal consistency between eye-tracking fixation points and operant points, peak eye movement frequency (CL), and CL recovery time; Action dimension score = 0.4·S_DTW + 0.3·(1 - |ΔO| / ΔO_max) + 0.3·(1- |ΔT| / ΔT_max), where S_DTW is the DTW similarity, ΔO is the operation order deviation, ΔO_max is the maximum allowable value of the operation order deviation, ΔT is the operation time deviation, and ΔT_max is the maximum allowable value of the operation time deviation; Tactile dimension score = 0.5·(1 - |ΔF| / ΔF_max) + 0.3·(1 -|ΔR| / ΔR_max) +0.2·(1 - |Δt_resp| / Δt_resp_max); The process dimension score = 1 - min(1, R_peak / E_i). Where R_peak = |R(t)|_max is the maximum absolute value of the process state residual within the training segment; E_i is the safe envelope half-width of the i-th process parameter; Cognitive dimension score = 0.5·C_spatial + 0.3·(1 -CL_peak / CL_max) + 0.2·(1-T_rec / T_rec_max); The overall score is calculated as follows: [Σ_{j=1}^{4} (w_j × S_j)]^λ × R_risk. Here, w_j is the weight of the j-th dimension (Σw_j = 1.0), S_j is the score of the j-th dimension, λ is the risk adjustment coefficient, and R_risk is the state residual risk term. R_risk = Σ_{i=1}^{n} (R_i_max / E_i × w_i^s). Where n is the total number of process parameters, R_i_max = |R_i(t)|_max is the maximum absolute value of the residual of the i-th process parameter within the training segment, E_i is the half-width of the safety envelope of the i-th process parameter, and w_i^s is the safety weight of the i-th process parameter, set according to the risk level of the parameter in the process safety envelope (e.g., overheat w_i^s=0.3, overpressure w_i^s=0.25, abnormal flow w_i^s=0.15, valve position deviation w_i^s=0.15, excessive vibration w_i^s=0.1, interlock failure w_i^s=0.05), and Σw_i^s = 1.0, Process parameters include temperature, pressure, flow rate, valve position, equipment vibration, and safety interlock status. The safety weight of each parameter is set according to the hazard level of that parameter in the process safety envelope (e.g., over-temperature is 0.3, over-pressure is 0.25, abnormal flow rate is 0.15, valve position deviation is 0.15, excessive vibration is 0.1, and interlock failure is 0.05), and the sum of the weights is 1.0; λ is the risk adjustment coefficient, which is dynamically determined according to the degree of deviation of the current process state from the center of the safety envelope; the greater the deviation, the larger the value of λ, λ∈[0.5, 2.0]; when the process state is at the center of the safety envelope, λ=0.5, and when the process state is close to the boundary of the safety envelope, λ=2.0; 3. The evaluation level is determined based on the above comprehensive score (out of 100): Excellent (≥90), Good (80-89), Pass (70-79), Fail (<70); Step S6: Define the covering matrix M = (M_fault, M_weak, M_envelope), where: M_fault is the fault chain coverage matrix, which records the coverage of each node in the device's causal fault graph by the generated counterfactual fault chains; M_weak is the weak operation coverage matrix, which records the dimensions of the weak operations that have been covered; M_envelope is the secure envelope coverage matrix, which records the coverage of the secure envelope boundary sampling points used for training.

[0042] The counterfactual training scenarios are updated based on fault chain coverage, weak operation coverage, and security envelope coverage. When the fault chain coverage, weak operation coverage, or security envelope coverage is lower than the preset coverage threshold, counterfactual fault scenarios in uncovered areas are generated first. When the coverage similarity is higher than the preset similarity threshold (e.g., 0.75), the generation of duplicate scenarios is suppressed.

[0043] The key technical parameters and descriptions involved in this embodiment are shown in Table 1 below.

[0044] Table 1. Correspondence Table of Key Technical Parameters

[0045] The other parts of this embodiment are the same as those in Embodiment 1 above, so they will not be described again.

[0046] Example 3: This embodiment is based on any one of the above embodiments 1-2, and takes semiconductor CVD equipment operation training as an example for explanation.

[0047] Application scenario: Chemical vapor deposition (CVD) equipment operator training, with training objectives including start-up checks, equipment startup, process parameter settings, troubleshooting, and equipment shutdown.

[0048] Training process: Phase 1: Initial Assessment (0-5 minutes) Trainees put on VR headsets, haptic gloves, and eye-tracking devices to enter the virtual CVD operating room. The system collected baseline data: pupil diameter 3.2 mm, fixation divergence 0.15, HRV 60 ms, and initial cognitive load 0.35. Phase 2: Basic Operation Training (5-20 minutes) Trainees perform the "start vacuum pump" operation System assessment: DTW similarity 0.72 (below standard 0.85), operation hesitation time 4.2s, pressing force 3.8N (standard 2.5N). Cognitive load assessment: pupil dilation 28%, fixation dispersion 0.42, HRV decreased by 15%, current cognitive load 0.82 (high load). Phase 3: Fault Scenario Generation (Automatic, <1 second) Identifying weaknesses: Vacuum pump startup operation Causal reasoning: Vacuum pump startup failure → Abnormal chamber pressure → Process cannot start (causality strength 0.76) Dynamically generated scenario: Vacuum pump start-up delay fault, difficulty level 0.6 (automatically reduced based on cognitive load of 0.82). Phase 4: Fault Scenario Training (20-35 minutes) The physics engine (10ms step) simulates the slow decrease of a pressure gauge pointer. The process simulation engine (100ms step size) calculates the chamber pressure change (1000mbar→800mbar, normally it should drop to 100mbar). The Kalman filter predicts intermediate states, ensuring data consistency between the two engines. The tactile gloves simulate abnormal vibrations (150Hz) and valve resistance (6.5N). Trainees identify and resolve faults within 18 seconds (standard <30 seconds, good). Phase 5: Iterative Training (35-60 minutes) The cognitive load was reduced to 0.58 (optimal range), and the system gradually increased the difficulty. Scenario 2: Abnormal gas flow (Difficulty 0.7) → Scenario 3: Temperature control failure (Difficulty 0.8) → Scenario 4: Multiple faults (Difficulty 0.9) The training results are shown in Table 2 below.

[0049] Table 2 Correspondence Table of Semiconductor CVD Equipment Operation Training Effectiveness

[0050] The other parts of this embodiment are the same as any one of the above embodiments 1-2, so they will not be described again.

[0051] Example 4: This embodiment is based on any one of the above embodiments 1-3, and takes emergency shutdown operation training of chemical plant as an example for explanation.

[0052] Application scenario: Emergency shutdown (ESD) operation training for petrochemical distillation towers. Key training process: System injection failure: The temperature at the bottom of the tower rose from 180°C to 220°C within 10 seconds, and the pressure rose from 2.5 bar to 3.2 bar; Physics-process co-simulation: The physics engine simulates the flashing of the alarm light (2Hz) and the rise of the temperature gauge pointer; the process simulation engine calculates the material temperature distribution and vaporization rate; critical event triggering (forced synchronization every 5°C increase in temperature); Haptic feedback: The ESD button cover requires 8N of force to open, and the button requires 5N of force to press down; Multimodal assessment: visual 88%, tactile 82%, cognitive 90%, overall score 87% (good).

[0053] The training results are shown in Table 3 below.

[0054] Table 3. Training Effectiveness of Emergency Shutdown Operations in Chemical Plants

[0055] The other parts of this embodiment are the same as any one of the embodiments 1-3 above, so they will not be described again.

[0056] Example 5: This embodiment is based on any one of the embodiments 1-4 above, and takes multi-person collaborative operation training as an example for detailed explanation.

[0057] Application scenario: Maintenance of large equipment (3 people working together: main operator, assistant operator, and safety supervisor); The system independently assesses cognitive load for each learner: Trainee 1 (Main Operator): Cognitive Load 0.72 (High, requires decision-making); Trainee 2 (Assistant Operator): Cognitive load 0.48 (normal, executing instructions); Trainee 3 (Safety Supervisor): Cognitive load 0.55 (normal, under observation and monitoring); The system reduces fault complexity and generates collaborative fault scenarios based on the high cognitive load of the main operator.

[0058] Multimodal evaluation: Collaboration efficiency: 85% (timely communication, good cooperation); Individual operational accuracy: main operator 88%, assistant operator 92%, safety supervisor 90%.

[0059] The other parts of this embodiment are the same as any one of the embodiments 1-4 above, so they will not be described again.

[0060] Example 6: Based on any one of Embodiments 1-5 above, this embodiment proposes an immersive operation training system in an industrial metaverse environment, based on the aforementioned virtual operation training method based on an industrial metaverse, to execute the aforementioned virtual operation training method based on an industrial metaverse; it includes a multimodal acquisition unit, a real-time cognitive load assessment unit, a residual attribution unit, a counterfactual scenario generation unit, a digital twin collaborative simulation unit, and a coverage matrix training unit. The multimodal acquisition unit is used to collect eye-tracking data, gesture operation data, physiological signal data and tactile operation data of trainees in the industrial metaverse training scenario, and simultaneously collect the equipment motion state of the physics engine and the process state of the process simulation engine. The real-time cognitive load assessment unit is used to calculate the cognitive load value of the trainee based on the eye movement data, gesture operation data and physiological signal data, and to calculate the operation deviation based on the standard operation trajectory, standard force curve and the trainee's actual operation data. The residual attribution unit is used to calculate the process state residual based on the predicted state vector output by the process simulation engine under standard operation and the state vector triggered by the actual operation of the trainee, and to jointly attribute the operation deviation, cognitive load value and process state residual to the weak operation vector. The counterfactual scenario generation unit is used to map weak operation vectors into counterfactual intervention variables, perform counterfactual reasoning on the equipment causal fault graph, and pre-establish the safety envelope of each process parameter, that is, the value range of each process parameter within the normal operation and allowable deviation range; based on the safety envelope, generate candidate fault chains that satisfy the causal strength threshold, fault chain length threshold, safety envelope constraint and coverage similarity constraint. The digital twin co-simulation unit is used to generate fault injection parameters based on the candidate fault chain, perform co-simulation through the physics engine, process simulation engine and haptic feedback engine, and generate a consistency certificate for the training segment based on dual clock synchronization, state prediction, event latching and residual backfeedback. The coverage matrix training unit is used to calculate the attribution operation score based on action deviation, tactile force deviation, process state residual and cognitive load fluctuation when the consistency certificate meets the preset threshold, and update the counterfactual training scenario according to the fault chain coverage, weak operation coverage and safety envelope coverage.

[0061] This embodiment also proposes an electronic device, including a memory and a processor; the memory stores a computer program; when the computer program is executed on the processor, it implements the above-described virtual operation training method based on an industrial metaverse.

[0062] This embodiment also proposes a computer-readable storage medium storing computer instructions; when the computer instructions are executed on the aforementioned electronic device, the aforementioned virtual operation training method based on an industrial metaverse is implemented.

[0063] The training efficiency of this embodiment is significantly improved, and the training effect is shown in Table 4 below; Table 4 Comparison of Training Effectiveness

[0064] All training in this embodiment is conducted in a virtual environment, completely eliminating the risk of damage to real equipment, and expanding from the traditional 5-10 types to 50+ types (dynamically generated), covering low-probability, high-risk scenarios.

[0065] This embodiment integrates visual, tactile, and cognitive three-dimensional data, increasing the evaluation dimension from 1 dimension to 3 dimensions; a quantitative score and improvement suggestions are given immediately after the operation is completed.

[0066] This embodiment automatically maintains the learner's cognitive load within the optimal range (0.4-0.7) for maximum learning efficiency; it dynamically generates targeted training scenarios based on each learner's weaknesses to achieve personalized training paths.

[0067] The other parts of this embodiment are the same as any one of the above embodiments 1-5, so they will not be described again.

[0068] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A virtual operation training method based on an industrial metaverse, characterized in that, Specifically, the following steps are included: Step S1: Collect eye-tracking data, gesture operation data, physiological signal data and tactile operation data of trainees in the Industrial Metaverse training scenario, and simultaneously collect the equipment motion status of the physics engine and the process status of the process simulation engine. Step S2: Calculate the cognitive load value of the trainee based on the eye movement data, gesture operation data, and physiological signal data; Step S3: Calculate the process state residual based on the predicted state vector output by the process simulation engine under standard operation and the state vector triggered by the trainee's actual operation, and construct the weak operation vector based on the cognitive load value and the process state residual. Step S4: Map the weak operation vectors to counterfactual intervention variables, perform counterfactual reasoning on the constructed device causal fault graph based on the constructed security envelope, and generate candidate fault chains; Step S5: Generate fault injection parameters based on the candidate fault chain, and perform collaborative simulation through the physics engine, process simulation engine and haptic feedback engine to generate a conformance certificate; Step S6: If the conformity certificate meets the set threshold, calculate the attribution operation score based on action deviation, tactile force deviation, process state residual and cognitive load fluctuation, and update the counterfactual training scenario based on fault chain coverage, weak operation coverage and safety envelope coverage.

2. The virtual operation training method based on the industrial metaverse according to claim 1, characterized in that, The eye-tracking data in step S1 includes fixation point coordinates, pupil diameter, and blink frequency; the gesture operation data includes operation trajectory and operation hesitation duration; the physiological signal data includes heart rate variability; and the tactile operation data includes operation force, force change rate, and vibration response time.

3. The virtual operation training method based on the industrial metaverse according to claim 2, characterized in that, Step S2 specifically includes the following steps: Step S21: Calculate the pupil dilation rate based on the current pupil diameter and baseline diameter; obtain the attentional distraction based on the standardized normalized value of the fixation point coordinates; and obtain the eye movement load based on the pupil dilation rate and attentional distraction. Step S22: Calculate the operational load based on DTW similarity and operational hesitation duration; Step S23: Calculate the HRV reduction rate based on the current heart rate variability and baseline heart rate variability to obtain the physiological load; Step S24: Obtain the cognitive load value based on the set load weights, eye movement load, operational load, and physiological load.

4. The virtual operation training method based on the industrial metaverse according to claim 2, characterized in that, Step S3 specifically includes the following steps: Step S31: Compare the predicted state vector obtained from the process simulation engine with the simulation state vector triggered by the student's operation to obtain the process state residual; wherein the state vector includes temperature, pressure, flow rate, valve position, equipment vibration and safety interlock status; Step S32: Construct a weak operation vector based on the operation object, operation trajectory deviation, operation time deviation, tactile force deviation, operation hesitation duration, process state residual type, and cognitive load level.

5. The virtual operation training method based on the industrial metaverse according to claim 4, characterized in that, Step S4 specifically includes the following steps: Step S41: Based on the valve opening, button trigger time, process parameter setting, operation sequence and interlock confirmation action, and trainable intervention value after safety envelope constraint correction, map the weak operation vector into counterfactual intervention variables; Step S42: Perform counterfactual reasoning on the constructed device causal fault graph based on the counterfactual intervention variables, and generate candidate fault chains based on the set security envelope.

6. The virtual operation training method based on the industrial metaverse according to claim 1, characterized in that, Step S5 specifically includes the following steps: Step S51: Generate fault injection time, duration, severity, observable signs and tactile signs based on candidate fault chains, and adjust information density according to cognitive load; Step S52: Perform physical clock synchronization and process clock synchronization according to the set fixed step size and the set adaptive step size; Step S53: Based on the set key events, feed back the process state residuals before and after the events into the physics engine and the haptic feedback engine to obtain the feedback residual data; Step S54: Generate a consistency certificate based on the residual data after backfeeding. If any indicator in the consistency certificate exceeds the set threshold, pause the current scene scoring and resynchronize. The consistency certificate includes timestamp error, state residual norm, event latch identifier, key event trigger record, haptic mapping version, and process safety envelope verification result.

7. The virtual operation training method based on the industrial metaverse according to claim 6, characterized in that, Step S6 specifically includes the following steps: Step S61: When the conformity certificate meets the set threshold, calculate the attribution operation score based on the action deviation, tactile force deviation, process state residual and cognitive load fluctuation. Step S62: Update the counterfactual training scenario based on the fault chain coverage, weak operation coverage, and security envelope coverage. When the fault chain coverage, weak operation coverage, or security envelope coverage is lower than the set coverage threshold, generate counterfactual fault scenarios for uncovered areas. When the coverage similarity is higher than the set similarity threshold, suppress the generation of duplicate scenarios.

8. An immersive operational training system in an industrial metaverse environment, used to execute the virtual operational training method based on an industrial metaverse as described in claim 1; characterized in that, It includes a multimodal acquisition unit, a real-time cognitive load assessment unit, a residual attribution unit, a counterfactual scenario generation unit, a digital twin co-simulation unit, and a coverage matrix training unit; The multimodal acquisition unit is used to collect eye-tracking data, gesture operation data, physiological signal data and tactile operation data of trainees in the industrial metaverse training scenario, and simultaneously collect the equipment motion state of the physics engine and the process state of the process simulation engine. The real-time cognitive load assessment unit is used to calculate the cognitive load value of the trainee based on the eye movement data, gesture operation data and physiological signal data, and to calculate the operation deviation based on the standard operation trajectory, standard force curve and the trainee's actual operation data. The residual attribution unit is used to calculate the process state residual based on the predicted state vector output by the process simulation engine under standard operation and the state vector triggered by the actual operation of the trainee, and to jointly attribute the operation deviation, cognitive load value and process state residual to the weak operation vector. The counterfactual scenario generation unit is used to map the weak operation vector into counterfactual intervention variables, perform counterfactual reasoning on the equipment causal fault diagram, and pre-establish the safety envelope of each process parameter, that is, the value range of each process parameter within the normal operation and allowable deviation range. Based on the security envelope, candidate fault chains that satisfy the causal strength threshold, fault chain length threshold, security envelope constraint, and coverage similarity constraint are generated. The digital twin co-simulation unit is used to generate fault injection parameters based on the candidate fault chain, perform co-simulation through the physics engine, process simulation engine and haptic feedback engine, and generate a consistency certificate for the training segment based on dual clock synchronization, state prediction, event latching and residual backfeedback. The coverage matrix training unit is used to calculate the attribution operation score based on action deviation, tactile force deviation, process state residual and cognitive load fluctuation when the consistency certificate meets the preset threshold, and update the counterfactual training scenario according to the fault chain coverage, weak operation coverage and safety envelope coverage.

9. An electronic device, characterized in that, It includes a memory and a processor; the memory stores a computer program; when the computer program is executed on the processor, it implements a virtual operation training method based on an industrial metaverse as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions; when the computer instructions are executed on the electronic device as described in claim 9, they implement a virtual operation training method based on an industrial metaverse as described in any one of claims 1-7.

Images