A VR-based multi-person virtual live inspection training method, device and medium

Abstract

This invention discloses a VR-based multi-person virtual on-site inspection training method, equipment, and medium, relating to the field of virtual reality simulation training technology. The method is executed collaboratively by a server, multiple student VR terminals, and an IoT gateway, including: synchronizing the real device status to the VR scene and connecting multiple students; collecting and time-aligning multi-student operation data with device status data; constructing a behavior chain containing dual channels of multi-student action events and aligned device status; comparing this behavior chain with a standard collaborative process to evaluate sequence, path, and status matching, wherein the path evaluation uses a dynamic fault tolerance threshold jointly determined by operational dynamics characteristics and device risk level; real-time risk assessment and feedback to the students' field of vision; and finally generating a comprehensive evaluation result of visualized debriefing data. This invention achieves dynamic risk assessment and debriefing for high-risk collaborative inspection operations, improving the management efficiency and teaching effectiveness of the training loop.

Description

Technical Field

[0001] This invention relates to the field of virtual reality simulation training technology, specifically to a VR-based multi-person virtual on-site inspection training method, equipment, and medium. Background Technology

[0002] In modern industrial systems, regular and specialized on-site inspections are crucial for ensuring safe equipment operation and preventing major accidents. These operations typically require two or more personnel to collaborate strictly following work permit procedures, such as "one person operates, one person monitors," or multiple people conducting area-specific inspections and then coordinating responses. Traditional on-site training models face significant challenges, including high risk, high cost, difficulty in organization, and limited training scenarios.

[0003] In recent years, virtual reality-based training systems have addressed the issue of individual skills training to some extent. However, existing technologies have significant shortcomings when supporting complex scenarios such as multi-person collaborative inspections: First, there is a disconnect between the virtual and real environments: the virtual environment is completely detached from the actual equipment operation, preventing trainees from perceiving and responding to real-time changes in key parameters such as pressure, temperature, and current, thus deviating from practical application. Second, there is a lack of collaborative logic evaluation: the standardized training evaluation of operational procedures typically relies on human experts to analyze video or data records step by step and judge the standardization of each key step based on subjective experience. This process is not only time-consuming and labor-intensive but also easily influenced by the experience differences of the evaluators when dealing with continuous, overlapping, and multi-objective operations, resulting in a lack of uniformity and objectivity in the evaluation conclusions. Finally, the risk assessment model is rigid: existing operational assessments mostly use fixed tolerance standards, failing to reflect the dynamic safety principles that "the higher the equipment risk, the lower the operational error tolerance should be" and "the more unstable the operation, the higher the risk bonus should be," thus the evaluation results fail to truly reflect the operational risks.

[0004] Therefore, there is an urgent need for a multi-user virtual on-site inspection training solution that can deeply integrate real-time data from physical equipment, support dynamic online interaction among multiple users, and conduct dynamic risk assessments of collaborative work processes. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a VR-based multi-user virtual on-site inspection training method, equipment, and medium.

[0006] A VR-based multi-user virtual on-site inspection training method, collaboratively executed by a server, at least two trainee VR terminals, and an IoT data gateway, the method comprising: S1: Synchronization of Virtual and Real Status and Multi-User Access: Through the IoT data gateway, real-time operating status data of physical equipment in the industrial field is obtained, and the real-time operating status data is mapped to the corresponding virtual objects in the virtual reality inspection scene according to the equipment number; responding to the login of at least two trainees, generating corresponding virtual avatars for them in the virtual scene, and assigning inspection roles; wherein the server issues a unified clock reference to each VR terminal and timestamps the data from each VR terminal and the IoT data gateway. S2: Multi-source collaborative data acquisition and binding: Real-time acquisition of raw data of each student's operation behavior, which includes at least timestamps, hand spatial trajectories, action triggering states, operation target device numbers, and cross-student voice communication data; at the same time, binding the raw data of operation behavior with the real-time operating status data of the device at the corresponding moment; S3: Collaborative Behavior Chain Construction: Based on the hand spatial trajectory and action triggering state of each student, identify and generate their individual action segment data; and fuse all students' action segments, voice communication events and device state sequences in timeline to construct dual-channel behavior chain data; wherein the first channel includes the event sequence of multiple students' action segments and their role identifiers, and the second channel includes the device state sequence aligned with the event sequence and the identifier of the reason for the state change; S4: Dynamic Collaborative Risk Assessment and Real-time Judgment: The dual-channel behavior chain data is compared with pre-stored standard collaborative inspection process data to calculate the team's operational sequence deviation, path offset, and equipment status matching degree caused by the operation, and to generate a basic collaborative risk score. The path offset determination uses a dynamic fault tolerance threshold, which is jointly determined by the basic fault tolerance threshold, trainee operational dynamic characteristic parameters, and equipment risk level. The dynamic characteristic parameters include at least instantaneous acceleration and trajectory jitter coefficient. The server performs real-time judgment on path offset based on the dynamic fault tolerance threshold and outputs risk events. S5: Real-time feedback and comprehensive evaluation generation: During the training, the risk events are fed back to the corresponding trainees' VR view in real time through overlay markers, directional guidance, or voice prompts; after the training, a comprehensive evaluation result is generated, which includes at least individual scores, team collaboration scores, and visualized debriefing data that integrates the movement trajectories of multiple trainees and risk event markers.

[0007] Further, in step S2, the raw data of the collected operation behavior includes: S21: By using the spatial positioning device on each student's VR terminal, collect the three-dimensional position coordinate sequence of the student's hands in the virtual space to form hand spatial trajectory data; S22: By detecting the trigger events of interactive objects in the virtual scene, record the start and end timestamps of the action, and obtain the field device number bound to the operated virtual object; S23: Acquire the voice communication stream between trainees through the audio acquisition device of the VR terminal, and perform voice activity detection and key instruction word recognition.

[0008] Furthermore, in step S3, identifying and generating motion fragment data includes performing kinematic analysis on the trajectory: S31: Calculate the real-time velocity and acceleration of the hand's spatial trajectory; S32: Identify trajectory segments with acceleration exceeding a preset threshold as potentially high-risk or high-intent operation zones; S33: Match and verify the operation interval with the time interval of the action trigger state to form action segment data with mobility tags.

[0009] Furthermore, the calculation of the dynamic fault tolerance threshold T in step S4 satisfies the following relationship: T = T0 * (1 + k1*F1 + k2*F2) * g(R), Where T0 is the basic fault tolerance threshold, F1 is the normalized value of the trajectory jitter coefficient, F2 is the normalized value of the instantaneous acceleration, k1 and k2 are adjustment coefficients, R is the equipment risk level, and g(R) is a function that monotonically decreases as the risk level R increases.

[0010] Furthermore, the step of calculating the order deviation of team operations in step S4 includes: S41: Extract the actual execution sequence formed by the interweaving of multi-student operations from the dual-channel behavior chain; S42: Compare the actual execution sequence with the sequence specified in the standard collaborative process, and calculate the deviation using a sequence comparison algorithm based on a role-step dependency constraint graph. The constraint graph defines a set of parallelizable steps, sequential dependencies, and a set of mutually exclusive steps. During the comparison, parallel operations that satisfy the constraints are not included in the order deviation. Step S4 also includes a collaborative compliance determination based on spatial relationships: S43: When the first student performs a preset high-risk operation, check whether the virtual avatar of the second student, who is collaborating with him, is located within the preset security monitoring area; S44: Further detect whether the field of vision of the second student covers the first student or key equipment during the operation duration; If S43 or S44 fails, it will be recorded as a collaboration violation and will affect the team collaboration score.

[0011] Furthermore, the generation of comprehensive evaluation results in step S5 includes generating an immersive interactive debriefing scenario: S51: Reconstruct the complete training process timeline, including all trainees' virtual avatars, in a VR environment; S52: Allows users to replay from a free perspective, and during the replay process, the recorded real-time risk warnings, path deviation points, and collaborative violation events are visualized in the spatiotemporal location of the event in the form of 3D annotation, highlighting, and charts. S53: Provides team collaboration review and annotation functions based on playback scenarios.

[0012] Furthermore, the timestamp alignment includes: the server performs deviation correction on the local timestamps of each VR terminal based on network round-trip latency measurement, and uses a sliding time window to align and bind voice communication events, action triggering events and device status change events; the status change reason identifier is used to distinguish whether the device status change is triggered by student operation, by fault injection on the tutor end, or by actual changes in the status of the device on site.

[0013] A VR-based multi-user virtual on-site inspection training system for implementing the above method includes: The server side includes: The IoT data synchronization service module is used to access and forward physical device status data; The multi-user session management and synchronization module is used to manage student connections, role assignments, virtual scene state synchronization, and timestamp alignment. The collaborative behavior chain construction and risk assessment engine is used to perform dual-channel behavior chain construction, dynamic fault tolerance threshold calculation, collaborative compliance determination, and risk event generation. The assessment and debriefing data generation module is used to generate scoring reports and immersive debriefing scenario data; Multiple VR terminals for trainees are connected to the server to present first-person virtual scenes and collect trainee operation and voice data. At least one tutor monitoring terminal is connected to the server to view the overall scene, intervene in the training process, manually trigger fault events, and view evaluation reports.

[0014] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the VR-based multi-person virtual on-site inspection training method described above.

[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the VR-based multi-person virtual on-site inspection training method described above.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention constructs a dual-channel behavior chain data structure with synchronized status by aligning and fusing real device status data from the Internet of Things with operation, trajectory, and voice data from multi-student VR terminals using timestamps. This enables a digital, structured, and precise representation of the entire process of multi-person collaborative inspection operations without human intervention. This data foundation not only fully records "who, when, what device, and what operation was performed," but also links it to "the actual status of the device at the time of the operation," providing multi-dimensional correlation information for subsequent in-depth analysis.

[0017] Furthermore, this invention introduces a dynamic fault-tolerant assessment model driven by both operational dynamics characteristics and real-time equipment risk levels. This model makes the judgment of operational path compliance no longer a rigid fixed threshold, but can intelligently adjust according to the dynamic safety logic of "the more unstable the operation, the stricter the assessment" and "the higher the equipment risk, the tighter the fault tolerance." Combined with sequence comparison based on role constraints and real-time judgment of collaborative compliance based on spatial relationships, it realizes a comprehensive and adaptive risk assessment of team operations in terms of sequence, path, state response, and collaborative cooperation. This elevates training assessment from a simple judgment of the correctness of steps to a precise measurement of operational risk.

[0018] Ultimately, this invention constructs a teaching closed loop of real-time early warning during training and in-depth analysis after training through real-time multimodal risk feedback and three-dimensional debriefing scenarios. Real-time feedback transforms abstract risk data into intuitive visual and auditory prompts, reinforcing trainees' safe operating habits; while the three-dimensional debriefing scenario allows teams to replay and examine risk details in a virtual space-time, enabling visualized diagnosis and efficient summarization of complex collaborative problems.

[0019] In summary, this invention achieves automated, intelligent, and high-fidelity training and evaluation of multi-person collaborative virtual inspection operations without relying on on-site supervision and subjective scoring by human experts, thereby improving the safety, effectiveness, and traceability of skills training for high-risk industry teams. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0021] Figure 1 The flowchart illustrates a VR-based multi-person virtual on-site inspection training method provided in Embodiment 1 of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1 Please see Figure 1 As shown, this embodiment discloses a VR-based multi-user virtual on-site inspection training method, which is executed collaboratively by a server, at least two trainee VR terminals, and an IoT data gateway. The method includes: S1: Through the IoT data gateway, obtain real-time operating status data (including but not limited to switch position, pressure value, temperature value, and current value) of physical equipment in the industrial field (such as circuit breakers in substations and valves in chemical plant reactors), and map the real-time operating status data to the corresponding virtual objects in the virtual reality inspection scene according to the equipment number, so as to ensure that the status of the virtual objects (such as model posture, indicator light color, and instrument reading) is consistent with the physical entities.

[0024] In response to login requests from at least two trainees, the server generates a unique virtual avatar for each trainee in the virtual scene and assigns inspection roles (e.g., main operator, coordinating monitor, recorder) according to the preset training script. To ensure the consistency of multi-source data timing, the server uses the NTP protocol or a hardware timestamp-based method to issue a unified clock reference to each VR terminal and performs arrival time deviation correction and timestamp alignment on data packets from each VR terminal (hand trajectory, motion events, voice stream) and the IoT data gateway (device status stream).

[0025] In one specific embodiment, the server maintains a global time alignment buffer. For any critical event (such as student A pressing the "operation confirmation" button), the system records the original timestamp t1 of the event occurring locally on the terminal. Combined with the network round-trip time (RTT) between the terminal and the server and the clock deviation δ, the unified timestamp t2 on the global time axis is calculated using the formula t2 = t1 + δ + RTT / 2. For IoT status data, time stamp conversion and alignment are also performed. The global timestamp accuracy of all events must be controlled within 10 milliseconds to ensure the accuracy of subsequent behavior chain fusion.

[0026] S2: Multi-source collaborative data acquisition and binding The system collects raw data of each student's actions in real time. Using a spatial positioning device (such as a laser positioning base station or an Inside-Out visual tracking system) on each student's VR terminal, the system collects the three-dimensional position coordinates (x, y, z) sequence of the student's hands in the virtual space at a preset sampling frequency (such as 90Hz), forming hand spatial trajectory data.

[0027] By detecting collision and trigger events of interactive objects (such as virtual buttons, handles, and tools) in the virtual scene, the start and end timestamps of the actions are recorded to form the action trigger state, and the field device number bound to the operated virtual object is obtained.

[0028] The system acquires voice communication data streams between trainees through audio acquisition devices integrated into the VR terminal. The system processes the voice streams in real time, including voice activity detection (VAD) to identify valid voice segments, and converts key command words (such as "start", "confirm", "stop") into text events through a voice recognition engine, marking the speaker and timestamp.

[0029] In a specific embodiment, the calculation logic for collecting spatial trajectory data of both hands is as follows: For each sampling period, the current position coordinates of the left and right handles are calculated respectively. For the single-hand trajectory, the cumulative trajectory length L is calculated as follows: Take the coordinates of the current sampling point P1 and the previous point P0, calculate the three-dimensional Euclidean distance d =√((x1 - x0)²+ (y1- y0)²+ (z1 - z0)²), and then add d to the total length L0 of the previous cycle, i.e., L = L0 + d. The trajectories of both hands are calculated and recorded independently.

[0030] Key binding operation: When an action trigger record is generated (for example, student A triggers the "close" operation on device "V101" at time t1), the system immediately retrieves the real-time status data of device "V101" received from the IoT data gateway within the time window [t1-Δt, t1+Δt] (for example, Δt=50ms) (such as "valve opening: 100%→0%), and binds this status snapshot with the action record. At the same time, it retrieves relevant voice events within the same time window, such as student B's repeating instruction "V101 has been confirmed closed".

[0031] S3: Construction of Collaborative Behavior Chains Based on each trainee's hand spatial trajectory and action triggering state, individual action segment data is identified and generated. First, kinematic analysis is performed on the trajectory: the real-time moving speed v and acceleration a of the hand spatial trajectory are calculated, and continuous trajectory segments with acceleration a exceeding a preset safety threshold (e.g., 5 m / s²) are identified. These segments are marked as potentially high-risk or high-intention operation intervals. Then, these intervals are matched and verified with the time intervals of the action triggering state. If the overlap between the two times exceeds a certain proportion (e.g., 70%), it is confirmed as a valid action segment. Each action segment data includes: start / end time, average / peak speed, peak acceleration, target device number, and executing role.

[0032] In a specific embodiment, the acceleration calculation logic of the action segment is as follows: take three consecutive sampling points P1, P2, P3 within the trajectory segment, with timestamps t1, t2, t3, corresponding to velocities v1, v2, and instantaneous acceleration a ≈ (v2 - v1) / (t3 - t1). The trajectory jitter coefficient J can be characterized by calculating the standard deviation of the acceleration of all sampling points within the segment. The larger the J value, the more unstable the operation.

[0033] All student action segments, recognized voice communication events (such as commands, confirmations, and alarms), and synchronized device status sequences are fused together according to a globally aligned timeline to construct dual-channel behavior chain data. This data is structured and stored as an event sequence along a single timeline. First channel (behavior channel): Records events in chronological order, such as [t1: Role A, action "approaching device V101"], [t2: Role A, voice "preparing to operate V101"], [t3: Role B, voice "received, can operate"], [t4: Role A, "action to close V101"].

[0034] The second channel (status channel) is precisely aligned with the events of the first channel, recording the corresponding equipment status, such as [t1: V101 status = open, pressure = 1.0MPa], [t4: V101 status = closed, pressure = 0.9MPa], and marking the reason for each status change (such as "student operation", "system fault injection", "real equipment abnormality").

[0035] S4: Dynamic Collaborative Risk Assessment and Real-time Judgment The constructed dual-channel behavior chain data is compared with the pre-stored standard collaborative inspection process data in multiple dimensions.

[0036] Sequence deviation calculation: Extract the actual operation sequence (including role and equipment number) from the behavior chain and compare it with the sequence specified in the standard process. Use an edit distance algorithm based on the role-step dependency constraint graph. For example, the standard process allows "role A to check the instrument" and "role B to record data" to be executed in parallel. When comparing, the algorithm does not calculate such compliant parallel operations as sequence errors. It only calculates the edit distance for sequences that violate dependency (such as "check before recording") or mutual exclusion (such as "two people operate the same equipment at the same time") and generates a sequence deviation score.

[0037] Path offset calculation: This is one of the core innovations of this invention. It compares the actual hand trajectory point sequence when the trainee operates the device with the ideal operation path point sequence defined in the standard procedure. The key point is the use of a dynamic fault tolerance threshold T. A specific embodiment of its calculation formula is as follows: T = T0 * (1 + k1*F1 + k2*F2) * g(R), in: T0 is the basic tolerance radius for this operation step (e.g., 0.1 meters). F1 is the normalized value of the trajectory jitter coefficient J (between 0 and 1, where 1 indicates severe jitter). F2 is the normalized proportion of the instantaneous acceleration a exceeding the safety threshold; k1 and k2 are adjustment coefficients (e.g., k1=0.3, k2=0.5) used to control the weight of the impact of jitter and acceleration on fault tolerance; R represents the real-time risk level of the equipment (e.g., low, medium, or high based on voltage, pressure, and temperature). g(R) is the risk discount function. For example, for low risk, g(R) = 1.2, for medium risk, g(R) = 1.0, and for high risk, g(R) = 0.6. The function decreases monotonically as the risk level increases, which means that the higher the risk, the stricter the fault tolerance margin. Calculate the distance D from each actual trajectory point to the corresponding point on the standard path. If D > T, the point is determined to be a path deviation point. Calculate the proportion and severity of all deviation points and generate a path deviation score.

[0038] State matching degree calculation: Check whether the actual sequence of equipment state changes after the "student operation" event in the behavior chain matches the expected sequence of state changes (including the direction of change and approximate time) in the standard process. For example, after the valve is closed, the pressure should drop to a certain range within a specified time. If it does not match, points will be deducted.

[0039] Real-time collaborative compliance assessment: When the behavior chain detects that trainee A (operator) begins to perform a high-risk operation (such as high-voltage testing), the system immediately initiates a assessment: Spatial location determination: Check whether the current location of the virtual avatar of trainee B (guardian) is within the preset safety monitoring radius (e.g., 2 meters) centered on A.

[0040] Line of sight coverage determination: A ray is emitted from the eye position of student B's virtual avatar in the direction it is facing, and it is detected whether the cumulative intersection of the ray with the collision object of student A or its handheld tool exceeds the minimum monitoring time threshold (e.g., 3 seconds) during the operation duration.

[0041] If any of the above judgments fails, the system will immediately generate a collaborative violation event, which will affect the subsequent team collaboration score.

[0042] Based on the above real-time comparison and judgment results, the server dynamically generates risk events (such as "path deviation exceeds limits" and "lack of supervision").

[0043] S5: Real-time feedback and comprehensive evaluation generation During training, the system converts risk events into multimodal feedback in real time and pushes them to the VR field of view of the relevant trainees: for path deviation, a dynamically scaling semi-transparent warning circle is displayed near the trainee's controller, with the color changing from green to red to represent the degree of deviation; for collaborative violations, a red warning bar flashes at the edge of the violating trainee's field of view and displays brief text (such as "Please pay attention to the operation point!"); if the trainee deviates too far from the standard inspection route, a dynamic arrow can appear on the ground to guide them back to the correct path; risk warning voice can be played from the three-dimensional spatial location where the event occurred, enhancing immersion and sense of direction.

[0044] After the training, the system automatically generates a comprehensive evaluation result: Individual scoring: The scoring is based on a weighted calculation of each student's order deviation, path offset, and operation time (e.g., weight: order 40%, path 40%, operation time 20%).

[0045] Team collaboration score: calculated based on factors such as the number of collaborative violations, voice command response latency, and synchronization of key parallel operations.

[0046] Visualized debriefing data: Generates an interactive 3D debriefing scene that integrates the movement trajectories of multiple trainees and risk event markers. Within this scene, the entire training process can be replayed in a timeline-sliding manner, simultaneously rendering the movement trajectories of all trainees (lined in different colors); at the location where a risk event occurred (e.g., next to a piece of equipment), a snapshot of the risk type at that time is displayed (e.g., "Excessive acceleration: 15.3 m / s"). 2 "), dynamic fault tolerance threshold (T=0.08m) and actual offset distance (D=0.12m).

[0047] Mentors or trainees can freely roam in the 3D scene, examine the team's position and line of sight from any angle, and add voice or text annotations to keyframes.

[0048] This invention provides a VR-based multi-person virtual on-site inspection training method, which has at least the following technical effects: It synchronizes real-time status data of real industrial equipment to the virtual scene via an IoT interface, ensuring a high degree of consistency between the multi-person collaborative training environment and the physical world. This fundamentally changes the disconnect between traditional VR training and real-world application, allowing trainees to practice team decision-making and response capabilities based on real-time operating conditions in a highly realistic environment. By constructing a dual-channel behavior chain that integrates multi-trainee behavior and equipment status, and performing sequence analysis based on role dependency constraints, the system can automatically and objectively determine the compliance of core collaborative safety elements such as "operation-monitoring" coordination, command repetition, and parallel operation sequence. This achieves precise quantitative evaluation of team collaboration quality, replacing reliance on manual observation. Subjective evaluation; pioneering a dynamic intelligent assessment model based on risk perception, the proposed dynamic fault tolerance threshold model combines the dynamic characteristics of operation (acceleration, jitter) with the real-time risk level of equipment, enabling the assessment criteria to be dynamically adjusted according to the operational risk. This marks a leap from static judgment of "right or wrong steps" to dynamic perception of "risk level," making the scoring results more scientifically reflect the safety of operations in real high-risk environments. The assessment results not only include structured scoring but also provide an interactive debriefing scenario that integrates multi-trainee action trajectories, three-dimensional annotation of risk events, and spatiotemporal information visualization. This allows trainees and instructors to examine the entire collaboration process from multiple perspectives, achieving intuitive and in-depth diagnosis of team coordination shortcomings, and improving the management efficiency and teaching effectiveness of the training loop.

[0049] Example 2 This embodiment discloses an electronic device, including: The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the VR-based multi-user virtual on-site inspection training method described above.

[0050] Since the electronic device described in this embodiment is the electronic device used to implement the VR-based multi-person virtual on-site inspection training method in this application embodiment, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this embodiment based on the VR-based multi-person virtual on-site inspection training method described in this application embodiment. Therefore, how the electronic device implements the method in this application embodiment will not be described in detail here. Any electronic device used by those skilled in the art to implement the VR-based multi-person virtual on-site inspection training method in this application embodiment falls within the scope of protection of this application.

[0051] Example 3 This embodiment discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the VR-based multi-person virtual on-site inspection training method described above.

[0052] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A VR-based multi-user virtual on-site inspection training method, characterized in that, The method, executed collaboratively by a server, at least two student VR terminals, and an IoT data gateway, includes: S1: Synchronization of Virtual and Real Status and Multi-User Access: Through the IoT data gateway, real-time operating status data of physical equipment in the industrial field is obtained, and the real-time operating status data is mapped to the corresponding virtual objects in the virtual reality inspection scene according to the equipment number; responding to the login of at least two trainees, generating corresponding virtual avatars for them in the virtual scene, and assigning inspection roles; wherein the server issues a unified clock reference to each VR terminal and timestamps the data from each VR terminal and the IoT data gateway. S2: Multi-source collaborative data acquisition and binding: Real-time acquisition of raw data of each student's operation behavior, which includes at least timestamps, hand spatial trajectories, action triggering states, operation target device numbers, and cross-student voice communication data; at the same time, binding the raw data of operation behavior with the real-time operating status data of the device at the corresponding moment; S3: Collaborative Behavior Chain Construction: Based on the hand spatial trajectory and action triggering state of each student, identify and generate their individual action segment data; and fuse all students' action segments, voice communication events and device state sequences in timeline to construct dual-channel behavior chain data; wherein the first channel includes the event sequence of multiple students' action segments and their role identifiers, and the second channel includes the device state sequence aligned with the event sequence and the identifier of the reason for the state change; S4: Dynamic Collaborative Risk Assessment and Real-time Judgment: The dual-channel behavior chain data is compared with pre-stored standard collaborative inspection process data to calculate the team's operational sequence deviation, path offset, and equipment status matching degree caused by the operation, and to generate a basic collaborative risk score. The path offset determination uses a dynamic fault tolerance threshold, which is jointly determined by the basic fault tolerance threshold, trainee operational dynamic characteristic parameters, and equipment risk level. The dynamic characteristic parameters include at least instantaneous acceleration and trajectory jitter coefficient. The server performs real-time judgment on path offset based on the dynamic fault tolerance threshold and outputs risk events. S5: Real-time feedback and comprehensive evaluation generation: During the training, the risk events are fed back to the corresponding trainees' VR view in real time through overlay markers, directional guidance, or voice prompts; after the training, a comprehensive evaluation result is generated, which includes at least individual scores, team collaboration scores, and visualized debriefing data that integrates the movement trajectories of multiple trainees and risk event markers.

2. The method according to claim 1, characterized in that, In step S2, the raw data of the data collection operation includes: S21: By using the spatial positioning device on each student's VR terminal, collect the three-dimensional position coordinate sequence of the student's hands in the virtual space to form hand spatial trajectory data; S22: By detecting the trigger events of interactive objects in the virtual scene, record the start and end timestamps of the action, and obtain the field device number bound to the operated virtual object; S23: Acquire the voice communication stream between trainees through the audio acquisition device of the VR terminal, and perform voice activity detection and key instruction word recognition.

3. The method according to claim 2, characterized in that, In step S3, identifying and generating motion fragment data includes performing kinematic analysis on the trajectory: S31: Calculate the real-time velocity and acceleration of the hand's spatial trajectory; S32: Identify trajectory segments with acceleration exceeding a preset threshold as potentially high-risk or high-intent operation zones; S33: Match and verify the operation interval with the time interval of the action trigger state to form action segment data with mobility tags.

4. The method according to claim 1, characterized in that, The calculation of the dynamic fault tolerance threshold T in step S4 satisfies the following relationship: T = T0 * (1 + k1*F1 + k2*F2) * g(R), Where T0 is the basic fault tolerance threshold, F1 is the normalized value of the trajectory jitter coefficient, F2 is the normalized value of the instantaneous acceleration, k1 and k2 are adjustment coefficients, R is the equipment risk level, and g(R) is a function that monotonically decreases as the risk level R increases.

5. The method according to claim 1, characterized in that, The step of calculating the order deviation of team operations in step S4 includes: S41: Extract the actual execution sequence formed by the interweaving of multi-student operations from the dual-channel behavior chain; S42: Compare the actual execution sequence with the sequence specified in the standard collaborative process, and calculate the deviation using a sequence comparison algorithm based on a role-step dependency constraint graph. The constraint graph defines a set of parallelizable steps, sequential dependencies, and a set of mutually exclusive steps. During the comparison, parallel operations that satisfy the constraints are not included in the order deviation. Step S4 also includes a collaborative compliance determination based on spatial relationships: S43: When the first student performs a preset high-risk operation, check whether the virtual avatar of the second student, who is collaborating with him, is located within the preset security monitoring area; S44: Further detect whether the field of vision of the second student covers the first student or key equipment during the operation duration; If S43 or S44 fails, it will be recorded as a collaboration violation and will affect the team collaboration score.

6. The method according to claim 1, characterized in that, Step S5, which involves generating a comprehensive evaluation result, includes generating an immersive interactive debriefing scenario: S51: Reconstruct the complete training process timeline, including all trainees' virtual avatars, in a VR environment; S52: Allows users to replay from a free perspective, and during the replay process, the recorded real-time risk warnings, path deviation points, and collaborative violation events are visualized in the spatiotemporal location of the event in the form of 3D annotation, highlighting, and charts. S53: Provides team collaboration review and annotation functions based on playback scenarios.

7. The method according to claim 1, characterized in that, The timestamp alignment includes: the server performs deviation correction on the local timestamps of each VR terminal based on network round-trip latency measurement, and uses a sliding time window to align and bind voice communication events, action triggering events and device status change events; the status change reason identifier is used to distinguish whether the device status change is triggered by student operation, by fault injection on the instructor's end, or by actual changes in the status of the device on site.

8. A VR-based multi-user virtual on-site inspection training system for implementing the method as described in any one of claims 1 to 7, characterized in that, include: The server side includes: The IoT data synchronization service module is used to access and forward physical device status data; The multi-user session management and synchronization module is used to manage student connections, role assignments, virtual scene state synchronization, and timestamp alignment. The collaborative behavior chain construction and risk assessment engine is used to perform dual-channel behavior chain construction, dynamic fault tolerance threshold calculation, collaborative compliance determination, and risk event generation. The assessment and debriefing data generation module is used to generate scoring reports and immersive debriefing scenario data; Multiple VR terminals for trainees are connected to the server to present first-person virtual scenes and collect trainee operation and voice data. At least one tutor monitoring terminal is connected to the server to view the overall scene, intervene in the training process, manually trigger fault events, and view evaluation reports.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images