A Virtual Reality-Based Interactive Safety Training Method and System

Abstract

This invention relates to the field of safety training technology, and discloses a virtual reality-based interactive safety training method and system. The method includes the following steps: constructing a virtual environment; designing training scenarios and safety training materials of varying difficulty; selecting corresponding safety training materials; demonstrating the safety training materials to trainees; performing virtual operations on the training content and recognizing their states; dynamically adjusting the difficulty level of the training scenarios based on the trainees' states; analyzing the training learning effect and optimizing the safety training materials based on the learning effect. This invention also discloses a virtual reality-based interactive safety training system. This invention can dynamically adjust the scenario difficulty according to the trainees' emotional state, achieving personalized training, and can provide an effective feedback mechanism to improve the effectiveness and efficiency of training, better meeting the safety training needs of enterprises.

Description

Technical Field

[0001] This invention relates to the field of safety training technology, and more specifically, to a virtual reality-based interactive safety training method and system. Background Technology

[0002] Safety training is a crucial component of modern industrial production. This training helps employees understand and master various safety knowledge and operational skills, thereby preventing and avoiding various safety accidents. This is of great significance for protecting the lives of employees, ensuring the smooth operation of production, and avoiding economic losses caused by safety accidents.

[0003] Traditional safety training methods primarily rely on lectures and demonstrations. Coaches or instructors explain and demonstrate correct operating procedures and methods to employees, who then imitate and learn from their demonstrations. While simple and easy to implement, this method is not ideal because it lacks hands-on practice, making it difficult for employees to translate learned knowledge and skills into practical abilities. Furthermore, this method fails to simulate real-world work environments and conditions; problems and difficulties that employees might encounter in actual work are often not effectively simulated and addressed during training.

[0004] To address this issue, people have begun experimenting with Virtual Reality (VR) technology for safety training. VR technology can create a realistic virtual environment where employees can simulate various safety procedures. This method not only improves training effectiveness but also avoids potential safety risks in actual operations. Employees can practice repeatedly in the virtual environment until they fully master the required knowledge and skills.

[0005] However, traditional VR-based safety training methods often simply simulate actual operations, failing to provide effective feedback mechanisms and neglecting the emotional state of trainees, which can easily negatively impact training effectiveness. Furthermore, traditional training methods primarily employ passive environmental displays and operational demonstrations, which suffer from a lack of immersive experience and the inability to engage in real-time interaction. Summary of the Invention

[0006] In response to the problems in related technologies, this invention proposes a virtual reality-based interactive safety training method and system to overcome the aforementioned technical problems in existing related technologies.

[0007] Therefore, the specific technical solution adopted by the present invention is as follows:

[0008] According to one aspect of the present invention, a virtual reality-based interactive safety training method is provided, the method comprising the following steps:

[0009] S1. Use 3D modeling software to build a virtual environment based on the actual working environment and operation process, and set virtual characters in the virtual environment;

[0010] S2. Design training scenarios and safety training materials of varying difficulty based on training needs;

[0011] S3. Trainees enter the virtual environment through an interactive terminal and select the corresponding security training materials according to their job requirements;

[0012] S4. Respond to the selected operation instructions of the trainees and use the virtual role to demonstrate the selected safety training materials to the trainees in a first-person perspective.

[0013] S5. Trainees use interactive terminals to virtually operate the training content based on the training results, and use fusion recognition algorithms to identify the trainees' status during the virtual operation process.

[0014] S6. Dynamically adjust the difficulty level of the training scenario according to the trainee's status until the trainee masters the safety training content corresponding to their position.

[0015] S7. Analyze the training effectiveness using all operational behaviors and process data of trainees, and optimize safety training materials based on the training effectiveness.

[0016] Furthermore, the virtual character is used to demonstrate actions to trainees, showcasing correct and incorrect operations from a first-person perspective. It is also used to interact with trainees and provide corresponding feedback based on their actions.

[0017] Furthermore, the design of training scenarios and safety training materials of varying difficulty based on training needs includes the following steps:

[0018] S21. Define the training objectives and needs, including the training topic, the target audience, and the difficulty level of the training;

[0019] S22. Design training scenarios of varying difficulty based on training needs, including basic safety operations and basic equipment use for basic training, common safety issues and advanced equipment use for intermediate training, and handling of sudden safety incidents and complex safety issues for advanced training.

[0020] S23. Prepare corresponding safety training materials according to the training scenarios, including detailed descriptions of different training scenarios, operating procedures and precautions.

[0021] Furthermore, the trainees, based on the training results, use an interactive terminal to virtually operate the training content, and the fusion recognition algorithm, combined with physiological parameters, is used to identify the trainees' state during the virtual operation process, including the following steps:

[0022] S51. Based on the training results, trainees select virtual operation options on the interactive terminal and perform the corresponding virtual operations in the virtual environment.

[0023] S52. Use an attention-based multimodal fusion recognition algorithm combined with physiological parameters to identify the state of trainees during virtual operation.

[0024] The method of using an attention-based multimodal fusion recognition algorithm combined with physiological parameters to identify the trainee's state during virtual operation includes the following steps:

[0025] S521. Extract the eye movement characteristics, EEG characteristics, and cross-modal interaction characteristics of trainees during virtual operation;

[0026] S522. Use semantic conversion technology to map eye-tracking features, EEG features, and cross-modal up-down interaction features to the same vector space;

[0027] S523. Fuse the residual information of low-dimensional features from different periods, and classify the fused features into states to obtain the recognition results based on the fused features.

[0028] S524. Use the physiological parameters of trainees during the virtual operation process to identify the trainees' state during the virtual operation process and obtain the identification results based on the physiological parameters.

[0029] S525. The final state recognition result is obtained by analyzing the recognition results based on fusion features and the recognition results based on physiological parameters.

[0030] Furthermore, the extraction of eye-tracking features, electroencephalogram (EEG) features, and cross-modal interaction features of trainees during virtual operation includes the following steps:

[0031] S5211. Collect eye movement data and electroencephalogram data of trainees during virtual operation, and perform sliding window and normalization processing.

[0032] S5212. Features of EEG data and eye-tracking data are extracted using the Transformer-CNN model and the Transformer-based multi-head attention mechanism, respectively.

[0033] S5213. Similarity calculation is used to obtain the similarity between cross-modal data, and the weights are updated through an attention mechanism to obtain cross-modal interaction features.

[0034] The step of obtaining the similarity between cross-modal data by calculating similarity and updating the weights through an attention mechanism to obtain cross-modal interaction features includes the following steps:

[0035] Linear transformation techniques are used to map EEG and eye-tracking data into the attention space;

[0036] The similarity of attention vectors between data points in EEG and eye-tracking data is calculated using dot product and then normalized.

[0037] The influence of eye-tracking data on the context of EEG data at a certain time step is calculated using attention weights to obtain the transmembrane context vector;

[0038] The cross-modal context vector is mapped to the attention space through a linear transformation and normalized as the attention weight of the cross-modal context vector. The cross-modal interaction features are obtained by weighting the cross-modal context vector with the attention weight.

[0039] Furthermore, the process of fusing residual information from low-dimensional features at different times and classifying the fused features to obtain a recognition result based on the fused features includes the following steps:

[0040] S5231. Perform deep feature fusion on the converted eye movement features, EEG features, and cross-modal up-down interaction features at different stages: early, middle, and late.

[0041] S5232. The trainees' status is classified by using a fully connected layer and a Softmax activation function to fused features, resulting in a recognition result based on the fused features.

[0042] Furthermore, the feature fusion at different stages—early, middle, and late—of the converted eye-tracking features, EEG features, and cross-modal up-down interaction features includes:

[0043] In the early fusion stage, the eye-tracking data and EEG data before feature extraction are fused using the concat fusion strategy. Multi-head attention is used to assign weights to the early fused features and perform dimensional transformation.

[0044] In the intermediate fusion stage, a deep fusion strategy is used to fuse the features extracted and semantically transformed eye-tracking data and the features fused in the early fusion of the EEG data domain;

[0045] In the late fusion stage, a deep fusion strategy is used to fuse the results of the mid-term fusion with cross-modal context interaction features;

[0046] The deep fusion strategy first fuses the two inputs by directly adding them together using the fusion strategy add. Then, it selects effective features and assigns them greater weights through a multi-head attention mechanism. Finally, it adds a multilayer perceptron after the multi-head attention mechanism to deeply fuse the features by first increasing the dimensionality of the features and then decreasing the dimensionality.

[0047] Furthermore, the analysis of the recognition results based on fusion features and the recognition results based on physiological parameters to obtain the final state recognition result includes the following steps:

[0048] Compare the identification results based on fusion features and those based on physiological parameters to see if they are the same. If they are, arbitrarily select one of the two identification results as the final state identification result. If not, conduct a comprehensive analysis based on the trainee's historical performance and the difficulty of the training task to obtain the final state identification result.

[0049] Furthermore, the step of dynamically adjusting the difficulty level of the training scenario based on the trainee's condition until the trainee masters the safety training content corresponding to their job position includes:

[0050] Obtain the final state recognition result of the trainee and determine whether the final recognition result is a tense state. If not, maintain the current difficulty level of the training scenario until the trainee masters the safety training content corresponding to their position. If so, reduce the difficulty level of the training scenario and repeat the training scenario at the reduced difficulty level until the trainee's state recognition result is a relaxed state. Then restore the difficulty level of the training scenario and have the virtual character provide virtual operation prompts corresponding to the training scenario at this difficulty level until the trainee's state recognition result returns to a normal or relaxed state. Finally, the trainee conducts training according to the prompts until they master the safety training content corresponding to their position.

[0051] According to another aspect of the present invention, a virtual reality-based interactive safety training system is provided. The system includes a virtual environment construction module, a safety training material design module, a safety training material selection module, a safety training material demonstration module, a virtual operation and status recognition module, a training scenario difficulty level dynamic adjustment module, and a training learning effect analysis module connected in sequence.

[0052] The virtual environment construction module is used to construct a virtual environment based on the actual working environment and operation process using 3D modeling software, and to set virtual characters in the virtual environment.

[0053] The safety training materials design module is used to design training scenarios and safety training materials of different difficulty levels according to training needs.

[0054] The safety training material selection module is used by trainees to enter a virtual environment through an interactive terminal and select the corresponding safety training materials according to their job requirements.

[0055] The safety training material demonstration module is used to respond to the selection operation instructions of the trainees and use virtual characters to demonstrate the selected safety training materials to the trainees in a first-person perspective.

[0056] The virtual operation and status recognition module is used by trainees to perform virtual operations of training content using an interactive terminal based on training results, and to recognize the status of trainees during the virtual operation process using a fusion recognition algorithm.

[0057] The training scenario difficulty level dynamic adjustment module is used to dynamically adjust the difficulty level of the training scenario according to the trainee's status until the trainee masters the safety training content corresponding to their job position.

[0058] The training effectiveness analysis module is used to analyze the training effectiveness using all operational behaviors and process data of trainees, and to optimize safety training materials based on the training effectiveness.

[0059] The beneficial effects of this invention are as follows:

[0060] 1) This invention can not only construct an immersive virtual training environment and set up humanized virtual characters to enable first-person perspective action demonstrations and voice interaction, but also use a fusion recognition algorithm to identify the emotional state of trainees during virtual operations. This allows for dynamic adjustment of the scene difficulty based on the trainees' emotional state, achieving personalized training. Furthermore, it can analyze the training learning effect based on all the trainees' operational behaviors and process data, and optimize safety training materials based on the training learning effect. This provides an effective feedback mechanism to improve the effectiveness and efficiency of training, thereby better meeting the safety training needs of enterprises.

[0061] 2) This invention can not only use the eye movement features, EEG features and cross-modal interaction features of trainees during virtual operation to obtain recognition results based on fusion features, but also use the physiological parameters of trainees during virtual operation to obtain recognition results based on physiological parameters. Thus, the recognition results based on fusion features and the recognition results based on physiological parameters can be comprehensively analyzed to obtain the final state recognition result, thereby effectively improving the accuracy of trainee state recognition. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is a flowchart of a virtual reality-based interactive safety training method according to an embodiment of the present invention. Detailed Implementation

[0064] To further illustrate the various embodiments, the present invention provides accompanying drawings, which are part of the disclosure of the present invention. These drawings are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these drawings, those skilled in the art should be able to understand other possible implementation methods and the advantages of the present invention. The components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0065] According to an embodiment of the present invention, a virtual reality-based interactive safety training method and system are provided.

[0066] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments, such as... Figure 1 As shown, according to an embodiment of the present invention, a virtual reality-based interactive safety training method is provided, the method comprising the following steps:

[0067] S1. Use 3D modeling software to build a virtual environment based on the actual working environment and operation process, and set virtual characters in the virtual environment;

[0068] Specifically, some interactive items need to be pre-embedded during modeling, such as buttons, switches, and meters;

[0069] The virtual character is used to demonstrate actions to trainees, showing correct and incorrect operations from a first-person perspective. It also interacts with trainees, providing feedback based on their actions; specifically, the virtual character can guide learners at key points or evaluate their performance. Furthermore, this embodiment can include multiple virtual characters with different personalities, increasing the diversity of interactive scenarios.

[0070] S2. Design training scenarios and safety training materials of varying difficulty based on training needs;

[0071] The step of designing training scenarios and safety training materials of varying difficulty based on training needs includes the following steps:

[0072] S21. Define the training objectives and needs, including the training topic, target audience, and difficulty level.

[0073] S22. Design training scenarios of varying difficulty based on training needs, including:

[0074] Beginner Level: This level of training is primarily designed for beginners or those without prior knowledge. Basic scenarios can be designed, such as fundamental safety procedures and basic equipment usage.

[0075] Intermediate Level: This level of training is suitable for individuals who have already grasped some basic knowledge but need to further improve their skills. Slightly more complex scenarios can be designed, such as handling common security issues or using advanced equipment.

[0076] Advanced Level: This level of training scenarios is suitable for individuals with some experience who need to further enhance their professional skills. Complex scenarios can be designed, such as responding to sudden security incidents and handling complex security issues.

[0077] S23. Compile corresponding safety training materials according to the training scenarios, including detailed descriptions of different training scenarios, operating procedures and precautions, etc.

[0078] S3. Trainees enter the virtual environment through an interactive terminal and select the corresponding security training materials according to their job requirements;

[0079] The process of trainees entering a virtual environment via an interactive terminal and selecting appropriate security training materials based on their job requirements includes the following steps:

[0080] Starting the interactive terminal: First, the trainer needs to start the interactive terminal, which may be a computer, a smartphone, or a virtual reality device.

[0081] Entering the virtual environment: After launching the interactive terminal, trainees need to enter the virtual environment through the terminal's user interface. This may require entering a username and password, or performing other authentication procedures.

[0082] Selecting a Role: After entering the virtual environment, trainees need to select their role. This may be done by clicking on an option on the screen or by moving to the corresponding location within the virtual environment.

[0083] Selecting Safety Training Materials: After selecting a job position, the system will display corresponding safety training materials. Trainers need to select appropriate safety training materials based on their needs.

[0084] S4. Respond to the selected operation instructions of the trainees and use the virtual role to demonstrate the selected safety training materials to the trainees in a first-person perspective.

[0085] The step of responding to the trainee's selected operation command and using a virtual character to demonstrate the selected safety training materials to the trainee from a first-person perspective includes the following steps:

[0086] Receiving operation instructions: The system first needs to receive and parse the selection operation instructions issued by the trainees through the interactive terminal.

[0087] Load security training materials: The system loads the corresponding security training materials based on the parsed operation instructions.

[0088] Demonstration and training materials: The system controls a virtual character to demonstrate the content of the safety training materials from a first-person perspective.

[0089] Responding to user interactions: During the demonstration, the system needs to respond to the trainees' interactive operations in real time, such as pausing, fast forwarding, and replaying.

[0090] S5. Trainees use interactive terminals to virtually operate the training content based on the training results, and use fusion recognition algorithms to identify the trainees' status during the virtual operation process.

[0091] The process of trainees virtually operating the training content using an interactive terminal based on the training results, and identifying the trainees' state during the virtual operation using a fusion recognition algorithm combined with physiological parameters, includes the following steps:

[0092] S51. Based on the training results, trainees select virtual operation options on the interactive terminal and perform the corresponding virtual operations in the virtual environment.

[0093] S52. Use an attention-based multimodal fusion recognition algorithm combined with physiological parameters to identify the state of trainees during virtual operation.

[0094] Specifically, the method of using an attention-based multimodal fusion recognition algorithm combined with physiological parameters to identify the trainee's state during virtual operation includes the following steps:

[0095] S521. Extract the eye movement characteristics, EEG characteristics, and cross-modal interaction characteristics of trainees during virtual operation;

[0096] Specifically, the extraction of eye-tracking features, electroencephalogram (EEG) features, and cross-modal interaction features of trainees during virtual operation includes the following steps:

[0097] S5211. Collect eye movement data and electroencephalogram data of trainees during virtual operation, and perform sliding window and normalization processing.

[0098] Specifically, eye trackers are used to collect eye movement data such as pupil movement and gaze point of trainees in the virtual environment; EEG helmets are used to collect EEG signals of trainees when they operate in the virtual environment.

[0099] S5212. Extract features from EEG and eye-tracking data using the Transformer-CNN model and a Transformer-based multi-head attention mechanism, respectively, including the following steps:

[0100] The preprocessed EEG data window is input into the Transformer-CNN model. The Transformer module is responsible for learning the temporal features of the EEG data and uses a self-attention mechanism to model the long-range dependence of EEG signals. The CNN module is responsible for learning the local features of the EEG data and extracting time-frequency features through convolutional filtering. The temporal and local features learned by the Transformer and CNN are concatenated to obtain the overall feature representation of the EEG data.

[0101] The preprocessed eye-tracking data window is then input into a Transformer-based multi-head attention model. Multi-head attention allows for the simultaneous learning of features across different subspaces of the eye-tracking data. Each head learns a feature representation for one subspace, and these features are then concatenated to obtain the overall features of the eye-tracking data.

[0102] Finally, high-level feature representations of EEG and eye-tracking data are extracted using the two models mentioned above, providing information input for subsequent state recognition.

[0103] S5213. Similarity calculation is used to obtain the similarity between cross-modal data, and the weights are updated through an attention mechanism to obtain cross-modal interaction features.

[0104] The step of obtaining the similarity between cross-modal data by calculating similarity and updating the weights through an attention mechanism to obtain cross-modal interaction features includes the following steps:

[0105] Linear transformation techniques are used to map EEG and eye-tracking data into the attention space;

[0106] The formula for calculating the attention vector of the i-th data point in an EEG sample is:

[0107]

[0108] The formula for calculating the attention vector of the j-th data point in an eye-tracking sample is:

[0109]

[0110] In the formula, This represents the i-th data point of the EEG sample. W represents the j-th data point of the eye-tracking sample. q b q W k b k represents the parameters to be trained, and tanh represents the activation function;

[0111] The similarity of attention vectors between data points in EEG and eye-tracking data is calculated using dot product and then normalized.

[0112] The formula for calculating the similarity between the i-th data point of an EEG sample and the j-th data point of an eye-tracking sample is:

[0113] w i,j =q i ·k j

[0114] The formula for calculating the attention weight of the j-th data point in the eye-tracking sample to the i-th data point in the EEG sample is:

[0115]

[0116] The influence of eye-tracking data on the context of EEG data at a certain time step is calculated using attention weights to obtain the transmembrane context vector;

[0117] The formula for calculating the transmembrane context vector is:

[0118]

[0119] D = [d1 d2 ··· d T ]

[0120] In the formula, d j (j=1…T) represents the influence of the eye movement data at the j-th time step on the context of the EEG data, i.e., the transmembrane context vector, and D represents the transmembrane context similarity matrix;

[0121] The cross-modal context vector is mapped to the attention space through a linear transformation, and then normalized as the attention weights for the cross-modal context vector. The cross-modal up-down interaction features are obtained by weighting the cross-modal context vector using these attention weights, as shown in the following formula:

[0122] p j =tanh(W d d j +b d )

[0123]

[0124] C=∑ j β j d j

[0125] In the formula, p j d j The attention vector after linear transformation, β j d j The attention weights, C represents the final cross-membrane context interaction features, and W... d b d This represents the parameters to be trained.

[0126] S522. Using semantic transformation technology, eye-tracking features, EEG features, and cross-modal interaction features are mapped to the same vector space. The specific steps are as follows:

[0127] To construct a semantic transformation model, pre-trained semantic representation models such as BERT can be used.

[0128] Input eye-tracking features into a semantic transformation model to generate a semantic vector representation of the eye-tracking features.

[0129] Inputting EEG features into the same semantic transformation model generates a semantic vector representation of the EEG features.

[0130] Input cross-modal contextual interaction features into a semantic transformation model to generate its semantic vector representation.

[0131] Adjust the output vector dimension of the semantic transformation model to make the semantic vector representation dimension of the three features consistent.

[0132] The semantic vectors of eye-tracking features, EEG features, and cross-modal features are mapped to the same d-dimensional vector space.

[0133] In this d-dimensional vector space, the distance between different semantic vectors represents the semantic relevance of different modal features.

[0134] After semantic transformation, the three heterogeneous modal features are mapped to a semantically unified vector space, laying the foundation for subsequent feature fusion processing.

[0135] S523. Fuse the residual information of low-dimensional features from different periods, and classify the fused features into states to obtain the recognition results based on the fused features.

[0136] Specifically, the process of fusing residual information from low-dimensional features at different times and classifying the fused features to obtain a recognition result based on the fused features includes the following steps:

[0137] S5231. Perform deep feature fusion on the converted eye movement features, EEG features, and cross-modal up-down interaction features at different stages: early, middle, and late.

[0138] The feature fusion of the converted eye-tracking features, EEG features, and cross-modal up-down interaction features at different stages—early, middle, and late—includes the following:

[0139] In the early fusion stage, the eye-tracking data and EEG data before feature extraction are fused using the concat fusion strategy. Multi-head attention is used to assign weights to the early fused features and perform dimensional transformation.

[0140] In the intermediate fusion stage, a deep fusion strategy is used to fuse the features extracted and semantically transformed eye-tracking data and the features fused in the early fusion of the EEG data domain;

[0141] In the late fusion stage, a deep fusion strategy is used to fuse the results of the mid-term fusion with cross-modal context interaction features;

[0142] The deep fusion strategy first fuses the two inputs by directly adding them together using the fusion strategy add. Then, it selects effective features and assigns them greater weights through a multi-head attention mechanism. Finally, it adds a multilayer perceptron after the multi-head attention mechanism to deeply fuse the features by first increasing the dimensionality of the features and then decreasing the dimensionality.

[0143] S5232. The trainees' status is classified by using a fully connected layer and a Softmax activation function to fused features, and the recognition results based on fused features are obtained.

[0144] The formula for calculating state classification is as follows:

[0145] V = Softmax(FC(F) late ))

[0146] In the formula, F late The late-stage fusion features are represented by FC, Softmax represents the normalization function, and V = [v1, v2, ..., v m The figure indicates the confidence level of the EEG sample for each category.

[0147] S524. Use the physiological parameters of trainees during the virtual operation process to identify the trainees' state during the virtual operation process and obtain the identification results based on the physiological parameters.

[0148] Specifically, trainees can wear devices that detect physiological signals, such as sensors that measure heart rate and blood pressure, during virtual operations.

[0149] Specifically, the following is the correspondence between physiological parameters and emotional states in this embodiment:

[0150] When the trainee is in a relaxed emotional state, their physiological parameters (such as heart rate and blood pressure) may be relatively normal. For example, the heart rate may be 60-80 beats per minute and the blood pressure may be 120 / 80 mmHg.

[0151] The trainees' emotional state is normal, but their physiological parameters may be elevated to some extent. For example, their heart rate may be 80-100 beats per minute, and their blood pressure may be 130 / 85 mmHg.

[0152] When the emotional state is tense, the physiological parameters of trainees may be significantly elevated, for example, the heart rate may be 100-120 beats per minute, and the blood pressure may be 140 / 90 mmHg.

[0153] S525. The final state recognition result is obtained by analyzing the recognition results based on fusion features and the recognition results based on physiological parameters.

[0154] Specifically, the analysis of the recognition results based on fusion features and the recognition results based on physiological parameters to obtain the final state recognition result includes the following steps:

[0155] Compare the identification results based on fusion features and those based on physiological parameters to see if they are the same. If they are, arbitrarily select one of the two identification results as the final state identification result. If not, conduct a comprehensive analysis based on the trainee's historical performance and the difficulty of the training task to obtain the final state identification result.

[0156] Specifically, for cases where the identification results based on fusion features and physiological parameters are inconsistent, the specific steps for comprehensive analysis to obtain the final identification result are as follows:

[0157] Query the historical performance data of trainees, including previous training evaluation results, operational error rates, and other information.

[0158] Analyzing the difficulty level of the current training task reveals that the greater the difficulty, the greater the likelihood of state recognition deviation.

[0159] Compare the confidence levels of the two recognition results; the higher the confidence level, the stronger the reliability.

[0160] If trainees with a good historical track record produce inconsistent results in more challenging tasks, the recognition result with higher confidence should be prioritized.

[0161] If trainees with average historical performance show inconsistent recognition results in simple tasks, the physiological parameter recognition results should be given priority.

[0162] By constructing a matrix table based on the above rules and making comprehensive references, a more reliable identification result under the current circumstances is obtained, which is then used as the final state identification result.

[0163] At the same time, the analysis process is recorded to optimize and improve subsequent matrix table decision rules, thereby enhancing the intelligence level of the analysis.

[0164] Finally, the final identification result is fed back to the training and control system, completing one state identification and control process.

[0165] S6. Dynamically adjust the difficulty level of the training scenario according to the trainee's status until the trainee masters the safety training content corresponding to their position.

[0166] The step of dynamically adjusting the difficulty level of the training scenario based on the trainee's condition until the trainee masters the safety training content corresponding to their job position includes:

[0167] Obtain the final state recognition result of the trainee and determine whether the final recognition result is a tense state. If not, maintain the current difficulty level of the training scenario until the trainee masters the safety training content corresponding to their position. If so, reduce the difficulty level of the training scenario and repeat the training scenario at the reduced difficulty level until the trainee's state recognition result is a relaxed state. Then restore the difficulty level of the training scenario and have the virtual character provide virtual operation prompts corresponding to the training scenario at this difficulty level until the trainee's state recognition result returns to a normal or relaxed state. Finally, the trainee conducts training according to the prompts until they master the safety training content corresponding to their position.

[0168] By dynamically adjusting the difficulty of the scenario based on the trainees' status recognition results, it is possible to ensure that learners are in the optimal state of situational flow and are not overly fatigued or stressed.

[0169] S7. Analyze the training effectiveness using all operational behaviors and process data of the trainees, and optimize the safety training materials based on the training effectiveness. The specific steps are as follows:

[0170] Collect all operational behavior data of trainees in the virtual environment, including operation sequence, time, and erroneous operations.

[0171] Collect process data such as physiological parameters and emotional state of trainees in each training scenario.

[0172] The above-mentioned multi-source heterogeneous data are preprocessed to extract effective feature information.

[0173] Algorithmic models are used to analyze the correlation between operational behavior characteristics and process data characteristics.

[0174] Analyze the differences in data characteristics among different trainees and training content.

[0175] Based on the output of the above model, the learning effectiveness of the trainees is evaluated, such as their operational proficiency and level of understanding.

[0176] Compare the learning effectiveness evaluation results of different training content to identify the training content with poor results.

[0177] For content that yields poor results, adjust the arrangement of training materials, the design of virtual scenarios, and the difficulty of operation.

[0178] Repeated training sessions were conducted to verify the effectiveness of the adjusted materials until the learning outcomes of each training item met the predetermined requirements.

[0179] Continuous optimization ensures that safety training materials are constantly adapted to training needs.

[0180] According to another aspect of the present invention, a virtual reality-based interactive safety training system is provided. The system includes a virtual environment construction module, a safety training material design module, a safety training material selection module, a safety training material demonstration module, a virtual operation and status recognition module, a training scenario difficulty level dynamic adjustment module, and a training learning effect analysis module connected in sequence.

[0181] The virtual environment construction module is used to construct a virtual environment based on the actual working environment and operation process using 3D modeling software, and to set virtual characters in the virtual environment.

[0182] The safety training materials design module is used to design training scenarios and safety training materials of different difficulty levels according to training needs.

[0183] The safety training material selection module is used by trainees to enter a virtual environment through an interactive terminal and select the corresponding safety training materials according to their job requirements.

[0184] The safety training material demonstration module is used to respond to the selection operation instructions of the trainees and use virtual characters to demonstrate the selected safety training materials to the trainees in a first-person perspective.

[0185] The virtual operation and status recognition module is used by trainees to perform virtual operations of training content using an interactive terminal based on training results, and to recognize the status of trainees during the virtual operation process using a fusion recognition algorithm.

[0186] The training scenario difficulty level dynamic adjustment module is used to dynamically adjust the difficulty level of the training scenario according to the trainee's status until the trainee masters the safety training content corresponding to their job position.

[0187] The training effectiveness analysis module is used to analyze the training effectiveness using all operational behaviors and process data of trainees, and to optimize safety training materials based on the training effectiveness.

[0188] In summary, by utilizing the above-mentioned technical solutions of this invention, this invention can not only construct an immersive virtual training environment and set up humanized virtual characters, thereby enabling first-person perspective action demonstrations and voice interaction, but also use a fusion recognition algorithm to identify the emotional state of trainees during virtual operations. This allows for dynamic adjustment of the scene difficulty based on the trainees' emotional state, achieving personalized training. Furthermore, it can analyze the training learning effect based on all the trainees' operational behaviors and process data, and optimize safety training materials based on the training learning effect. This provides an effective feedback mechanism to improve the effectiveness and efficiency of training, thereby better meeting the safety training needs of enterprises.

[0189] Furthermore, this invention can not only utilize the eye movement features, electroencephalogram features, and cross-modal interaction features of trainees during virtual operation to obtain recognition results based on fusion features, but also utilize the physiological parameters of trainees during virtual operation to obtain recognition results based on physiological parameters. Thus, the recognition results based on fusion features and the recognition results based on physiological parameters can be comprehensively analyzed to obtain the final state recognition result, thereby effectively improving the accuracy of trainee state recognition.

[0190] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A virtual reality-based interactive safety training method, characterized in that, The method includes the following steps: S1. Use 3D modeling software to build a virtual environment based on the actual working environment and operation process, and set virtual characters in the virtual environment; S2. Design training scenarios and safety training materials of varying difficulty based on training needs; S3. Trainees enter the virtual environment through an interactive terminal and select the corresponding security training materials according to their job requirements; S4. Respond to the selected operation instructions of the trainees and use the virtual role to demonstrate the selected safety training materials to the trainees in a first-person perspective. S5. Trainees use interactive terminals to virtually operate the training content based on the training results, and a fusion recognition algorithm is used to identify the trainees' state during the virtual operation process; specifically including: S51. Based on the training results, trainees select virtual operation options on the interactive terminal and perform the corresponding virtual operations in the virtual environment. S52. Use an attention-based multimodal fusion recognition algorithm combined with physiological parameters to identify the state of trainees during virtual operation. The method of using an attention-based multimodal fusion recognition algorithm combined with physiological parameters to identify the trainee's state during virtual operation includes the following steps: S521. Extract the eye movement characteristics, EEG characteristics, and cross-modal interaction characteristics of trainees during virtual operation; S522. Use semantic conversion technology to map eye-tracking features, EEG features, and cross-modal up-down interaction features to the same vector space; S523. Fuse the residual information of low-dimensional features from different periods, and classify the fused features into states to obtain the recognition results based on the fused features. S524. Use the physiological parameters of trainees during the virtual operation process to identify the trainees' state during the virtual operation process and obtain the identification results based on the physiological parameters. S525. The final state recognition result is obtained by analyzing the recognition results based on fusion features and the recognition results based on physiological parameters; S6. Dynamically adjust the difficulty level of the training scenario according to the trainee's status until the trainee masters the safety training content corresponding to their position. S7. Analyze the training effectiveness using all operational behaviors and process data of trainees, and optimize safety training materials based on the training effectiveness.

2. The virtual reality-based interactive safety training method according to claim 1, characterized in that, The virtual character is used to demonstrate actions to trainees, showcasing correct and incorrect operations from a first-person perspective. It is also used to interact with trainees and provide corresponding feedback based on their actions.

3. The virtual reality-based interactive safety training method according to claim 1, characterized in that, The design of training scenarios and safety training materials of varying difficulty based on training needs includes the following steps: S21. Define the training objectives and needs, including the training topic, the target audience, and the difficulty level of the training; S22. Design training scenarios of varying difficulty based on training needs, including basic safety operations and basic equipment use for basic training, common safety issues and advanced equipment use for intermediate training, and handling of sudden safety incidents and complex safety issues for advanced training. S23. Prepare corresponding safety training materials according to the training scenarios, including detailed descriptions of different training scenarios, operating procedures and precautions.

4. The virtual reality-based interactive safety training method according to claim 1, characterized in that, The extraction of trainees' eye movement characteristics, electroencephalogram (EEG) characteristics, and cross-modal interaction characteristics during virtual operation includes the following steps: S5211. Collect eye movement data and electroencephalogram data of trainees during virtual operation, and perform sliding window and normalization processing. S5212. Features of EEG data and eye-tracking data are extracted using the Transformer-CNN model and the Transformer-based multi-head attention mechanism, respectively. S5213. Similarity calculation is used to obtain the similarity between cross-modal data, and the weights are updated through an attention mechanism to obtain cross-modal interaction features. The step of obtaining the similarity between cross-modal data by calculating similarity and updating the weights through an attention mechanism to obtain cross-modal interaction features includes the following steps: Linear transformation techniques are used to map EEG and eye-tracking data into the attention space; The similarity of attention vectors between data points in EEG and eye-tracking data is calculated using dot product and then normalized. The influence of eye-tracking data at a certain time step on the context of EEG data is calculated using attention weights to obtain the transmembrane context vector; The cross-modal context vector is mapped to the attention space through a linear transformation and normalized as the attention weight of the cross-modal context vector. The cross-modal interaction features are obtained by weighting the cross-modal context vector with the attention weight.

5. The virtual reality-based interactive safety training method according to claim 1, characterized in that, The process of fusing residual information from low-dimensional features at different time periods and classifying the fused features to obtain a recognition result based on the fused features includes the following steps: S5231. Perform feature fusion at different stages: early, middle and late stages, respectively, on the converted eye movement features, EEG features and cross-modal interaction features. S5232. The trainees' status is classified by using a fully connected layer and a Softmax activation function to fused features, resulting in a recognition result based on the fused features.

6. The virtual reality-based interactive safety training method according to claim 5, characterized in that, The feature fusion of the converted eye-tracking features, EEG features, and cross-modal up-down interaction features at different stages (early, middle, and late) includes: In the early fusion stage, the eye-tracking data and EEG data before feature extraction are fused using the concat fusion strategy. Multi-head attention is used to assign weights to the early fused features and perform dimensional transformation. In the intermediate fusion stage, a deep fusion strategy is used to fuse the features extracted and semantically transformed eye-tracking data and the features fused in the early fusion of the EEG data domain; In the late fusion stage, a deep fusion strategy is used to fuse the results of the mid-term fusion with cross-modal context interaction features; The deep fusion strategy first fuses the two inputs by directly adding them together using the fusion strategy add. Then, it selects effective features and assigns them greater weights through a multi-head attention mechanism. Finally, it adds a multilayer perceptron after the multi-head attention mechanism to deeply fuse the features by first increasing the dimensionality of the features and then decreasing the dimensionality.

7. The virtual reality-based interactive safety training method according to claim 1, characterized in that, The analysis of the recognition results based on fusion features and the recognition results based on physiological parameters to obtain the final state recognition result includes the following steps: Compare the identification results based on fusion features and those based on physiological parameters to see if they are the same. If they are, arbitrarily select one of the two identification results as the final state identification result. If not, conduct a comprehensive analysis based on the trainee's historical performance and the difficulty of the training task to obtain the final state identification result.

8. The virtual reality-based interactive safety training method according to claim 7, characterized in that, The method of dynamically adjusting the difficulty level of the training scenario based on the trainee's status until the trainee masters the safety training content corresponding to their job position includes: Obtain the final state recognition result of the trainee and determine whether the final recognition result is a tense state. If not, maintain the current difficulty level of the training scenario until the trainee masters the safety training content corresponding to their position. If so, reduce the difficulty level of the training scenario and repeat the training scenario at the reduced difficulty level until the trainee's state recognition result is a relaxed state. Then restore the difficulty level of the training scenario and have the virtual character provide virtual operation prompts corresponding to the training scenario at this difficulty level until the trainee's state recognition result returns to a normal or relaxed state. Finally, the trainee conducts training according to the prompts until they master the safety training content corresponding to their position.

9. A virtual reality-based interactive safety training system, used to implement the steps of the virtual reality-based interactive safety training method according to any one of claims 1-8, characterized in that, The system includes a virtual environment construction module, a safety training material design module, a safety training material selection module, a safety training material demonstration module, a virtual operation and status recognition module, a training scenario difficulty level dynamic adjustment module, and a training learning effect analysis module, which are connected in sequence. The virtual environment construction module is used to construct a virtual environment based on the actual working environment and operation process using 3D modeling software, and to set virtual characters in the virtual environment. The safety training materials design module is used to design training scenarios and safety training materials of different difficulty levels according to training needs. The safety training material selection module is used by trainees to enter a virtual environment through an interactive terminal and select the corresponding safety training materials according to their job requirements. The safety training material demonstration module is used to respond to the selection operation instructions of the trainees and use virtual characters to demonstrate the selected safety training materials to the trainees in a first-person perspective. The virtual operation and status recognition module is used by trainees to perform virtual operations of training content using an interactive terminal based on training results, and to recognize the status of trainees during the virtual operation process using a fusion recognition algorithm. The training scenario difficulty level dynamic adjustment module is used to dynamically adjust the difficulty level of the training scenario according to the trainee's status until the trainee masters the safety training content corresponding to their job position. The training effectiveness analysis module is used to analyze the training effectiveness using all operational behaviors and process data of trainees, and to optimize safety training materials based on the training effectiveness.

Images