Personnel training scheme evaluation method and device based on virtual reality environment, storage medium and electronic equipment

Abstract

The invention discloses a person training scheme evaluation method and device based on a virtual reality environment, a storage medium and electronic equipment, and the method comprises the steps: obtaining the brain data, physiological data and behavior data of a to-be-evaluated person in a training process based on a training scheme; performing feature fusion on the brain data, the physiological data and the behavior data to obtain fusion features of the to-be-evaluated person; obtaining a plurality of state features of the to-be-evaluated person based on the fusion features; and determining an evaluation result of the training scheme according to the scores of the plurality of state features. According to the evaluation method, the training effect can be comprehensively and accurately evaluated, the monitoring accuracy and robustness are improved, and powerful support is provided for skill training and task preparation of the to-be-evaluated personnel.

Description

Technical Field

[0001] The present application relates to the field of evaluation technology, and in particular to a personnel training program evaluation method based on a virtual reality environment, a computer-readable storage medium, an electronic device, and a personnel training program evaluation device based on a virtual reality environment. Background Art

[0002] In high-pressure and stressful environments, such as special service operations, accurate monitoring of personnel status is crucial to ensuring mission success and personnel safety. At the same time, in order to improve the performance of special service personnel in complex tasks, VR (Virtual Reality) training has gradually become an important simulation training method. Although traditional brain science equipment can provide real-time brain activity data, it is easily disturbed by motion artifacts in a dynamic interactive environment, making it difficult to fully monitor individual status. At the same time, this type of equipment is difficult to accurately reflect the functional activities of specific brain areas when monitoring individual self-control, behavioral decision-making, and task execution capabilities.

[0003] In addition, virtual reality technology combined with EEG monitoring provides testers with a highly simulated training environment that can monitor the tester's brain activity, assess their cognitive load and stress response, adjust the difficulty and content of training based on EEG feedback, and improve the personalization and effectiveness of training. This training method is particularly suitable for improving the decision-making ability and operational skills of testers in complex or high-pressure situations. However, such systems often use EEG and VR helmet-integrated devices, which can only cover brain areas such as the occipital lobe and frontal lobe. These brain areas are easily affected by electrooculography, and EEG equipment is particularly sensitive to motion artifacts, which limits the range of motion during training, and the accuracy and robustness of monitoring are low. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the first purpose of the present application is to propose a personnel training program evaluation method based on a virtual reality environment. In the process of training based on the training program, the brain data, physiological data and behavioral data of the person to be evaluated are obtained, and the brain data, physiological data and behavioral data are subjected to feature fusion to obtain the fusion feature of the person to be evaluated. Based on the fusion feature, multiple state features of the person to be evaluated are obtained, and the evaluation result of the training program is determined according to the score values ​​of the multiple state features. Thus, the training effect can be comprehensively and accurately evaluated, the accuracy and robustness of monitoring are improved, and strong support is provided for the skill training and task preparation of the person to be evaluated.

[0005] A second object of the present application is to provide a computer-readable storage medium.

[0006] The third objective of the present application is to provide an electronic device.

[0007] The fourth objective of the present application is to propose a personnel training program evaluation device based on a virtual reality environment.

[0008] To achieve the above-mentioned purpose, the first aspect of the embodiment of the present application proposes a personnel training program evaluation method based on a virtual reality environment, the method comprising: in the process of training based on the training program, obtaining brain data, physiological data and behavioral data of the person to be evaluated; performing feature fusion on the brain data, the physiological data and the behavioral data to obtain the fusion feature of the person to be evaluated; obtaining multiple state features of the person to be evaluated based on the fusion feature; and determining the evaluation result of the training program according to the scoring scores of the multiple state features.

[0009] According to the personnel training program evaluation method based on a virtual reality environment of the embodiment of the present application, in the process of training based on the training program, the brain data, physiological data and behavioral data of the person to be evaluated are obtained, and the brain data, physiological data and behavioral data are feature fused to obtain the fusion features of the person to be evaluated, and multiple state features of the person to be evaluated are obtained based on the fusion features, and the evaluation results of the training program are determined according to the scoring values ​​of the multiple state features. As a result, the method can comprehensively and accurately evaluate the training effect, improve the accuracy and robustness of monitoring, and provide strong support for the skill training and task preparation of the person to be evaluated.

[0010] In addition, the personnel training program evaluation method based on the virtual reality environment according to the above embodiment of the present application may also have the following additional technical features:

[0011] According to one embodiment of the present application, the multiple state characteristics include fatigue characteristics, load characteristics, stress characteristics and attention characteristics, and the evaluation results of the training program are determined according to the score values ​​of the multiple state characteristics, including: determining the weight coefficient of the fatigue characteristic based on the score difference between the preset fatigue characteristic score and the score value of the fatigue characteristic; determining the weight coefficient of the load characteristic based on the score difference between the preset load characteristic score and the score value of the load characteristic; determining the weight coefficient of the stress characteristic based on the score difference between the preset stress characteristic score and the score value of the stress characteristic; determining the weight coefficient of the attention characteristic based on the score difference between the preset attention characteristic score and the score value of the attention characteristic. weight coefficient; based on a first product of the score value of the fatigue feature and its corresponding weight coefficient, a second product of the score value of the load feature and its corresponding weight coefficient, a third product of the score value of the stress feature and its corresponding weight coefficient, and a fourth product of the score value of the attention feature and its corresponding weight coefficient, a comprehensive score is determined by the sum of the first product, the second product, the third product and the fourth product, wherein the weight coefficients of the fatigue feature, the load feature and the stress feature are negative numbers, and the weight coefficient of the attention feature is a positive number; an evaluation result of the training program is determined based on the comprehensive score, wherein the comprehensive score corresponds to the degree of fatigue, the degree of load, the degree of stress and the degree of attention.

[0012] According to one embodiment of the present application, the brain data, the physiological data and the behavioral data are feature fused to obtain the fused features of the person to be evaluated, including: feature fusion of the brain data, the physiological data and the behavioral data based on a pre-trained multimodal evaluation model, the training process of the multimodal evaluation model including: taking time series feature data of different modalities as input of the multimodal evaluation model, and feature preprocessing and encoding the time series feature data, wherein the multimodal time series features include brain data, physiological data and behavioral data; in the multimodal evaluation model In the CNN layer of the multimodal evaluation model, a convolutional neural network branch is designed for the time series feature data of each modality, and in the Transformer layer of the multimodal evaluation model, the CNN extracted features from each modality are spliced ​​or weightedly fused to integrate the time series feature data of different modalities; the integrated time series feature data is used as the input of the state prediction layer, and the scoring values ​​corresponding to the fatigue feature, load feature, stress feature and attention feature are predicted respectively based on the global average pooling and the fully connected layer; a plurality of the scoring values ​​are weightedly combined with their corresponding weight coefficients to obtain a comprehensive scoring value.

[0013] According to one embodiment of the present application, the method also includes: using different task parameters and task completion times in the same scenario to respectively construct scoring values ​​for stress characteristics, fatigue characteristics, load characteristics, and attention characteristics, and labeling the integrated time series feature data of different modalities according to the tasks in the training plan, assigning a label to each group of integrated time series feature data, and the label is used to associate the time series feature data with the scoring values ​​of stress characteristics, fatigue characteristics, load characteristics, and attention characteristics; based on the variance analysis method, the extracted time series feature data is screened, and the feature data with a P value greater than a preset threshold is screened as the input of the multimodal evaluation model.

[0014] According to one embodiment of the present application, the method further includes: optimizing the intensity and / or content of the training program based on the evaluation results, wherein the intensity of the training program includes the complexity of the task and the duration of the training, and the content of the training program includes skill training, simulation scenarios, and decision making.

[0015] According to one embodiment of the present application, the brain data includes hemoglobin concentration, the physiological data includes heart rate variability, and the obtaining of the brain data and physiological data of the person to be evaluated includes: obtaining the hemoglobin concentration and the heart rate variability based on functional near-infrared spectroscopy imaging technology FNIRS, wherein the hemoglobin concentration is comprehensively obtained based on light intensity data conversion, signal filtering, motion artifact processing, hemoglobin concentration calculation and feature extraction; the heart rate variability is comprehensively obtained based on light intensity data conversion, signal filtering, superposition averaging, R point detection, and feature extraction.

[0016] According to one embodiment of the present application, obtaining the behavioral data of the person to be evaluated includes: in a virtual reality VR environment, obtaining the behavioral data of the person to be evaluated in a target task training environment based on a VR device, wherein the behavioral data includes operation accuracy, response time and error rate.

[0017] To achieve the above-mentioned purpose, the second aspect of the present application proposes a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned personnel training program evaluation method based on a virtual reality environment is implemented.

[0018] According to the computer-readable storage medium of the embodiment of the present application, by implementing the above-mentioned personnel training program evaluation method based on the virtual reality environment during execution, it is possible to comprehensively and accurately evaluate the training effect, improve the accuracy and robustness of monitoring, and provide strong support for the skill training and task preparation of the personnel to be evaluated.

[0019] To achieve the above-mentioned purpose, an electronic device proposed in the third aspect embodiment of the present application includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned personnel training program evaluation method based on the virtual reality environment is implemented.

[0020] According to the electronic device of the embodiment of the present application, by executing the above-mentioned personnel training program evaluation method based on the virtual reality environment, it is possible to comprehensively and accurately evaluate the training effect, improve the accuracy and robustness of monitoring, and provide strong support for the skill training and task preparation of the personnel to be evaluated.

[0021] To achieve the above-mentioned purpose, the fourth aspect of the present application proposes a personnel training program evaluation device based on a virtual reality environment, wherein the device includes: a first acquisition module, used to obtain brain data and physiological data of the person to be evaluated during training based on the training program; a second acquisition module, used to obtain behavioral data of the person to be evaluated; a fusion module, used to perform feature fusion on the brain data, the physiological data and the behavioral data to obtain a fusion feature of the person to be evaluated; a third acquisition module, used to obtain multiple state features of the person to be evaluated based on the fusion feature; and a determination module, used to determine the evaluation result of the training program according to the scoring scores of the multiple state features.

[0022] According to the personnel training program evaluation device based on the virtual reality environment of the embodiment of the present application, the first acquisition module is used to obtain the brain data and physiological data of the person to be evaluated during the training based on the training program, the second acquisition module is used to obtain the behavioral data of the person to be evaluated, the fusion module is used to perform feature fusion on the brain data, physiological data and behavioral data to obtain the fusion feature of the person to be evaluated, the third acquisition module is used to obtain multiple state features of the person to be evaluated based on the fusion feature, and the determination module is used to determine the evaluation result of the training program according to the score values ​​of multiple state features. As a result, the device can comprehensively and accurately evaluate the training effect, improve the accuracy and robustness of monitoring, and provide strong support for the skill training and task preparation of the person to be evaluated.

[0023] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of a personnel training program evaluation method based on a virtual reality environment according to an embodiment of the present application;

[0025] Figure 2 A flowchart of a method for evaluating a personnel training program based on a virtual reality environment according to a specific example of the present application;

[0026] Figure 3 is a block diagram of an electronic device according to an embodiment of the present application;

[0027] Figure 4 It is a block diagram of a personnel training program evaluation device based on a virtual reality environment according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0029] Currently, in high-pressure and stressful environments, such as special service operations, accurate monitoring of personnel status is crucial to ensuring the success of the mission and personnel safety. At the same time, in order to improve the performance of special service personnel in complex tasks, VR training has gradually become an important simulation training method. Although traditional brain science equipment (such as EEG (Electroencephalogram)) can provide real-time brain activity data, it is easily disturbed by motion artifacts in a dynamic interactive environment, making it difficult to fully monitor individual status. At the same time, this type of equipment is difficult to accurately reflect the functional activities of specific brain areas when monitoring individual self-control, behavioral decision-making, and task execution capabilities.

[0030] Although VR technology has been widely used in simulation training scenarios, the current VR system lacks real-time monitoring and quantification of psychological and physiological states during training. Since it is impossible to obtain real-time physiological data, it is impossible to adjust or optimize the training task scenario based on feedback, which limits the personalization and optimization of the training effect. There are also solutions evaluated through VR+physiological technology, that is, some systems have tried to combine physiological data (such as heart rate, skin electrode response, etc.) to evaluate psychological and physiological states such as mental workload and fatigue, but these systems may show low sensitivity to certain dimensions (such as mental workload). At the same time, these systems often require wearing additional equipment, which not only increases the burden on users, but may also affect the naturalness and immersion of training. There are also solutions evaluated through VR+EEG technology, that is, although EEG-based technology can provide rich information about brain activity, because most of the electrodes are concentrated in areas such as the forehead and occipital lobe, and are easily affected by electrooculography and motion artifacts during training, the stability and accuracy of its data are limited. The limitations of this technology reduce its reliability and application scenarios in dynamic interactions.

[0031] That is, most of the current VR intelligent training evaluation systems focus on a single data source, while ignoring the deep integration between physiological data and brain science. Moreover, there is a lack of a comprehensive intelligent training system, and it is impossible to effectively utilize the time series data prediction algorithm. There is no predictive analysis of changes in the physiological and psychological states of the training subjects, nor is there any evaluation of the training effect based on these prediction results. More importantly, the existing system cannot automatically optimize the selection of training scenarios and the setting of training parameters based on the comprehensive training effect evaluation results, resulting in the inability to personalize and dynamically adjust the training according to the actual situation of the trainees.

[0032] In contrast, functional near-infrared spectroscopy (fNIRS) technology has strong anti-interference capabilities and can detect the activation of specific functional areas of the brain (such as the prefrontal lobe) in real time, making it a technology more suitable for dynamic training scenarios. In addition, fNIRS can also extract physiological indicators such as heart rate and breathing, and combine them with external data such as operational behavior to comprehensively evaluate the status and performance of personnel in training, providing an important reference for optimizing training programs.

[0033] To this end, this application proposes an evaluation method for training programs based on virtual reality and FNIRS (Functional Near-Infrared Spectroscopy) technology, which can monitor and evaluate personnel's brain function, peripheral physiological state and operational behavior in real time in VR training scenarios, and combine performance data in training scenarios to comprehensively evaluate training effects, help identify potential risks and optimize training strategies. Through the fusion analysis of multimodal data, the system improves the accuracy and robustness of monitoring, and provides strong support for special service personnel's skill training and mission preparation. And combining the advantages of CNN and Transformer, effectively processing and analyzing multivariate time series data, and evaluating special service personnel's fatigue, load, stress and attention status during training in real time. Based on these evaluation results, the ability spectrum of special service personnel is generated, and personalized training programs are dynamically recommended according to changes in the ability spectrum. Through this mechanism, training tasks and parameters can be dynamically adjusted according to the actual status of each special service personnel, ensuring that the training is always in the optimal load state, improving the targetedness and effectiveness of the training, and thus being able to comprehensively and accurately evaluate and optimize the training results, thereby providing more personalized and efficient support for the special service personnel's skill improvement and mission preparation.

[0034] The following describes, with reference to the accompanying drawings, a personnel training program evaluation method based on a virtual reality environment, a computer-readable storage medium, an electronic device, and a personnel training program evaluation device based on a virtual reality environment proposed in an embodiment of the present application.

[0035] Figure 1 The present invention is a flowchart of a method for evaluating a personnel training program based on a virtual reality environment according to an embodiment of the present application.

[0036] like Figure 1 As shown, the personnel training program evaluation method based on the virtual reality environment of the embodiment of the present application may include the following steps:

[0037] S1, during the training process based on the training plan, the brain data, physiological data and behavioral data of the person to be evaluated are obtained.

[0038] S2, feature fusion of brain data, physiological data and behavioral data to obtain fused features of the person to be evaluated.

[0039] S3, obtaining multiple status features of the person to be evaluated based on the fused features.

[0040] S4, determining an evaluation result of the training program according to the scoring values ​​of the multiple state characteristics.

[0041] Specifically, in the process of training based on the training program, the brain data, physiological data and behavioral data of the person to be evaluated can be obtained first. For example, the hemodynamic response of the prefrontal region can be collected in real time through the fNIRS sensor embedded in the VR helmet. These data may include changes in oxygenated hemoglobin (HbO, Oxyhemoglobin), deoxyhemoglobin (HbR, Deoxyhemoglobin) and total hemoglobin concentration (HbT, Total Hemoglobin). Physiological data, including heart rate, respiratory rate and rhythm, can be collected through a single near-infrared device to reduce the use of other additional equipment, such as reducing the use of heart rate variability monitoring equipment and respiratory monitoring equipment. Behavioral data, such as operation accuracy, response time and error rate, can be collected through sensors and tracking devices in the VR system. In addition, it is necessary to ensure that brain data, physiological data and behavioral data are aligned at the same time step.

[0042] After obtaining the brain data, physiological data and behavioral data of the person to be evaluated, the brain data, physiological data and behavioral data can be fused to obtain the fused features of the person to be evaluated. For example, the collected raw data is cleaned, standardized and missing values ​​are processed to ensure data quality. The optical density signal is extracted from the brain data, and the HRV and respiratory variability (RESP) indicators are calculated; HRV features are extracted from the physiological data; and operation performance features are extracted from the behavioral data. The features of the near-infrared signal, the HRV features and the behavioral operation performance features are fused at the feature layer, and the same task parameters and task completion time are used to construct multimodal features of stress, fatigue, load and attention. Therefore, multiple state features of the person to be evaluated can be obtained based on the fused features. For example, a deep learning model is used, combined with the advantages of CNN and Transformer layers, to process and analyze the fused features, and the fatigue, load, stress and attention states of special agents are evaluated in real time, that is, the model output may include the prediction results of multiple states such as fatigue, load, stress and attention, and each state prediction uses an independent output node and a corresponding activation function.

[0043] Finally, the evaluation results of the training program can be determined according to the scores of multiple state characteristics. For example, the scores of multiple state characteristics are predefined, such as preset thresholds of fatigue characteristics, load characteristics, stress characteristics, and attention characteristics, and a comprehensive scoring system is formulated for overall state evaluation. The predicted scores of multiple state characteristics are compared with the preset thresholds to determine the effectiveness of the training program, and the training intensity or content is dynamically adjusted according to the evaluation results. For example, when the predicted score of the fatigue characteristic is lower than the preset fatigue characteristic threshold, the predicted score of the load characteristic is lower than the preset load characteristic threshold, the predicted score of the stress characteristic is lower than the preset stress characteristic threshold, and the predicted score of the attention characteristic is higher than the preset attention characteristic threshold, it can be said that the current load, stress, and fatigue levels are low and the attention level is high, indicating that the current training program is effective and suitable for the person to be evaluated to train.

[0044] Therefore, by real-time monitoring and evaluation of the brain function, peripheral physiological state and operational behavior of the persons to be evaluated, and combining the performance data of the persons to be evaluated in the training scenario, it is possible to comprehensively and accurately evaluate the training effect, improve the accuracy and robustness of monitoring, and provide strong support for the skill training and task preparation of the persons to be evaluated.

[0045] According to one embodiment of the present application, multiple state characteristics include fatigue characteristics, load characteristics, stress characteristics and attention characteristics, and the evaluation results of the training program are determined according to the score values ​​of the multiple state characteristics, including: determining the weight coefficient of the fatigue characteristic based on the score difference between the preset fatigue characteristic score and the fatigue characteristic score; determining the weight coefficient of the load characteristic based on the score difference between the preset load characteristic score and the load characteristic score; determining the weight coefficient of the stress characteristic based on the score difference between the preset stress characteristic score and the stress characteristic score; determining the weight coefficient of the attention characteristic based on the score difference between the preset attention characteristic score and the attention characteristic score. The weight coefficient of the characteristic; based on the first product of the score value of the fatigue characteristic and the corresponding weight coefficient, the second product of the score value of the load characteristic and the corresponding weight coefficient, the third product of the score value of the stress characteristic and the corresponding weight coefficient, and the fourth product of the score value of the attention characteristic and the corresponding weight coefficient, the comprehensive score is determined by the sum of the first product, the second product, the third product and the fourth product, wherein the weight coefficients of the fatigue characteristic, the load characteristic and the stress characteristic are negative numbers, and the weight coefficient of the attention characteristic is a positive number; the evaluation result of the training program is determined based on the comprehensive score, wherein the comprehensive score corresponds to the degree of fatigue, the degree of load, the degree of stress and the degree of attention.

[0046] Specifically, the multiple state characteristics may include fatigue characteristics, load characteristics, stress characteristics and attention characteristics. When determining the evaluation results of the training program according to the scores of the multiple state characteristics, for each state characteristic (fatigue, load, stress, attention), the weight coefficient of each characteristic may be determined according to the difference between the predicted actual characteristic score and the preset score. That is, the difference between the preset fatigue characteristic score and the actual fatigue characteristic score is calculated to determine the weight coefficient of the fatigue characteristic, the difference between the preset load characteristic score and the actual load characteristic score is calculated to determine the weight coefficient of the load characteristic, the difference between the preset stress characteristic score and the actual stress characteristic score is calculated to determine the weight coefficient of the stress characteristic, and the difference between the preset attention characteristic score and the actual attention characteristic score is calculated to determine the weight coefficient of the attention characteristic.

[0047] The weight coefficients of fatigue, load and stress characteristics are negative, which means that the higher the score of these characteristics, the more negative their contribution to the comprehensive score, that is, the increase of these characteristic scores will lead to a decrease in the comprehensive score. The weight coefficient of the attention characteristic is positive, which means that the higher the score of the attention characteristic, the more positive its contribution to the comprehensive score, that is, the increase of the attention characteristic score will lead to an increase in the comprehensive score. That is, the comprehensive score can be obtained by multiplying the score of each state characteristic by its corresponding weight coefficient, and then adding these products. The specific calculation formula is: Comprehensive score = (fatigue characteristic score × fatigue weight coefficient) + (load characteristic score × load weight coefficient) + (stress characteristic score × stress weight coefficient) + (attention characteristic score × attention weight coefficient).

[0048] Therefore, the effect of the training program can be evaluated according to the calculated comprehensive score. The comprehensive score corresponds to the fatigue level, load level, stress level and attention level, that is, the comprehensive score reflects the comprehensive influence of the fatigue level, load level, stress level and attention level of the special service personnel in training. For example, a high comprehensive score can reflect that the current attention level of the personnel to be evaluated is high, and the fatigue, load and stress levels are low, which indicates that the training program is effective and the special service personnel are in a good training state. If the comprehensive score is not ideal, it can reflect that the current attention level of the personnel to be evaluated is low, and the fatigue, load and stress levels are high. It can be recommended to adjust the training program to optimize the training effect. Therefore, it includes the judgment of the fatigue, load, stress, concentration and other states of the central nervous system, that is, it can fully cover the physiological and cognitive states of the personnel to be evaluated, and can capture the distraction, excessive cognitive load or fatigue state in time, provide a scientific basis for optimizing the design of training tasks, ensure the personalization and effectiveness of training, so as to comprehensively and accurately evaluate and optimize the training effect, and provide more personalized and efficient support for the skill improvement and task preparation of the personnel to be evaluated.

[0049] According to one embodiment of the present application, feature fusion is performed on brain data, physiological data and behavioral data to obtain fused features of the person to be evaluated, including: feature fusion of brain data, physiological data and behavioral data based on a pre-trained multimodal evaluation model, and the training process of the multimodal evaluation model includes: taking the time series feature data of different modalities as the input of the multimodal evaluation model, and performing feature preprocessing and encoding on the time series feature data, wherein the multimodal time series features include brain data, physiological data and behavioral data; in the CNN layer of the multimodal evaluation model, a convolutional neural network branch is designed for the time series feature data of each modality, and in the Transformer layer of the multimodal evaluation model, the CNN extracted features from each modality are spliced ​​or weightedly fused to integrate the time series feature data of different modalities; the integrated time series feature data is used as the input of the state prediction layer, and the score values ​​corresponding to the fatigue feature, load feature, stress feature and attention feature are predicted respectively based on the global average pooling and the fully connected layer; multiple score values ​​are weightedly combined with their corresponding weight coefficients to obtain a comprehensive score value.

[0050] Specifically, when the brain data, physiological data and behavioral data are feature fused to obtain the fused features of the person to be evaluated, the time series features of the brain data, physiological data and behavioral data can be used as the input of the multimodal evaluation model. Before entering the model, the time series feature data is feature preprocessed. This may include steps such as data cleaning, denoising, and normalization to improve data quality and model training effects. Feature encoding is to map the time series feature data to a unified feature space. Fully connected layers and position encoding techniques may be used to retain time series information, such as using sine-cosine position encoding to retain time series information. Then, in the CNN layer of the multimodal evaluation model, an independent 1D convolutional neural network branch is designed for each modality. Each branch first extracts local time series features through a series of convolutional layers (e.g., two Conv1D layers, each followed by batch normalization and maximum pooling). These features can capture short-term patterns and dynamic changes in each modality data. Subsequently, a global average pooling layer is used to compress the convolution output into a fixed-dimensional feature vector, ensuring that the features of different modalities have consistent dimensions and structures in the subsequent fusion stage. This independent processing of each modality not only retains the characteristics of each modality, but also lays a solid foundation for the effective fusion of multimodal features and subsequent long-term dependency modeling.

[0051] In the Transformer layer of the multimodal evaluation model, the CNN extracted features from each modality are concatenated or weighted fused to integrate multimodal information. Subsequently, sine-cosine position encoding is added to retain temporal order information. The fused feature vector is input into multiple stacked Transformer encoder layers, each of which contains a multi-head self-attention mechanism and a feedforward neural network. The training process is stabilized by residual connections and layer normalization. The self-attention mechanism enables the model to capture long-distance dependencies between different time steps and enhance the understanding of complex temporal patterns, thereby effectively modeling long-term dependencies in multimodal data and improving the accuracy and robustness of state recognition. Therefore, according to the global features output by the Transformer layer, multiple states such as fatigue, load, stress and attention can be predicted respectively through global average pooling and fully connected layers. Each state prediction uses an independent output node and corresponding activation function to achieve multi-task learning and improve the accuracy and generalization of the prediction, that is, to predict the score corresponding to the fatigue feature, load feature, stress feature and attention feature respectively. The prediction results of each state are weighted and combined using preset weights to generate a comprehensive evaluation index that comprehensively reflects the individual's overall state. The weights of stress, load, and fatigue are negative, and the higher the level, the lower the score. The weight of attention level is positive, and the higher the level, the higher the comprehensive score. The weight of each indicator under a certain task is determined by expert evaluation.

[0052] In addition, model parameters and computational complexity can be reduced through lightweight technologies such as model pruning, quantization, and architecture optimization to ensure efficient and low-latency real-time prediction on the server side. In this way, multimodal data can be effectively integrated to provide comprehensive evaluation results, providing a scientific basis for optimizing the training program for the person to be evaluated. This method has important application value in multimodal learning and can improve the robustness and accuracy of the model.

[0053] According to one embodiment of the present application, a personnel training program evaluation method based on a virtual reality environment also includes: using different task parameters and task completion times in the same scene to respectively construct scoring values ​​for stress characteristics, fatigue characteristics, load characteristics, and attention characteristics, and labeling the integrated time series feature data of different modalities according to the tasks in the training program, assigning a label to each group of integrated time series feature data, and the label is used to associate the time series feature data with the scoring values ​​of stress characteristics, fatigue characteristics, load characteristics, and attention characteristics; based on the variance analysis method, the extracted time series feature data is screened, and the feature data with a P value greater than a preset threshold is screened as the input of the multimodal evaluation model.

[0054] Specifically, in the same VR training scenario, different training tasks are constructed by changing the parameters of the task (such as difficulty, complexity, etc.) and the time to complete the task. For each training task, scoring scores for stress characteristics, fatigue characteristics, load characteristics, and attention characteristics are constructed. These scoring scores reflect the psychological and physiological state of special agents when completing specific tasks. Integrate time series feature data from different modalities (such as physiological data, brain function data, and behavioral performance data), and assign a label to each set of integrated time series feature data. This label associates the time series feature data with the corresponding stress, fatigue, load, and attention scoring scores. That is, such annotations help to match specific psychological and physiological states with specific training tasks in subsequent analysis.

[0055] For example, based on the task design, the fused data is labeled, and a label is assigned to each sample to indicate its corresponding state, namely, stress labels: low stress, medium stress, and high stress. Fatigue labels: low fatigue, medium fatigue, and high fatigue. Load labels: low load, medium load, and high load. Attention labels: low attention, medium attention, and high attention. For example, if a person to be evaluated performs a high-stress task (such as defusing a bomb in the presence of hostile actions and civilian interference) and collects multimodal data in the process, then these data will be labeled as high stress. And the low, medium, and high levels can correspond to corresponding scoring scores respectively. In this way, a multimodal dataset containing different stresses, fatigue, loads, and attention can be constructed, and corresponding labels can be assigned to each sample. These data and labels can be used to train deep learning models to predict the psychological and physiological states of the person to be evaluated when performing different tasks.

[0056] The extracted time series feature data can be screened using the analysis of variance (ANOVA) method. ANOVA is a statistical method used to analyze whether there are significant differences in the means between different groups. Feature data with a P value greater than a preset threshold are screened out. The P value is used in statistics to measure the probability of the occurrence of a result by chance. If the P value is greater than the preset threshold (such as 0.05), the feature is considered to be statistically insignificant and can be excluded from the model input. The screened feature data, that is, those features with a P value greater than the preset threshold, will be used as the input of the multimodal evaluation model. These feature data can reflect the key psychological and physiological states of the special service personnel in training, and have statistical significance for the evaluation of the training effect. Therefore, the most helpful features for model prediction can be identified and selected, and the multimodal evaluation model can be trained using the screened feature data so that the model can learn how to predict the psychological and physiological states of the special service personnel based on the time series feature data. In actual training, the model will receive new time series feature data in real time and predict the state of the special service personnel, thereby providing a basis for adjusting the training program.

[0057] According to one embodiment of the present application, the personnel training program evaluation method based on the virtual reality environment also includes: optimizing the intensity and / or content of the training program based on the evaluation results, wherein the intensity of the training program includes the complexity of the task and the duration of the training, and the content of the training program includes skill training, simulation scenarios and decision making.

[0058] Specifically, after evaluating the performance of the person to be evaluated in a specific skill training task, such as operating skills, reaction speed and accuracy, and evaluating the decision-making ability of the person to be evaluated in complex or stressful situations, the evaluation results are obtained. After obtaining the evaluation results, the effectiveness of the current training program can also be determined based on the evaluation results, and it can be determined whether the training program needs to be adjusted. For example, the intensity of the training program can be optimized based on the evaluation results. Among them, the intensity of the training program includes the complexity of the task and the duration of the training. For example, the complexity of the task can be adjusted based on the brain and physiological data of the person to be evaluated, such as increasing the requirements for multitasking or reducing the interference factors of the task, or according to the evaluation results, a task type that is more suitable for the current ability level of the person to be evaluated can be selected. It is also possible to appropriately extend or shorten the duration of the training, or introduce intermittent training, based on the fatigue and concentration of the person to be evaluated, to prevent excessive fatigue and improve training efficiency.

[0059] The content of the training program can also be optimized based on the evaluation results. The content of the training program includes skill training, simulation scenarios, and decision making. For example, based on the performance of the person to be evaluated in a specific skill, the training of related skills can be strengthened or supplemented. Alternatively, simulation scenarios can be updated or added to better simulate the challenges that may be encountered in a real environment. More tasks involving complex decisions can also be designed to improve the decision-making ability of the person to be evaluated.

[0060] The content and intensity of the training program can also be optimized simultaneously based on the evaluation results, thereby ensuring that the training program always matches the actual capabilities and needs of the people to be evaluated, thereby improving the effectiveness and pertinence of the training.

[0061] In other words, based on the results of multimodal evaluation model training, the model can generate instant prediction results of each state, fit the training curve, compare the prediction results with the preset threshold, and calculate the comprehensive index score to evaluate the overall state, and judge whether the comprehensive score shows low load, stress and fatigue levels and high attention levels, etc., and determine the effectiveness of the current training program based on the evaluation results, and judge whether adjustments are needed, so that the training intensity or content can be automatically or manually adjusted according to the evaluation results to achieve adaptive optimization of the training program. In addition, the status of the person to be evaluated can be continuously monitored, and the training program can be iterated in a cycle to ensure the personalization and effectiveness of the training, so that training parameters and training scenarios can be adaptively recommended to the person to be evaluated.

[0062] According to one embodiment of the present application, brain data includes hemoglobin concentration, and physiological data includes heart rate variability. The brain data and physiological data of the person to be evaluated are obtained, including: obtaining hemoglobin concentration and heart rate variability based on functional near-infrared spectroscopy imaging technology FNIRS, wherein the hemoglobin concentration is comprehensively obtained based on light intensity data conversion, signal filtering, motion artifact processing, hemoglobin concentration calculation and feature extraction; heart rate variability is comprehensively obtained based on light intensity data conversion, signal filtering, superposition averaging, R point detection, and feature extraction.

[0063] Specifically, brain data may include hemoglobin concentration, and physiological data may include heart rate variability. When obtaining brain data and physiological data of the person to be evaluated, hemoglobin concentration and heart rate variability may be obtained according to functional near-infrared spectroscopy imaging technology FNIRS. fNIRS measures hemoglobin concentration by emitting near-infrared light and detecting its scattering and absorption in tissues. When obtaining hemoglobin concentration, it can be obtained comprehensively according to light intensity data conversion, signal filtering, motion artifact processing, hemoglobin concentration calculation and feature extraction. That is, light intensity data conversion refers to converting the original light intensity signal into an optical density signal to reflect the changes in oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR). The intensity changes of light of different wavelengths after passing through brain tissue can be recorded, and these light intensity changes are converted into optical density values ​​using Beer-Lambert's law. Signal filtering can remove noise and interference in the signal, such as low-frequency noise and high-frequency interference. For example, the optical density signal is subjected to a 0.01-0.2Hz bandpass filter to remove low-frequency noise and high-frequency interference, and signals such as breathing, blood pressure pulse wave and heartbeat. Motion artifact processing can identify and process artifacts caused by head movement. For example, a motion artifact detection algorithm can be used to identify and mark artifacts, and spline interpolation smoothing can be used to fill in missing or abnormal parts of the signal. The hemoglobin concentration is calculated based on the modified Beer-Lambert law. The concentration changes of oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) are calculated through the filtered signal, and the total hemoglobin concentration (HbT) is combined. Finally, feature extraction is performed, that is, the hemoglobin concentration change features related to neural activity are extracted, and the HbO, HbR and HbT concentration change data of the prefrontal region are output to evaluate the neural activity status of the person.

[0064] In addition, every time the heart beats, it pushes blood through various parts of the body, including the brain. The brain is supplied with oxygen and nutrients through blood flow, while taking away metabolic waste. Therefore, the blood fluctuations caused by the heartbeat cause slight pressure fluctuations in the blood vessels, especially in the blood vessels of the brain. These fluctuations will affect the blood oxygen saturation and hemoglobin concentration, which will be reflected in the near-infrared light signal. Therefore, the heart rate variability characteristics can also be extracted based on the near-infrared signal. Therefore, the fNIRS sensing system can be embedded in the VR helmet to collect the hemodynamic response of the frontal lobe area in real time, ensure the optimization of the equipment structure and lightweight design, adapt to the use requirements of the VR helmet, and do not affect the user's interactive experience.

[0065] When obtaining heart rate variability, it can be obtained comprehensively according to light intensity data conversion, signal filtering, superposition averaging, R point detection, and feature extraction. The original light intensity signal can be converted into optical density data of red light and infrared light. Signal filtering can extract signals related to changes in heart rhythm, for example. The signal is filtered at 0.5-4.5Hz to obtain a PPG (Photoplethysmography) signal. Superposition averaging can improve the signal-to-noise ratio of the heartbeat waveform, such as superimposing and averaging the data of all channels to obtain a clearer heartbeat waveform. When detecting the R point, a 200ms sliding window is used in combination with the local maximum method to gradually calculate the signal peak in each window, and the amplitude difference between adjacent peaks is set to at least 20% to exclude noise or pseudo-peaks, and finally identify the true R point position (the R point is the peak point in the electrocardiogram, representing the beginning of the heartbeat).

[0066] Feature extraction can analyze PPG signals in detail from three aspects: time domain, frequency domain and nonlinear analysis. Among them, time domain features mainly reflect the rhythm of the heart and the regulatory ability of the autonomic nervous system by calculating RR intervals (heartbeat intervals), heart rate and heart rate variability (such as SDNN (Standard Deviation of the NN intervals, standard deviation NN intervals), RMSSD (Root Mean Square of the Successive Differences, the root mean square of the difference between adjacent NN intervals), etc. RMSSD is sensitive to rapidly changing heart rate variability). Frequency domain features extract low frequency (LF), high frequency (HF) and LF / HF ratio through power spectrum analysis to reveal the relative activity levels of sympathetic and parasympathetic nerves. Nonlinear features analyze the complexity and regularity of signals through indicators such as approximate entropy (ApEn), sample entropy (SampEn) and fractal dimension (FD), providing deeper physiological state information than traditional linear methods.

[0067] In this way, the brain data (hemoglobin concentration) and physiological data (heart rate variability) of the person to be evaluated can be comprehensively obtained, providing a scientific basis for further analysis and evaluation.

[0068] According to one embodiment of the present application, obtaining behavioral data of the person to be evaluated includes: in a virtual reality VR environment, obtaining the behavioral data of the person to be evaluated in a target task training environment based on a VR device, wherein the behavioral data includes operation accuracy, response time, and error rate.

[0069] Specifically, when obtaining the behavioral data of the person to be evaluated, the behavioral data of the person to be evaluated in the target task training environment can be obtained based on the VR device in the virtual reality VR environment, for example, using the sensors built into the VR helmet and handle to track the movement of the head and hands, and record the precise position and action. Or use the game or simulation engine in the VR environment to automatically record the operation data, such as the accuracy and response time of shooting. Among them, the behavioral data includes operation accuracy, response time and error rate. Operation accuracy refers to the accuracy and meticulousness of the person to be evaluated when performing the task. For example, operation accuracy may include positioning accuracy, such as using VR equipment to track the movement and position of the person to be evaluated, and record the accuracy of its operation, for example, whether the person to be evaluated can accurately hit the target or place the object in the correct position. Operation accuracy may also include task completion, such as evaluating whether the person to be evaluated can complete the task according to the predetermined steps and sequence, and the quality of completion. Operation accuracy may also include target achievement rate, that is, recording the number of times the person to be evaluated achieves the target within a specific time, for example, the number of times the target is hit in a shooting game.

[0070] Response time refers to the time required for the person to be evaluated to respond to a specific stimulus or task requirement, such as reaction initiation time: the time from the start of the task to the time the person to be evaluated starts to respond, decision time: the time the person to be evaluated spends before making a decision, for example, the time to choose the correct action plan in a complex scenario, action execution time: the time required from making a decision to completing an action, for example, the time from aiming to shooting. Error rate refers to the frequency and type of errors made by the person to be evaluated during the execution of the task, such as operation errors: record the errors made by the person to be evaluated during the operation, such as wrong operation, missing steps or wrong judgment, and task failure times: count the number of times the person to be evaluated failed to complete the task.

[0071] In this way, the behavioral performance of the person to be evaluated in the VR environment can be comprehensively evaluated, providing a scientific basis for further training and evaluation.

[0072] Combine the following Figure 2 To describe the method of the present application.

[0073] As a specific example, the personnel training program evaluation method based on a virtual reality environment of the present application may include the following steps:

[0074] S101, during the training process based on the training plan, the hemoglobin concentration and heart rate variability of the assessee are obtained based on the functional near-infrared spectroscopy imaging technology FNIRS, and in a virtual reality VR environment, the behavioral data of the person to be assessed in the target task training environment is obtained based on the VR device, wherein the behavioral data includes operation accuracy, response time and error rate.

[0075] S102, based on the pre-trained multimodal assessment model, feature fusion is performed on the brain data, physiological data and behavioral data to obtain fused features of the person to be assessed.

[0076] S103, obtaining multiple status features of the person to be evaluated based on the fused features.

[0077] S104, determining the weight coefficient of the fatigue characteristic based on the score difference between the preset fatigue characteristic score and the score of the fatigue characteristic, determining the weight coefficient of the load characteristic based on the score difference between the preset load characteristic score and the score of the load characteristic, determining the weight coefficient of the stress characteristic based on the score difference between the preset stress characteristic score and the score of the stress characteristic, and determining the weight coefficient of the attention characteristic based on the score difference between the preset attention characteristic score and the score of the attention characteristic.

[0078] S105, based on the first product of the score value of the fatigue characteristic and its corresponding weight coefficient, the second product of the score value of the load characteristic and its corresponding weight coefficient, the third product of the score value of the stress characteristic and its corresponding weight coefficient, and the fourth product of the score value of the attention characteristic and its corresponding weight coefficient, the comprehensive score is determined by the sum of the first product, the second product, the third product and the fourth product.

[0079] S106, determining an evaluation result of the training program based on the comprehensive scoring value, wherein the comprehensive scoring value corresponds to the fatigue level, load level, stress level and attention level.

[0080] S107, optimizing the intensity and / or content of the training program based on the evaluation results, wherein the intensity of the training program includes the complexity of the task and the duration of the training, and the content of the training program includes skill training, simulation scenarios, and decision making.

[0081] In summary, according to the personnel training program evaluation method based on the virtual reality environment of the embodiment of the present application, in the process of training based on the training program, the brain data, physiological data and behavioral data of the person to be evaluated are obtained, and the brain data, physiological data and behavioral data are feature fused to obtain the fusion features of the person to be evaluated, and multiple state features of the person to be evaluated are obtained based on the fusion features, and the evaluation results of the training program are determined according to the scoring values ​​of the multiple state features. Therefore, this method can comprehensively and accurately evaluate the training effect, improve the accuracy and robustness of monitoring, and provide strong support for the skill training and task preparation of the person to be evaluated.

[0082] Corresponding to the above embodiments, the present application also proposes a computer-readable storage medium.

[0083] The computer-readable storage medium of the embodiment of the present application stores a program thereon, and when the program is executed by a processor, the above-mentioned personnel training program evaluation method based on a virtual reality environment is implemented.

[0084] According to the computer-readable storage medium of the embodiment of the present application, by executing the above-mentioned personnel training program evaluation method based on the virtual reality environment, the training effect can be comprehensively and accurately evaluated, the accuracy and robustness of monitoring are improved, and strong support is provided for the skill training and task preparation of the personnel to be evaluated.

[0085] Corresponding to the above embodiment, the present application also proposes an electronic device.

[0086] like Figure 3 As shown, the electronic device 200 of the embodiment of the present application may include: a memory 210, a processor 220, and a program stored in the memory 210 and executable on the processor 220. When the processor 220 executes the program, the above-mentioned personnel training program evaluation method based on the virtual reality environment is implemented.

[0087] According to the electronic device of the embodiment of the present application, by executing the above-mentioned personnel training program evaluation method based on the virtual reality environment, it is possible to comprehensively and accurately evaluate the training effect, improve the accuracy and robustness of monitoring, and provide strong support for the skill training and task preparation of the personnel to be evaluated.

[0088] Corresponding to the above-mentioned embodiment, the present application also proposes a personnel training program evaluation device based on a virtual reality environment.

[0089] like Figure 4 As shown, the personnel training program evaluation device 100 based on the virtual reality environment of the embodiment of the present application includes: a first acquisition module 110, a second acquisition module 120, a fusion module 130, a third acquisition module 140 and a determination module 150.

[0090] Among them, the first acquisition module 110 is used to obtain brain data and physiological data of the person to be evaluated during the training process based on the training program. The second acquisition module 120 is used to obtain behavioral data of the person to be evaluated. The fusion module 130 is used to perform feature fusion on brain data, physiological data and behavioral data to obtain fusion features of the person to be evaluated. The third acquisition module 140 is used to obtain multiple state features of the person to be evaluated based on the fusion features. The determination module 150 is used to determine the evaluation result of the training program according to the scoring scores of multiple state features.

[0091] According to one embodiment of the present application, multiple state characteristics include fatigue characteristics, load characteristics, stress characteristics and attention characteristics. The determination module 150 determines the evaluation result of the training program according to the score values ​​of the multiple state characteristics, and is specifically used to: determine the weight coefficient of the fatigue characteristic based on the score difference between the preset fatigue characteristic score and the fatigue characteristic score; determine the weight coefficient of the load characteristic based on the score difference between the preset load characteristic score and the load characteristic score; determine the weight coefficient of the stress characteristic based on the score difference between the preset stress characteristic score and the stress characteristic score; determine the weight coefficient of the attention characteristic based on the score difference between the preset attention characteristic score and the attention characteristic score. Determine the weight coefficient of the attention feature; based on the first product of the score value of the fatigue feature and the corresponding weight coefficient, the second product of the score value of the load feature and the corresponding weight coefficient, the third product of the score value of the stress feature and the corresponding weight coefficient, and the fourth product of the score value of the attention feature and the corresponding weight coefficient, the sum of the first product, the second product, the third product and the fourth product determines the comprehensive score, wherein the weight coefficients of the fatigue feature, the load feature and the stress feature are negative numbers, and the weight coefficient of the attention feature is a positive number; determine the evaluation result of the training program based on the comprehensive score, wherein the comprehensive score corresponds to the degree of fatigue, the degree of load, the degree of stress and the degree of attention.

[0092] According to one embodiment of the present application, the fusion module 130 performs feature fusion on brain data, physiological data and behavioral data to obtain fusion features of the person to be evaluated, which is specifically used for: performing feature fusion on brain data, physiological data and behavioral data based on a pre-trained multimodal evaluation model, and the training process of the multimodal evaluation model includes: taking the time series feature data of different modalities as the input of the multimodal evaluation model, and performing feature preprocessing and encoding on the time series feature data, wherein the multimodal time series features include brain data, physiological data and behavioral data; in the CNN layer of the multimodal evaluation model, a convolutional neural network branch is designed for the time series feature data of each modality, and in the Transformer layer of the multimodal evaluation model, the CNN extracted features from each modality are spliced ​​or weightedly fused to integrate the time series feature data of different modalities; the integrated time series feature data is used as the input of the state prediction layer, and the score values ​​corresponding to the fatigue feature, load feature, stress feature and attention feature are predicted respectively based on the global average pooling and the fully connected layer; multiple score values ​​are weightedly combined with their corresponding weight coefficients to obtain a comprehensive score value.

[0093] According to one embodiment of the present application, the fusion module 130 is also used to: use different task parameters and task completion times in the same scenario to respectively construct scoring values ​​for stress characteristics, fatigue characteristics, load characteristics, and attention characteristics, and label the integrated time series feature data of different modalities according to the tasks in the training plan, and assign a label to each group of integrated time series feature data, and the label is used to associate the time series feature data with the scoring values ​​of stress characteristics, fatigue characteristics, load characteristics, and attention characteristics; based on the variance analysis method, the extracted time series feature data is screened, and the feature data with a P value greater than a preset threshold is screened as the input of the multimodal evaluation model.

[0094] According to one embodiment of the present application, the determination module 150 is also used to: optimize the intensity and / or content of the training program based on the evaluation results, wherein the intensity of the training program includes the complexity of the task and the duration of the training, and the content of the training program includes skill training, simulation scenarios, and decision making.

[0095] According to one embodiment of the present application, brain data includes hemoglobin concentration, and physiological data includes heart rate variability. The first acquisition module 110 acquires brain data and physiological data of the person to be evaluated, and is specifically used for: acquiring hemoglobin concentration and heart rate variability based on functional near-infrared spectral imaging technology FNIRS, wherein the hemoglobin concentration is comprehensively acquired based on light intensity data conversion, signal filtering, motion artifact processing, hemoglobin concentration calculation and feature extraction; heart rate variability is comprehensively acquired based on light intensity data conversion, signal filtering, superposition averaging, R point detection, and feature extraction.

[0096] According to one embodiment of the present application, the second acquisition module 120 acquires the behavioral data of the person to be evaluated, and is specifically used for: in a virtual reality VR environment, acquiring the behavioral data of the person to be evaluated in a target task training environment based on a VR device, wherein the behavioral data includes operation accuracy, response time and error rate.

[0097] It should be noted that for details not disclosed in the personnel training program evaluation device based on a virtual reality environment in the embodiment of the present application, please refer to the details disclosed in the personnel training program evaluation method based on a virtual reality environment in the embodiment of the present application, and the details will not be repeated here.

[0098] According to the personnel training program evaluation device based on the virtual reality environment of the embodiment of the present application, the first acquisition module is used to obtain the brain data and physiological data of the person to be evaluated during the training based on the training program, the second acquisition module is used to obtain the behavioral data of the person to be evaluated, the fusion module is used to perform feature fusion on the brain data, physiological data and behavioral data to obtain the fusion feature of the person to be evaluated, the third acquisition module is used to obtain multiple state features of the person to be evaluated based on the fusion feature, and the determination module is used to determine the evaluation result of the training program according to the score values ​​of multiple state features. As a result, the device can comprehensively and accurately evaluate the training effect, improve the accuracy and robustness of monitoring, and provide strong support for the skill training and task preparation of the person to be evaluated.

[0099] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0100] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0101] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0102] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0103] In this application, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0104] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A personnel training program evaluation method based on a virtual reality environment, characterized in that: The method comprises: During the training process based on the training plan, brain data, physiological data and behavioral data of the person to be evaluated are obtained; Performing feature fusion on the brain data, the physiological data, and the behavioral data to obtain fusion features of the person to be evaluated; Obtaining multiple status features of the person to be evaluated based on the fusion features; An evaluation result of the training program is determined according to the scoring values ​​of the multiple status characteristics.

2. The personnel training program evaluation method based on virtual reality environment according to claim 1 is characterized in that: The multiple state characteristics include fatigue characteristics, load characteristics, stress characteristics and attention characteristics, and determining the evaluation result of the training program according to the scoring values ​​of the multiple state characteristics includes: Determining a weight coefficient of the fatigue feature based on a score difference between a preset fatigue feature score and the score of the fatigue feature; Determining a weight coefficient of the load characteristic based on a score difference between a preset load characteristic score and the score of the load characteristic; Determining a weight coefficient of the stress feature based on a score difference between a preset stress feature score and the score of the stress feature; Determining a weight coefficient of the attention feature based on a score difference between a preset attention feature score and the score of the attention feature; Based on a first product of the score of the fatigue feature and its corresponding weight coefficient, a second product of the score of the load feature and its corresponding weight coefficient, a third product of the score of the stress feature and its corresponding weight coefficient, and a fourth product of the score of the attention feature and its corresponding weight coefficient, a comprehensive score is determined by the sum of the first product, the second product, the third product and the fourth product, wherein the weight coefficients of the fatigue feature, the load feature and the stress feature are negative numbers, and the weight coefficient of the attention feature is a positive number; An evaluation result of the training program is determined based on the comprehensive scoring score, wherein the comprehensive scoring score corresponds to the fatigue level, load level, stress level and concentration level.

3. The personnel training program evaluation method based on virtual reality environment according to claim 1 is characterized in that: The brain data, the physiological data and the behavioral data are subjected to feature fusion to obtain fusion features of the person to be evaluated, including: The brain data, the physiological data and the behavioral data are subjected to feature fusion based on a pre-trained multimodal evaluation model, wherein the training process of the multimodal evaluation model includes: Using time series feature data of different modalities as input of the multimodal evaluation model, and performing feature preprocessing and encoding on the time series feature data, wherein the multimodal time series features include brain data, physiological data, and behavioral data; In the CNN layer of the multimodal evaluation model, a convolutional neural network branch is designed for the temporal feature data of each modality, and in the Transformer layer of the multimodal evaluation model, the CNN extracted features from each modality are spliced ​​or weightedly fused to integrate the temporal feature data of different modalities; The integrated time series feature data is used as the input of the state prediction layer, and the score values ​​corresponding to the fatigue feature, load feature, stress feature and attention feature are predicted respectively based on the global average pooling and the fully connected layer; The multiple scoring scores are weighted and combined with their corresponding weight coefficients to obtain a comprehensive scoring score.

4. The personnel training program evaluation method based on virtual reality environment according to claim 3 is characterized in that: The method further comprises: Using different task parameters and task completion times in the same scenario, respectively construct scoring values ​​for stress characteristics, fatigue characteristics, load characteristics, and attention characteristics, and annotate the integrated temporal feature data of different modalities according to the tasks in the training program, and assign a label to each group of integrated temporal feature data, wherein the label is used to associate the temporal feature data with the scoring values ​​for stress characteristics, fatigue characteristics, load characteristics, and attention characteristics; The extracted time series feature data are screened based on variance analysis, and feature data with a P value greater than a preset threshold are screened as input to the multimodal evaluation model.

5. The personnel training program evaluation method based on virtual reality environment according to claim 1 is characterized in that: The method further comprises: The intensity and / or content of the training program is optimized based on the evaluation results, wherein the intensity of the training program includes the complexity of the task and the duration of the training, and the content of the training program includes skill training, simulation scenarios and decision making.

6. The personnel training program evaluation method based on virtual reality environment according to claim 1 is characterized in that: The brain data includes hemoglobin concentration, the physiological data includes heart rate variability, and the step of obtaining the brain data and physiological data of the person to be evaluated includes: The hemoglobin concentration and the heart rate variability are obtained based on functional near-infrared spectroscopy imaging technology FNIRS, wherein the hemoglobin concentration is comprehensively acquired based on light intensity data conversion, signal filtering, motion artifact processing, hemoglobin concentration calculation and feature extraction; the heart rate variability is comprehensively acquired based on light intensity data conversion, signal filtering, superposition averaging, R point detection, and feature extraction.

7. The personnel training program evaluation method based on virtual reality environment according to claim 1 is characterized in that: Obtaining behavioral data of the person to be evaluated, including: In a virtual reality (VR) environment, the behavioral data of the person to be evaluated in a target task training environment is obtained based on a VR device, wherein the behavioral data includes operation accuracy, response time, and error rate.

8. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a personnel training program evaluation method based on a virtual reality environment according to any one of claims 1 to 7 is implemented.

9. An electronic device, characterized in that: include: A memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the method for evaluating a personnel training program based on a virtual reality environment according to any one of claims 1 to 7 is implemented.

10. A personnel training program evaluation device based on a virtual reality environment, characterized in that: The device comprises: A first acquisition module is used to acquire brain data and physiological data of the person to be evaluated during the training process based on the training plan; The second acquisition module is used to obtain the behavior data of the person to be evaluated; A fusion module, used for performing feature fusion on the brain data, the physiological data and the behavioral data to obtain fusion features of the person to be evaluated; A third acquisition module, used for obtaining a plurality of status features of the person to be evaluated based on the fusion feature; A determination module is used to determine the evaluation result of the training program according to the scoring scores of the multiple state characteristics.

Images