Human factor intelligence data feedback training method, system, device and storage medium

Abstract

The present application relates to the field of artificial intelligence, and specifically to a training method, system, device and storage medium based on human intelligence data feedback, aiming to improve the training effect. The training method based on human intelligence data feedback of the present application includes: collecting multimodal data information of trainees when they are trained in virtual reality and / or augmented reality scenes; calculating the statistical parameters of the set state of the trainees during training through a preset neural network model based on the multimodal data information; judging the safety level of the trainees based on the statistical parameters of the set state; generating customized training feedback based on the safety level, so as to adjust the virtual reality and / or augmented reality scenes to further train the trainees. The present application can fully and comprehensively understand the current status of the trainees, effectively improving the training effect and safety of the training.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to a human factors intelligence data feedback training method, system, device and storage medium. Background Art

[0002] Artificial intelligence technology provides intelligent assistance and evaluation for simulation training. Among them, virtual reality (VR) technology is a technology that places users in a computer-generated virtual environment; augmented reality (AR) technology is a technology that creates an enhanced visual experience by superimposing virtual information on the real world.

[0003] In related technologies, AR / VR-based personnel training systems usually use a general training system to train a certain type of trainees, which cannot provide sufficiently detailed information to understand the status and needs of a specific trainee, thereby limiting the training effect. Summary of the invention

[0004] In order to solve the above problems in the prior art, the present application proposes a training method, system, device and storage medium based on human factors intelligent data feedback to improve the training effect.

[0005] In a first aspect of the present application, a human factor intelligence data feedback training method is proposed, the method comprising:

[0006] Collect multimodal data information when trainees are trained in virtual reality and / or augmented reality scenarios;

[0007] Calculating statistical parameters of a set state of the trainee during training by using a preset neural network model according to the collected multimodal data information, wherein the set state includes at least one of a fatigue state and a distraction state;

[0008] Determining the safety level of the trainee according to the statistical parameters of the set state;

[0009] Customized training feedback is generated based on the safety level.

[0010] Preferably, the multimodal data information includes at least: EEG signals, eye movement signals, physiological signals, spatiotemporal trajectory signals, motion capture signals, behavior numbers and interaction information.

[0011] Preferably, the statistical parameter of the set state is the probability of the set state;

[0012] The step of calculating the statistical parameters of the set state of the trainee during training by using a preset neural network model based on the collected multimodal data information includes:

[0013] The attention distribution of each modality feature relative to other modality features is calculated through the multi-head attention mechanism according to the following formula:

[0014]

[0015] Among them, Attention i represents the attention distribution of the i-th modality feature relative to other modality features; e i,j represents the correlation between the i-th modal feature and the j-th modal feature, n represents the number of modalities; the value range of i and j is [1, n];

[0016] Calculate weighted features based on the attention distribution:

[0017]

[0018] Among them, Weighted_Feature i Represents the weighted feature of the i-th mode; Feature j represents the j-th modal feature of the neural network model input;

[0019] Calculate the state scores of each set state of the trainee according to the weighted features and the weight of each modality:

[0020]

[0021] Among them, W m represents weight;

[0022] Calculate the probability that the trainee is in the set state according to the score:

[0023] p=σ(Fatigue Score )

[0024] Among them, σ represents the sigmoid activation function.

[0025] Preferably, the step of collecting multimodal data information of trainees when they are trained in virtual reality and / or augmented reality scenarios includes:

[0026] Pre-create virtual reality and / or augmented reality scenarios to simulate training environments;

[0027] According to the simulated training environment, the training environment parameters in the neural network model are established.

[0028] Preferably, the method further comprises:

[0029] Analyzing statistical parameters of the set state of the trainee during training through the neural network model in combination with the training environment parameters and the collected multimodal data information;

[0030] The safety level of the trainee is determined according to the statistical parameters of the set state.

[0031] Preferably, the method further comprises:

[0032] The neural network model is trained using a binary cross entropy loss function, and during training, an Adam optimizer is used to adjust the parameters of the neural network model to minimize the loss function.

[0033] In a second aspect of the present application, a human factors intelligence data feedback training system is proposed, wherein the system performs feedback training according to the method described above.

[0034] The third aspect of the present application proposes a human factors intelligent data feedback training device, comprising a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute the method described above.

[0035] In a fourth aspect of the present application, a computer-readable storage medium is provided, storing a computer program that can be loaded by a processor and execute the method described above.

[0036] This application has the following beneficial effects:

[0037] The human factors intelligent data feedback training method proposed in this application collects multimodal data of trainees when they are trained in AR / VR scenes, and then uses a neural network model to analyze the multimodal data in real time, so as to fully and comprehensively understand the current status of trainees, and judge the safety level according to the current status of trainees and generate customized training feedback, so as to adjust the AR / VR scene for further training of current trainees. Through the above method, this application effectively improves the training effect and safety of training.

[0038] The multimodal data information of the present application includes at least: EEG signals, eye movement signals, physiological signals, spatiotemporal trajectory signals, motion capture signals, behavior numbers and interaction information, which can provide a comprehensive understanding of the trainee's status from multiple angles.

[0039] This application uses a multi-head attention mechanism to dynamically calculate the correlation between the features of each modality, dynamically adjust the attention, and calculate the weighted features, so that the model pays more attention to the features that have a decisive impact on the current task. The attention mechanism makes the neural network model more interpretable and can explain the decision basis of the model on different modal features. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the main steps of an embodiment of the human factor intelligence data feedback training method of the present application;

[0041] Figure 2 It is a schematic diagram of the main components of an embodiment of the human factors intelligent data feedback training system of the present application. DETAILED DESCRIPTION

[0042] The preferred embodiments of the present application are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] It should be noted that, in the description of the present application, the terms "first" and "second" are only for the convenience of description, and do not indicate or imply the relative importance of the devices, elements or parameters, and therefore cannot be understood as a limitation of the present application. In addition, the term "and / or" in the present application is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.

[0045] Figure 1 Schematic diagram of the main steps of the first embodiment of the human factor intelligence data feedback training method of the present application. Figure 1 As shown, the feedback training method of this embodiment includes steps A10-A40:

[0046] Step A10, collecting multimodal data information of the trainee when receiving training in the virtual reality and / or augmented reality scene.

[0047] Among them, the multimodal data information includes at least: EEG signals, eye movement signals, physiological signals, spatiotemporal trajectory signals, motion capture signals, behavior numbers and interaction information.

[0048] Step A20, based on the collected multimodal data information, calculate the statistical parameters of the set state of the trainee during training through a preset neural network model, wherein the set state includes at least one of a fatigue state and a distraction state.

[0049] In this embodiment, the statistical parameter of the set state may be the probability of the set state, and this step may further include steps A21-A24:

[0050] Step A21, - Calculate the attention distribution of each modality feature relative to other modality features according to formula (1) through the multi-head attention mechanism:

[0051]

[0052] Among them, Attention i represents the attention distribution of the i-th modality feature relative to other modality features; e i,j It represents the correlation between the i-th modal feature and the j-th modal feature, n represents the number of modalities; the value range of i and j is [1, n].

[0053] In the attention mechanism, the relevance can be calculated by following the steps (1)-(2):

[0054] (1) Calculate the similarity metric. Use a metric such as dot product or cosine similarity to calculate the similarity between feature i and feature j. This usually involves normalizing the feature vectors and then calculating their dot product or cosine similarity, as shown in formula (2):

[0055] similarity(Feature i ,Feature j )(2)

[0056] The similarity(.) here can be dot product, cosine similarity, etc.

[0057] (2) Calculate the degree of association. By appropriately scaling and transforming the similarity measure, the degree of association is obtained, as shown in formula (3):

[0058] e i,j =normalize(similarity(Feature i ,Feature j )) (3)

[0059] This step usually involves performing a softmax operation on the similarities to ensure that the values ​​of the associations are in the range [0, 1] and that they can be viewed as probability distributions.

[0060] Step A22, calculate the weighted features according to the attention distribution, as shown in formula (4):

[0061]

[0062] Among them, Weighted_Feature i Represents the weighted feature of the i-th mode; Feature j Represents the j-th modal feature of the neural network model input.

[0063] Step A23, calculate the scores of each set state of the trainee according to the weighted features of each mode and the weight of each mode, as shown in formula (5):

[0064]

[0065] Among them, W m Represents weight.

[0066] Step A24, calculate the probability that the trainee is in the set state according to the score, as shown in formula (6):

[0067] p=σ(Fatigue Score ) (6)

[0068] Among them, σ represents the sigmoid activation function.

[0069] Step A30, determining the safety level of the trainee according to the statistical parameters of the set state.

[0070] Step A40, generating customized training feedback according to the safety level so as to adjust the virtual reality and / or augmented reality scene to further train the trainee.

[0071] In this embodiment, if the trainee is a driver, the trainee's current state can be divided into three safety levels: safe, warning and dangerous according to the probability of the set state and the preset threshold. Among them, the safe level indicates that the driver is in a good state, the warning level indicates that the driver may be distracted or nervous, and the dangerous level indicates that the driver's state is not suitable for continuing to drive.

[0072] When the trainee is at a safe level, the system can continue to provide standard training tasks to maintain skill levels; at a warning level, the system can introduce some distractions, such as simulating the complexity of traffic conditions, to help drivers improve their ability to concentrate; at a dangerous level, the system can trigger emergency situations, such as simulating braking or avoiding obstacles, to test the driver's response ability.

[0073] In different training tasks (such as driver training, firefighter training, pilot training, surgeon training, etc.), the state score formula, threshold and training scenario generation strategy can be adjusted according to actual needs.

[0074] Figure 2 Schematic diagram of the main steps of the second embodiment of the human factor intelligence data feedback training method of the present application. Figure 2 As shown, the feedback training method of the present application may include steps B10-B60:

[0075] Step B10, pre-create a virtual reality and / or augmented reality scene to simulate a training environment.

[0076] Step B20, establishing training environment parameters in the neural network model according to the simulated training environment.

[0077] Step B30, collecting multimodal data information of the trainee when receiving training in the virtual reality and / or augmented reality scene.

[0078] Step B40, combining the training environment parameters and the collected multimodal data information, and analyzing the statistical parameters of the trainees in the set state during training through the neural network model.

[0079] Step B50, determining the safety level of the trainee according to the statistical parameters of the set state.

[0080] Step B60, generating customized training feedback according to the safety level so as to adjust the virtual reality and / or augmented reality scene to further train the trainee.

[0081] In an optional embodiment, step B25 may also be included:

[0082] Step B25, using a binary cross entropy loss function to train the neural network model, and using an Adam optimizer to adjust the parameters of the neural network model during training to minimize the loss function.

[0083] Although the various steps in the above embodiment are described in the above-mentioned chronological order, those skilled in the art can understand that in order to achieve the effect of this embodiment, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple changes are within the scope of protection of this application.

[0084] Based on the same technical concept as the above method embodiment, the present application also provides an embodiment of a human factors intelligence data feedback training system. The system of this embodiment performs feedback training according to the method described above.

[0085] Furthermore, the present application also provides an embodiment of a human factor intelligence data feedback training device. The processing device of this embodiment includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute the method described above.

[0086] Furthermore, the present application also provides an embodiment of a computer-readable storage medium. The storage medium of this embodiment stores a computer program that can be loaded by a processor and execute the method described above.

[0087] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0088] Those skilled in the art should be able to appreciate that the method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0089] So far, the technical solution of the present application has been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present application.

Claims

1. A training method based on human intelligence data feedback, characterized in that: The method comprises: Collect multimodal data information when trainees are trained in virtual reality and / or augmented reality scenarios; Calculating statistical parameters of a set state of the trainee during training by using a preset neural network model according to the collected multimodal data information, wherein the set state includes at least one of a fatigue state and a distraction state; Determining the safety level of the trainee according to the statistical parameters of the set state; generating customized training feedback based on the safety level; in, The statistical parameter of the set state is the probability of the set state; The step of calculating the statistical parameters of the set state of the trainee during training by using a preset neural network model based on the collected multimodal data information includes: The attention distribution of each modality feature relative to other modality features is calculated through the multi-head attention mechanism according to the following formula: Among them, Attention i represents the attention distribution of the i-th modality feature relative to other modality features; e i,j represents the correlation between the i-th modal feature and the j-th modal feature, n represents the number of modalities; the value range of i and j is [1, n]; Calculate weighted features based on the attention distribution: Among them, Weighted_Feature i Represents the weighted feature of the i-th mode; Feature j represents the j-th modal feature of the neural network model input; Calculate each of the set state scores of the trainee according to the weighted features and the weight of each modality: Among them, W m represents weight; Calculate the probability that the trainee is in the set state according to the score: p=σ(Fatigue Score ) Among them, σ represents the sigmoid activation function.

2. The method according to claim 1, characterized in that The multimodal data information includes at least: electroencephalogram signals, eye movement signals, physiological signals, spatiotemporal trajectory signals, motion capture signals, behavior numbers and interaction information.

3. The method according to claim 1, characterized in that The method of collecting multimodal data information of trainees when they are trained in virtual reality and / or augmented reality scenarios includes: Pre-create virtual reality and / or augmented reality scenarios to simulate training environments; According to the simulated training environment, the training environment parameters in the neural network model are established.

4. The method according to claim 3, characterized in that The method further comprises: Analyzing statistical parameters of the set state of the trainee during training through the neural network model in combination with the training environment parameters and the collected multimodal data information; The safety level of the trainee is determined according to the statistical parameters of the set state.

5. The method according to claim 3, further comprising: The neural network model is trained using a binary cross entropy loss function, and during training, an Adam optimizer is used to adjust the parameters of the neural network model to minimize the loss function.

6. A training system based on human factor intelligence data feedback, characterized in that: The system performs feedback training according to the method according to any one of claims 1-5.

7. A training device based on human factors intelligent data feedback, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 5.

Images