Method for dynamic control of multi-path accident simulation experience process based on VR scene

Abstract

This application relates to a method, apparatus, computer device, and storage medium for dynamic control of a multi-path accident simulation experience process based on a VR scene. The method includes: acquiring multimodal physiological state data and task complexity data of a target user in a virtual scene; determining cognitive load assessment results through a dynamic coupling model; acquiring visual gaze data and limb operation data and performing spatiotemporal matching analysis to determine attention allocation assessment results; using the cognitive load and attention allocation assessment results as state inputs, and outputting scene parameter adjustment strategies through a reinforcement learning model; and dynamically adjusting the information presentation density, fault complexity, and / or task interference intensity of the virtual scene based on these strategies. This application achieves dynamic adaptation between the complexity of the virtual scene and the user's real-time cognitive state, avoiding cognitive overload and improving training efficiency.

Description

Technical Field

[0001] This application relates to the fields of virtual reality and human-computer interaction technology, and in particular to a method, device, computer equipment and storage medium for dynamic control of multi-path accident simulation experience process based on VR scene. Background Technology

[0002] With the rapid development of virtual reality (VR) technology, it has been widely used in accident simulation and emergency drill training in high-risk industries (such as power, petrochemical, and special equipment operation). These VR accident simulation systems construct highly immersive three-dimensional virtual environments, allowing trainees to repeatedly experience sudden malfunctions or accidents in a physically safe environment. This enables them to effectively accumulate response experience and learn standardized emergency response procedures under safe and controllable conditions. Compared to traditional on-site drills or theoretical lectures, this digital training method demonstrates significant advantages in reducing training costs and physical safety risks.

[0003] Currently, mainstream VR accident simulation training systems generally use pre-set fixed processes or logic based on simple conditional branches to drive the experience. Their core control logic is usually pre-fixed in the scene script: that is, under specific trigger conditions, the system plays corresponding visual effects and audio prompts step by step, guiding the user to complete predetermined operational steps. However, this "pre-set script" control mechanism has a significant limitation: it can only reflect an idealized or averaged training process, completely ignoring the huge individual differences among trainees when facing complex accident scenarios. Specifically, different users have vastly different thresholds for handling psychological stress, processing multi-source information, and hand-eye coordination. When the complexity of the accident evolution in the virtual scene (such as the number of concurrent failures or the density of interfering information) exceeds the user's current actual tolerance, it can easily lead to cognitive overload, which not only seriously weakens the effectiveness of training but may even cause psychological resistance to training due to excessive fright or frustration, affecting the acquisition of emergency response capabilities in high-stress scenarios. Summary of the Invention

[0004] Therefore, it is necessary to provide an adaptive control method, device, computer equipment, and storage medium for virtual scenes that can adaptively adjust the complexity of virtual scenes according to the user's real-time physical and mental state, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a dynamic control method for a multi-path accident simulation experience process based on a VR scene. The method includes: Acquire multimodal physiological state data of the target user in a virtual scene, as well as the current task complexity data of the virtual scene; based on the multimodal physiological state data and the task complexity data, determine the cognitive load assessment result of the target user through a dynamic coupling model; The system acquires visual gaze data of the target user toward a predefined safe area, as well as limb movement data of the target action; it performs spatiotemporal matching analysis on the visual gaze data and the limb movement data to determine the target user's attention allocation evaluation result. The cognitive load assessment results and the attention allocation assessment results are used as state inputs, and a scene parameter adjustment strategy is output through a reinforcement learning model. Based on the scenario parameter adjustment strategy, the information presentation density, fault complexity, and / or task interference intensity of the virtual scenario are dynamically adjusted.

[0006] In one embodiment, the multimodal physiological state data includes at least two of the following: electroencephalogram (EEG) signals, heart rate variability (HRV) signals, and skin conductance signals. The task complexity data includes a weighted combination of the number of device parameter reads and the number of process steps.

[0007] In one embodiment, determining the cognitive load assessment result of the target user through the dynamic coupling model includes: The dynamic coupling coefficient is calculated using the dynamic coupling model described above. The formula for calculating the dynamic coupling coefficient is as follows: Among them, among them, Let be the dynamic coupling coefficient at time t. The power of the brainwave β wave at time t is extracted based on the aforementioned brainwave signal. The heart rate variability index at time t is extracted based on the new path variability signal. This is a skin conductivity response index at time t based on the signal extracted from the skin point. Score the complexity of the character at time t. , and Preset weighting coefficients; Here is the regularization constant; The cognitive load assessment result is determined based on the dynamic coupling coefficient.

[0008] In one embodiment, spatiotemporal matching analysis is performed on the visual gaze data and the limb movement data to determine the attention allocation evaluation result of the target user, including: Based on the visual gaze data, determine the percentage of visual gaze time of the target user in the predefined safe area; Based on the limb manipulation data, determine the percentage of correct operation time for the target user when performing the correct limb manipulation; The spatiotemporal overlap is calculated based on the visual fixation duration ratio and the correct operation time ratio. The attention allocation evaluation result is determined based on the spatiotemporal overlap.

[0009] In one embodiment, determining the attention allocation evaluation result based on the spatiotemporal overlap includes: The Visual Attention Deficit Index is calculated using the following formula: in, The visual attention deficit index is mentioned above. The total number of sampling times. For the first The task stage weights corresponding to each sampling time point For the first The percentage of visual gaze time of the target user in the predefined safe area at each sampling time point. For the first The percentage of correct operation time for the target user at each sampling time point. For the first The total gaze duration of the target user at each sampling time point For the first The total operation time of the target user at each sampling time point; The visual attention deficit index is used as the result of the attention allocation assessment.

[0010] In one embodiment, the state space of the reinforcement learning model includes the cognitive load assessment result and the attention allocation assessment result; The action space of the reinforcement learning model includes information presentation density adjustment, fault complexity adjustment, and task interference intensity adjustment. The reward function of the reinforcement learning model is constructed based on the attention allocation evaluation result, the cognitive load evaluation result, and the task completion degree, which is used to characterize the overall progress of the current task.

[0011] In one embodiment, the method further includes: When the burst power of the target physiological signal in the multimodal physiological state data exceeds a preset adaptive threshold, a scenario simplification protocol is triggered; the target physiological signal includes the β-band signal in electromyography and / or electroencephalography, and the burst power is the sum of the squares of the sampled values ​​of the target physiological signal within a preset short time window; The scenario simplification protocol includes reducing the information presentation density and / or the task interference intensity of the virtual scenario.

[0012] Secondly, this application also provides a dynamic control device for a multi-path accident simulation experience process based on a VR scene. The device includes: The data acquisition module is used to acquire multimodal physiological state data of the target user in the virtual scene, as well as the current task complexity data of the virtual scene; acquire visual gaze data of the target user towards a predefined safe area, as well as limb operation data for the target action; The cognitive assessment module is used to determine the cognitive load assessment result of the target user based on the multimodal physiological state data and the task complexity data through a dynamic coupling model. An attention assessment module is used to perform spatiotemporal matching analysis on the visual gaze data and the limb operation data to determine the attention allocation assessment result of the target user. The scene adjustment module is used to take the cognitive load assessment results and the attention allocation assessment results as state inputs, and output scene parameter adjustment strategies through a reinforcement learning model; based on the scene parameter adjustment strategies, the information presentation density, fault complexity and / or task interference intensity of the virtual scene are dynamically adjusted.

[0013] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the aforementioned dynamic control method for multi-path accident simulation experience flow based on VR scenes.

[0014] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the aforementioned dynamic control method for multi-path accident simulation experience flow based on a VR scene.

[0015] The aforementioned dynamic control method, device, computer equipment, and storage medium for multi-path accident simulation experience based on VR scenarios achieves dynamic and precise adaptation between the complexity of accident simulation and the user's real-time cognitive state by establishing a real-time closed-loop control architecture of "perception-evaluation-intervention." Specifically, it first simultaneously acquires the user's multimodal internal physiological responses and external task environment load in the virtual scene, and uses a dynamic coupling model to perform nonlinear correlation analysis on the two, thereby accurately quantifying the user's true cognitive load level under the current accident process and overcoming the one-sidedness problem caused by single-dimensional evaluation. At the same time, by conducting spatiotemporal matching analysis on the spatial orientation of visual gaze and the effectiveness of limb operation function, an attention allocation evaluation mechanism capable of identifying implicit defects such as "not seeing" or "blind operation" is constructed, further enriching the representation dimension of the user's state from the behavioral execution level. On this basis, the dual-dimensional evaluation results of cognitive load and attention allocation are used together as the input state of the reinforcement learning model, enabling the model to output continuous and smooth scene parameter adjustment strategies according to the user's real-time physical and mental state, thereby dynamically adjusting the information presentation density and task interference intensity of the virtual scene, so that the rhythm and difficulty of the accident evolution can adaptively follow the changes in the user's ability. This mechanism avoids cognitive breakdown and training interruption caused by the difficulty of the scenario exceeding the user's tolerance threshold, and also prevents distraction and training failure caused by the scenario being too simple. Thus, while ensuring the psychological safety of emergency drills, it improves the adaptability and efficiency of emergency response training in complex accident scenarios. Attached Figure Description

[0016] Figure 1 This is an application environment diagram of a dynamic control method for a multi-path accident simulation experience process based on a VR scene in one embodiment. Figure 2 This is a flowchart illustrating a dynamic control method for a multi-path accident simulation experience based on a VR scene in one embodiment. Figure 3 This is a flowchart illustrating the process of determining the evaluation results of the target user's attention allocation in one embodiment; Figure 4 This is a structural block diagram of a dynamic control device for a multi-path accident simulation experience process based on a VR scene in one embodiment. Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0017] The dynamic control method for multi-path accident simulation experience process based on VR scenes provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, the virtual reality device 102 communicates with the server 104 via a network. A data storage system can store the data that the server 104 needs to process. The data storage system can be integrated onto the server 104 or placed in the cloud or on another network server. During the operation of the virtual scene, the virtual reality device 102 is responsible for collecting multimodal physiological state data, visual gaze data, and limb operation data of the target user, and transmitting the collected data to the server 104 via the network. After receiving the data, the server 104 performs real-time assessment of cognitive load and attention allocation based on the current task complexity, generates scene parameter adjustment strategies through a reinforcement learning model, and then returns the adjustment strategies to the virtual reality device 102. The virtual reality device 102 dynamically adjusts the information presentation density, fault complexity, and / or task interference intensity of the virtual scene according to the received adjustment strategies, forming a closed-loop control of perception-assessment-intervention. The virtual reality device 102 may be, but is not limited to, various virtual reality (VR) headsets, augmented reality (AR) glasses, mixed reality (MR) headsets, immersive training simulators, and supporting physiological signal acquisition devices (such as EEG acquisition headsets, heart rate sensors, skin conductance sensors), eye tracking devices, motion capture devices, and haptic feedback devices. The server 104 may be implemented using a standalone server or a server cluster consisting of multiple servers.

[0018] Furthermore, it should be noted that the multimodal physiological state data (including but not limited to EEG signals, heart rate variability signals, and skin conductance signals), visual gaze data, and limb manipulation data involved in the embodiments of this application are clearly and explicitly communicated to the target user before their first use, outlining the purpose, method, scope, storage period, and impact on their personal rights, and obtaining the user's separate consent; the user has the right to withdraw their consent at any time. Strictly adhering to the principle of minimum necessity, only the data types and amounts necessary to achieve the virtual scene adaptive control function are collected, and no data processing beyond the authorized scope is performed. The feature extraction of physiological signals (such as EEG β-wave power spectral density values, heart rate variability SDNN values, and skin conductance response SCR amplitude), cognitive load assessment, and attention allocation analysis in each embodiment are all solely for the purpose of virtual scene adaptive adjustment. The data processing results are output in the form of non-identified information such as statistics, feature vectors, or evaluation indicators, and are not used for any unauthorized purposes. Meanwhile, security protection measures such as encrypted storage and access control are implemented for user data, and a full-process data security management system is established. The storage period for user data will not exceed the shortest time necessary to achieve the processing purpose, and the data will be promptly deleted or irreversibly anonymized upon expiration. In addition, convenient channels are provided for users to view, copy, correct, and delete their personal information, ensuring users' right to know and control their personal information.

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0020] Example 1 In existing virtual reality simulation training systems, the experience flow of virtual scenes is typically controlled by preset fixed paths or simple conditional branching logic. This means all users experience the same scene content in the same preset order, or switch between several preset branches based on the results of their actions. This static scene control method cannot adaptively adjust the scene complexity according to the user's real-time physical and mental state within the virtual scene, easily leading to a mismatch between the scene difficulty and the user's actual tolerance, thus affecting the effectiveness and safety of the training. Therefore, as... Figure 2 As shown, a dynamic control method for multi-path accident simulation experience flow based on VR scenes is provided, which is then applied to... Figure 1 Taking the virtual reality device 102 as an example, the following steps are included: Step S100: Obtain the target user's multimodal physiological state data in the virtual scene, as well as the current task complexity data of the virtual scene. Based on the multimodal physiological state data and task complexity data, determine the target user's cognitive load assessment result through a dynamic coupling model.

[0021] Specifically, multimodal physiological state data is used to characterize the target user's internal physical and mental responses during task execution. This data can include, but is not limited to, biometric data reflecting central nervous system activation levels, autonomic nervous system arousal, or emotional stress states, such as electroencephalogram (EEG) signals, heart rate variability (HRV) signals, skin conductance signals, electromyography (EMG) signals, or respiratory signals. Task complexity data is used to quantify the external objective load currently imposed on the target user by the virtual scenario. This data can be calculated in real-time based on the preset task logic within the virtual scenario, such as a weighted combination based on dimensions like the amount of information to be processed at the current stage, the number of operation steps, time pressure, or environmental interference factors. This embodiment provides a multi-dimensional raw data foundation for accurately assessing the user's true cognitive state by simultaneously collecting both internal physiological state and external task load data, avoiding assessment biases caused by relying on only a single data source.

[0022] It should be noted that, in specific implementations, multimodal physiological state data is used to characterize the target user's intrinsic physical and mental responses during task execution. This data may include, but is not limited to, biometric data reflecting central nervous system activation levels, autonomic nervous system arousal, or emotional stress states, such as electroencephalogram (EEG) signals, heart rate variability (HRV) signals, skin conductance (SCD) signals, electromyography (EMG) signals, or respiratory signals. In actual data acquisition, different modalities of physiological signals typically have different sampling rates. For example, the sampling rate for EEG signals may be 128 Hz, for HRV signals it may be 64 Hz, and for SCD signals it may be 100 Hz. To ensure data consistency in subsequent multimodal fusion analysis, an interpolation resampling method can be used to unify the signals of each modality to a preset unified time reference. Specifically, using a 50-millisecond sliding time window, linear interpolation is performed on the sampling points of each modality signal within this window to align them to a unified sampling rate of 20 Hz. Linear interpolation is a well-known signal processing technique in the field; its basic approach is to calculate the signal value at the intermediate time using a linear function based on the signal values ​​of two adjacent original sampling points. After time alignment, the multimodal data is organized into a synchronized sequence of feature vectors. Each feature vector contains the feature values ​​of each modal signal at the same time, thereby eliminating the time deviation caused by the difference in sampling rate.

[0023] In this embodiment, the specific meanings and feature extraction methods of each modality of physiological signal are as follows: EEG signal specifically refers to the raw EEG signal collected by an EEG acquisition device in the prefrontal cortex region of the target user, and the β-band power spectral density value extracted after 13-30Hz bandpass filtering (denoted as...). This value reflects the intensity of cognitive processing and information processing load in the cerebral cortex, and is measured in μV. 2 / Hz. The specific extraction method is as follows: Perform a Fast Fourier Transform (FFT) on the filtered EEG time-domain signal, calculate the power spectral density within the β band, and take the mean. Heart rate variability specifically refers to the standard deviation of normal heart rate intervals (denoted as ) calculated from heart rate interval data collected by a heart rate monitoring device. This value characterizes the autonomic nervous system's ability to regulate cognitive load and its psychological resilience, measured in milliseconds (ms). The specific calculation method is as follows: within a preset time window (e.g., 60 seconds), the RR interval (i.e., the time interval between two adjacent heartbeat peaks) of all normal heartbeats is detected, and the standard deviation of these RR intervals is calculated. Skin conductance signal specifically refers to the continuously recorded skin conductance data acquired through a skin conductance sensor, and the skin conductance response amplitude extracted through event correlation analysis (denoted as...). This value reflects the level of sympathetic nerve arousal and emotional stress, and is measured in microsiemens (μS). The specific extraction method is as follows: after detrending the raw skin conductance signal, the rising edge and peak point of the skin conductance response are detected, and the difference between the peak value and the baseline is taken as the SCR amplitude.

[0024] Specifically, the dynamic coupling model does not simply superimpose physiological and task data linearly, but establishes a non-linear interaction between them. This is because in actual human-computer interaction, the cognitive load of the same task complexity varies for users in different physiological states, and the tolerance of the same user to task load also differs significantly at different physiological arousal levels. The dynamic coupling model captures this dynamically changing coupling effect by embedding physiological features as moderating factors into the calculation logic of task complexity, or by applying task complexity as a weight to the fusion process of physiological features. The cognitive load assessment results determined in this way can more accurately reflect the target user's actual psychological resource consumption in the current specific task environment, providing a reliable endogenous state basis for subsequent adaptive adjustment.

[0025] In practical implementation, after obtaining multimodal physiological state data and task complexity data, a dynamic coupling model is used to determine the cognitive load assessment results of the target user. The dynamic coupling model is not a simple linear superposition of physiological and task data, but rather establishes a non-linear interaction between the two. This is because in actual human-computer interaction, the same task complexity produces different cognitive loads for users in different physiological states, and the tolerance of the same user to task load also varies significantly at different physiological arousal levels. The dynamic coupling model captures this dynamically changing coupling effect by embedding physiological features as modulating factors into the calculation logic of task complexity, or by applying task complexity as a weight to the fusion process of physiological features. The dynamic coupling model can adopt a hybrid architecture based on Long Short-Term Memory (LSTM) networks and Transformer attention mechanisms. Specifically, in the LTM network layer, the input is a temporally aligned sequence of multimodal feature vectors. LTM network units include forget gates, input gates, and output gates, selectively retaining temporal features related to cognitive load and suppressing noise interference through gating mechanisms. The Transformer layer uses a self-attention mechanism to weight and fuse the interactions between signals from different modalities, enabling the model to automatically learn the relative importance of each modal signal under different cognitive states. The input to this hybrid architecture is a time-aligned sequence of multimodal feature vectors and the current task complexity score; the output is a dynamic coupling coefficient. Before deployment, the dynamic coupling model needs to be pre-trained using historical experimental data. The training process includes collecting multimodal physiological data and corresponding subjective cognitive load scores from multiple target users performing tasks of different complexity. Subjective scores are used as labels to supervise and optimize the model parameters. The training objective is to minimize the mean squared error loss function, ensuring that the Pearson correlation coefficient between the model's output dynamic coupling coefficient and the subjective cognitive load score reaches a preset threshold (e.g., above 0.85). After training, the model parameters are fixed and deployed to the inference environment. It should be understood that the above-described Long Short-Term Memory (LSTM) network-Transformer hybrid architecture is only one exemplary implementation of the dynamic coupling model. In other implementations, a combination of a pure Transformer architecture, a convolutional neural network, and fully connected layers, or other neural network architectures capable of capturing multimodal temporal features and nonlinear interactions can also be used.

[0026] Step S200: Obtain visual gaze data of the target user toward a predefined safe area and limb operation data toward the target action, and perform spatiotemporal matching analysis on the visual gaze data and limb operation data to determine the target user's attention allocation evaluation results.

[0027] Specifically, visual gaze data reflects the target user's visual attention direction. This data can be collected using eye-tracking devices, capturing raw information such as the 3D spatial coordinates of the user's gaze point, gaze duration, and saccade trajectory. The gaze point refers to the coordinates of the intersection between the target user's line of sight and the surface of an object in the virtual scene's 3D coordinate system, calculated by the eye-tracking device using pupil-corneal reflection technology; it is essentially the 3D spatial location point where the target user's eyes are focused. Specifically, visual gaze data for predefined safe areas refers to the target user's gaze behavior data for predefined key interactive areas in the virtual scene. Limb manipulation data reflects the target user's actual behavioral execution. This data can be collected using motion capture devices, data gloves, or hand controllers, capturing the 3D spatial coordinates, motion trajectory, and gesture states (including grasping, clicking, or rotating operations). Correct operation means that the target user's hand coordinates are within the operational tolerance range required for the current task stage, and the gesture state conforms to the operational specifications for that task stage. The operational tolerance range can be a spherical spatial region with the target object's center coordinates as the center and a preset radius (e.g., 5 cm). When the hand coordinates are within this spherical region, the hand position is considered correct.

[0028] After acquiring visual gaze data and motor action data, spatiotemporal matching analysis is performed on these two types of data to determine the target user's attention allocation assessment results. Spatiotemporal matching analysis aims to evaluate the synergistic consistency between visual attention and motor action in the temporal and spatial dimensions, that is, to determine whether the target user is looking at the right place while performing the right action. This analysis can effectively identify implicit attention allocation deficiencies such as glancing at key areas without performing actions, performing actions without glancing at key areas, or visual-motor delay.

[0029] Step S300: The cognitive load assessment results and attention allocation assessment results are used as state inputs, and the scene parameter adjustment strategy is output through the reinforcement learning model.

[0030] Specifically, the state space of a reinforcement learning model simultaneously includes cognitive load assessment results representing the user's internal mental and physical state and attention allocation assessment results representing the quality of the user's explicit behavior. The reason for needing both types of assessment results as state input is that a single dimension cannot comprehensively represent the user's state in a complex human-computer interaction system: having only cognitive load without attention assessment may lead the system to misjudge a temporarily increased load due to focus as overload, or a decreased load due to distraction as ease; having only attention assessment without cognitive load cannot distinguish between operational errors caused by insufficient ability and performance degradation caused by physiological fatigue. Based on this composite state space, the reinforcement learning model automatically explores the optimal scene parameter adjustment strategy through a pre-trained reward function. The output adjustment directly affects the information presentation density of the virtual scene (such as the number of interface elements and the richness of detail) and the intensity of task interference (such as background noise and the frequency of sudden alarms). This closed-loop adjustment mechanism based on two-dimensional state-driven principles enables the virtual scene to proactively and adaptively optimize itself according to the user's real-time physical and mental state and behavior. This avoids cognitive overload and frustration caused by excessive difficulty, as well as ineffective training and distraction caused by excessive difficulty, thereby maximizing training efficiency while ensuring training safety.

[0031] S400 dynamically adjusts the information presentation density, fault complexity, and / or task interference intensity of the virtual scene based on scene parameter adjustment strategies.

[0032] In practical implementation, reinforcement learning models can employ algorithms suitable for continuous action spaces, such as Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO), or Flexible Action-Critic (SAC). Taking DDPG as an example, it comprises two core components: an Actor Network and a Critic Network. The Actor Network receives the current state vector (i.e., a two-dimensional or multi-dimensional state vector composed of cognitive load assessment results and attention allocation assessment results), and outputs a continuous action vector after mapping through a multi-layer fully connected neural network. This action vector contains specific values ​​for various dimensions, such as information presentation density adjustment, fault complexity adjustment, and task interference intensity adjustment. The Critic Network receives the concatenation result of the current state vector and the action vector output by the Actor Network, evaluates the expected cumulative reward value (i.e., Q-value) of the action in the current state, and guides the synchronous update of the parameters of the Actor Network and the Critic Network through temporal difference (TD) error backpropagation. To avoid instability during training, the DDPG algorithm employs two auxiliary mechanisms: a target network and a replay buffer. The target network slowly tracks the main network parameters through soft updates, stabilizing the training objective. The replay buffer stores transition tuples of historical state-action-reward-next state, and random sampling is used during training to break temporal correlations between data. Before deployment, the reinforcement learning model undergoes a pre-training phase. Pre-training can be performed on offline datasets or through online interaction in a simulation environment connected to a virtual scene engine. After pre-training, the model continues to incrementally learn through real-time interaction with users during the online deployment phase, continuously adapting to individual user differences and changes in the training process.

[0033] The information presentation density referred to here refers to the number of effective information elements simultaneously presented to the target user within a unit field of view in a virtual scene. This includes, but is not limited to, the number of digits of the display precision of instrument readings, the visibility of auxiliary prompt labels, the level of detail in environmental textures, and the number of simultaneously visible fault points. Higher information presentation density means a greater amount of visual information the target user needs to process, resulting in a corresponding increase in cognitive load. The task interference intensity referred to here refers to the level of environmental stressors in the virtual scene, excluding core task information. This includes, but is not limited to, the decibel level of background noise, the frequency of triggering sudden alarms, the amplitude of visual sway, and the frequency of occurrence of non-critical fault points. Higher task interference intensity means greater external interference experienced by the target user when performing core tasks, requiring a higher level of concentration. The fault complexity referred to here refers to the number, type diversity, and logical coupling degree of fault events active simultaneously in the virtual scene. Specifically, higher fault complexity means the target user needs to diagnose and handle more fault points simultaneously, the more complex the causal relationships and chain reactions between faults, and the higher the demands on the user's situational awareness and emergency decision-making capabilities. In this embodiment, the fault complexity can be weighted and quantified by the number of concurrent fault points in the scene, the impact range of each fault point, and the depth of the triggering chain between faults. By incorporating the fault complexity into the dynamic adjustment range, and combining information presentation density and task interference intensity, the system can achieve fine-grained control of scene cognitive load at multiple dimensions, avoiding the problem of excessive adjustment amplitude that would damage the scene realism when only adjusting a single dimension.

[0034] Based on the above, this embodiment simultaneously collects two types of heterogeneous data: internal physiological state and external task load. A dynamic coupling model is used to perform correlation analysis between the two to determine the cognitive load assessment result, enabling the cognitive load assessment to simultaneously reflect the interaction between the user's internal physiological state and the external task environment. Spatiotemporal matching analysis of visual gaze data and limb operation data is used to determine the attention allocation assessment result, quantifying the degree of coordination between visual attention and behavioral execution. The cognitive load assessment result and the attention allocation assessment result are used together as the state input of the reinforcement learning model. Based on the output scene parameter adjustment strategy, the information presentation density, fault complexity, and / or task interference intensity of the virtual scene are dynamically adjusted. This allows the complexity of the virtual scene to adaptively adjust according to the user's real-time physical and mental state, reducing the mismatch between scene difficulty and the user's actual tolerance, and improving the effectiveness and safety of virtual simulation training.

[0035] It should be understood that although this embodiment uses general virtual scene adaptive control as an example for illustration, the method is also applicable to various specific simulation training systems, rehabilitation medical systems or entertainment interactive systems, as long as they have the ability to perceive the user's state and adjust the scene parameters.

[0036] Although Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise expressly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0037] Example 2 When implementing adaptive control methods for virtual scenarios, the specific types of multimodal physiological state data and the specific composition of task complexity data used will affect the accuracy and comprehensiveness of cognitive load assessment. Different physiological signals characterize the user's cognitive processing state from different dimensions, while task complexity needs to comprehensively reflect multiple aspects of the objective workload of the task. Traditional single physiological indicator assessment methods are difficult to fully reflect the user's true physical and mental workload level in complex work environments. Therefore, based on Example 1, the multimodal physiological state data includes at least two of the following: electroencephalogram (EEG) signals, heart rate variability (HRV) signals, and skin conductance signals; the task complexity data includes a weighted combination of device parameter readings and the number of process steps.

[0038] Specifically, EEG signal refers to the raw brainwave signal collected by an EEG acquisition device in the prefrontal cortex of the target user, and the extracted β-band power spectral density value after bandpass filtering at 13 to 30 Hz. This value reflects the cognitive processing intensity and information processing load of the cerebral cortex, and is measured in microvolts squared per hertz (μV). 2 / Hz). The specific extraction method is as follows: the filtered EEG time-domain signal is divided into frames of 256 sampling points each, with a frame shift of 128 sampling points. A Hanning window is applied to each frame to suppress spectral leakage. Then, a Fast Fourier Transform is performed to calculate the power spectral density value in the β band (13 to 30 Hz) and take the average. Heart rate variability signal specifically refers to the standard deviation of normal heartbeat intervals calculated from heart rate interval data collected by heart rate monitoring equipment. This value characterizes the autonomic nervous system's ability to regulate cognitive load and psychological resilience, measured in milliseconds. The specific calculation method is as follows: within a preset time window (e.g., 60 seconds), the RR intervals (i.e., the time interval between two adjacent heartbeat peaks) of all normal heartbeats are detected, and the standard deviation of these RR intervals is calculated. Skin conductance signal specifically refers to the skin conductance response amplitude extracted from continuously recorded skin conductance data collected by a skin conductance sensor through event correlation analysis. This value reflects the degree of sympathetic nerve arousal and emotional stress level, measured in micro-Siemens. The specific extraction method is as follows: the original skin conductance signal is detrended (i.e., the long-term trend component of the signal is subtracted from the original signal), and then the rising edge of the skin conductance response is detected. Peak time Take the skin conductance value at the peak point. Skin conductance value at the start of the rising edge The difference between them is taken as the amplitude of the skin conductance response, i.e. The three physiological signals mentioned above were chosen because they characterize the user's cognitive processing state from different dimensions: electroencephalogram (EEG) signals directly reflect the level of cognitive activation in the cerebral cortex; heart rate variability (HRV) signals characterize the autonomic nervous system's ability to regulate cognitive load; and electrodermal conductance (EDC) signals sensitively capture the degree of emotional stress and sympathetic arousal. Using a combination of "at least two" signals ensures the accuracy of multidimensional data fusion assessment while providing flexibility for system deployment under different hardware configurations. For example, in lightweight wearable scenarios, only a combination of HRV and EDC signals can be used; in high-precision professional training scenarios, all three can be used together.

[0039] Task complexity data is defined as a weighted combination of the number of equipment parameter reads and the number of process steps. This is because in virtual simulation tasks, cognitive load comes not only from the number of operation steps (i.e., sequential load) but also from the information density that needs to be processed in each step (i.e., spatial load). Only a weighted combination of the two can truly reflect the psychological resource consumption of a complex task. The number of equipment parameter reads refers to the number of equipment parameters that the target user needs to read and understand from instruments, screens, or indicators in the virtual scene during the current task phase. For example, in a power operation scenario, the number of equipment parameter reads can include the total number of parameters such as voltmeter readings, ammeter readings, and switch status indications. The number of process steps refers to the number of consecutive operation steps that the target user needs to perform according to the operating procedures during the current task phase. The weighted combination of equipment parameter reads and process steps can be calculated in various ways, such as multiplying the equipment parameter reads by the number of process steps and then multiplying by a risk weight parameter, or multiplying the equipment parameter reads by the number of process steps by their respective weight coefficients and then summing them. In the case of multiple task components, the complexity scores of each task component are summed to obtain the total task complexity score.

[0040] Based on the above, this embodiment selects a combination of various physiological signals that are sensitive to cognitive processing, autonomic nervous system arousal, and emotional stress, and combines them with the amount of parameter readings and the number of process steps that reflect the objective workload of the task, thus providing a multi-dimensional data foundation for the assessment of cognitive workload and ensuring that the assessment results can comprehensively characterize the user's real physical and mental workload level in a complex work environment.

[0041] Furthermore, in determining the cognitive load assessment results based on multimodal physiological state data and task complexity data, establishing the mathematical relationship between physiological characteristics and task complexity to accurately quantify cognitive load is a crucial step affecting assessment accuracy. Traditional linear weighting methods struggle to capture the complex nonlinear interaction effects between physiological state and task load, such as the amplification of physiological responses under high task load and the tolerance of high task load under low physiological arousal. Therefore, in this embodiment, a dynamic coupling model is used to determine the cognitive load assessment results of the target user, including: The dynamic coupling coefficient is calculated using a dynamic coupling model. The formula for calculating the dynamic coupling coefficient is as follows: Among them, among them, Let be the dynamic coupling coefficient at time t. The power of the brainwave β wave at time t is extracted based on the aforementioned brainwave signal. The heart rate variability index at time t is extracted based on the new path variability signal. This is a skin conductivity response index at time t based on the signal extracted from the skin point. Score the complexity of the character at time t. , and Preset weighting coefficients; Here is the regularization constant; The cognitive load assessment results are determined based on the dynamic coupling coefficient.

[0042] Specifically, The unit is microvolt squared per hertz (μV) 2 / Hz), this value is obtained by averaging the power spectral density values ​​corresponding to the β band (13 to 30 Hz) after framing, windowing, and fast Fourier transform of the filtered EEG time-domain signal. The unit is milliseconds. This value is obtained by detecting the RR intervals of all normal heartbeats within a preset time window and calculating the standard deviation of these RR intervals. The end of the preset time window is aligned with time t. The unit is microSiemens. This value is obtained by detecting the rising edge and peak point of the skin conductance response after detrending the original skin conductance signal, and taking the difference between the peak value and the baseline.

[0043] This is a dimensionless positive real number that reflects the overall complexity of the task in the virtual scenario at time t; a higher value indicates a more complex task. It is calculated as follows: ,in, for The number of task components that are active simultaneously in a virtual scene at any given time. Index number of the task component ( Each task component corresponds to an independent task dimension in the virtual scene (such as device parameter reading task, emergency process execution task, environmental risk assessment task, etc.). for Time of the first The number of device parameters that the user needs to read or confirm in each task component; for Time of the first The number of emergency process steps that need to be executed in each task component, which is a non-negative integer; For the first The risk weight parameter corresponding to each task component is a positive real number, preset by the task designer based on the operational risk level of that task component (in high-risk live-line work task components, ...). It can be set to a higher value (e.g., 2.0), while in the regular inspection task component it is set to a lower value (e.g., 1.0)). When only one task component is active at time t (i.e. The above formula degenerates into This calculation method combines static task attributes with dynamic execution processes, ensuring that cognitive load assessment can keep pace with changes in the task rhythm in real time, providing accurate quantitative basis for subsequent adaptive scenario adjustments.

[0044] In the above formula, the weighting coefficients are preset. , , and middle, The weighting coefficient of the EEG signal represents the relative importance of the level of central nervous system cognitive activation in the overall cognitive load assessment. The weighting coefficient for heart rate variability signals represents the relative importance of the autonomic nervous system's regulatory capacity. The weighting coefficient of skin conductance signal represents the relative importance of the degree of sympathetic nerve stress arousal as a regulator of physiological tolerance. This is a regularization constant, taking a positive value greater than zero (e.g., 0.01). Its function is to prevent the regularization constant from being applied when... When the denominator approaches zero The numerical overflow problem caused by a value of zero. The aforementioned weighting coefficients. , , and The calibration can be achieved in the following way: First, recruit a predetermined number of target users (e.g., no less than 30) and have them perform standardized tasks covering low, medium, and high complexity levels in a virtual scenario; second, simultaneously collect the beta wave power of each user's brainwave. Heart rate variability SDNN value and skin conductance response amplitude The system requires users to complete a subjective cognitive load scale after each task phase to obtain subjective scores as label data. Finally, using the subjective cognitive load score as the dependent variable and each physiological characteristic as the independent variable, a multiple linear regression method is used to find the weight coefficient combination that minimizes the prediction error. After calibration, the weight coefficients are embedded in the dynamic coupling model and used directly during the online inference phase. It should be understood that the above calibration process is only an example; in other implementations, machine learning methods such as support vector regression, random forest regression, or neural network regression can be used for automatic learning of the weight coefficients.

[0045] Furthermore, in the above formula, the molecule part This characterizes the combined activation level of the user's central and autonomic nervous systems, i.e., the consumption of intrinsic cognitive resources. (Denominator part) Skin electrical conductance was introduced as a stress regulator, reflecting the user's physiological tolerance or buffering capacity under current stress: when the user is in a state of high tension or stress, As the value increases, the denominator also increases, which to some extent suppresses the interference of artificially inflated physiological signals caused solely by emotional excitement on cognitive load assessment. The entire fraction and Multiplication implies that the external task complexity amplifies the gain: even if the user's physiological adjustment ability is acceptable (i.e., the denominator is large), if the current task is extremely complex (i.e., ... (maximum value), dynamic coupling coefficient It will still increase significantly, thus accurately capturing the critical overload state where high workload overwhelms physiological regulation capacity.

[0046] To verify the effectiveness of the above formula for calculating the dynamic coupling coefficient, the following verification experiment was conducted during the pre-training phase: Multimodal physiological data of 30 target users were collected for 10 minutes at low, medium, and high task complexity levels, respectively, and NASA-TLX subjective cognitive load scores were recorded for each user. The dynamic coupling coefficient for each user at each task stage was calculated. ,Will Pearson correlation analysis was performed with the NASA-TLX score. Validation results showed that... The Pearson correlation coefficient with the NASA-TLX score was 0.87 to 0.92, which is higher than the correlation coefficient between traditional single physiological indicators and the NASA-TLX score. This indicates that the dynamic coupling coefficient proposed in this embodiment can more accurately quantify the user's cognitive load level compared with a single physiological indicator.

[0047] Based on the above, this embodiment constructs a nonlinear interaction relationship between physiological characteristics and task complexity through a specific mathematical model, enabling the molecular characterization of the comprehensive activation level of the central nervous system and the autonomic nervous system. The denominator introduces skin conductance response as a stress regulation factor and multiplies it with task complexity, thereby accurately capturing complex dynamic coupling effects such as the amplification of physiological responses under high task load or the tolerance of high task load under low physiological arousal, thus improving the accuracy of cognitive load quantification.

[0048] Example 3 In assessing a target user's attention allocation, aligning the spatial directionality of visual fixation with the temporal effectiveness of bodily actions to identify implicit attentional deficiencies—such as inconsistencies between fixation and action—is crucial for accurate assessment. Traditional methods typically focus only on eye movement trajectories or operational accuracy, failing to identify implicit attentional allocation deficiencies like "seeing but not seeing" (fixating but not acting) or "blindly acting" (acting but not focusing on key areas). Therefore, based on the example, such as... Figure 3 As shown, spatiotemporal matching analysis was performed on visual gaze data and limb movement data to determine the target user's attention allocation assessment results, including: Based on visual gaze data, determine the percentage of time a target user spends visually gazing in a predefined safe area; Based on limb movement data, determine the percentage of time that the target user takes to perform the correct limb movement. The spatiotemporal overlap is calculated based on the proportion of visual fixation time and the proportion of correct operation time. The evaluation results of attention allocation are determined based on spatiotemporal overlap.

[0049] Specifically, the spatiotemporal overlap here does not simply refer to the overlap of visual and action signals on the timeline, but emphasizes the precise alignment of the spatial orientation of visual attention and the functional effectiveness of physical operations within a specific spatiotemporal window. Predefined safe areas refer to key interactive objects or risk monitoring areas pre-marked according to task logic in a virtual scene, such as the instrument reading area or insulating glove inspection area in power simulation, or the lesion incision area or sterile instrument table in medical simulation. The proportion of visual gaze time reflects the user's visual capture of key information, while the proportion of correct operation time reflects the standardization of the user's behavior. Only when the user's gaze is focused on the safe area, and their hand performs the correct operation matching that area within that time period, is it considered a valid spatiotemporal overlap. This matching mechanism can effectively identify two types of hidden defects: first, blindness, where the gaze sweeps over the safe area but does not linger long enough to acquire information, or lingers but does not trigger the corresponding operation; second, blind operation, where the hand performs the action but the gaze does not confirm the state of the object being operated on, which can easily lead to safety accidents in high-voltage or high-risk operations. By calculating the spatiotemporal overlap between these two factors, the abstract quality of attention allocation is transformed into a quantifiable numerical indicator, providing a precise behavioral basis for subsequent adaptive adjustment.

[0050] Furthermore, in determining the attention allocation assessment results, to more sensitively reflect the risk differences of attention deficits at different task stages, a quantitative mechanism capable of dynamically adjusting the assessment sensitivity as the task progresses is needed. Traditional static statistical averaging methods cannot distinguish the differences in attention requirements between normal and high-risk stages, resulting in insufficient sensitivity to detect minute attention deviations at high-risk moments. Therefore, in this embodiment, the attention allocation assessment results are determined based on spatiotemporal overlap, including: calculating the visual attention deficit index, the formula for which is: in, Visual attention deficit index, The total number of sampling times. For the first The task stage weights corresponding to each sampling time point For the first The percentage of time a target user spends visually gazing at a predefined safe area at each sampling time. For the first The percentage of correct operation time for the target user at each sampling time point For the first The total gaze duration of the target user at each sampling time. For the first The total operation time of the target user at each sampling time; The Visual Attention Deficit Index was used as the result of attention allocation assessment.

[0051] Specifically, the formula mathematically constructs a normalized defect measurement model, with its value range limited to [0,1]. The fractional part on the right side of the formula represents the proportion of weighted effective eye-hand coordination behavior in the total behavior quantity; therefore, subtracting this ratio from 1 yields the defect index. The value range is limited to between 0 and 1. The closer the value is to 1, the more severe the separation between the target user's vision and actions, and the more significant the attention allocation deficit. A value closer to 0 indicates good eye-hand coordination and reasonable attention allocation. The total number of sampling moments is determined by dividing the preset sliding time window duration by the unit sampling interval. For example, when the window length is 5 seconds and the unit sampling interval is 50 milliseconds, n = 5000 milliseconds / 50 milliseconds = 100.

[0052] In actual virtual simulation tasks, operational risks at different stages are not uniformly distributed. For example, in the routine inspection phase of power training, brief eye-hand separation may not cause serious consequences. It can be set to a lower value (e.g., 1.0); however, in high-risk critical phases such as live wiring or fault clearing, the same eye-hand separation behavior may lead to catastrophic consequences, in which case it will automatically switch to a lower value (e.g., 1.0). Increase the value (e.g., 2.0 or higher). This means that in high-risk phases, the effective cooperating terms in the numerator of the formula will be amplified; if the actual cooperating term is insufficient, The value increases at a faster rate, making it more sensitive to defect detection. This dynamic evaluation mechanism based on risk weights makes attention assessment no longer a static statistical average, but an adaptive process that can adjust sensitivity in real time as the task progresses, ensuring that minute attention deviations can be captured first and intervened in a timely manner at high-risk moments. The following provides supplementary explanations on the specific calculation methods of each variable in the above formula. In this embodiment, the calculation of the visual attention deficiency index is based on a preset sliding time window (e.g., a window length of 5 seconds and a sliding step size of 1 second), with each sliding time window corresponding to one... The calculation is as follows: n is the total number of sampling moments within the sliding time window, which is determined by dividing the window length by the sampling interval; for example, when the window length is 5 seconds and the sampling interval is 50 milliseconds, n=100. The specific calculation method is as follows: within the unit sampling interval corresponding to the i-th sampling time, the coordinates of the target user's gaze point are obtained through the eye-tracking device, and it is determined whether the gaze point falls within the spatial range of the predefined safe area; if it falls within, the duration of the sampling interval is recorded as the effective gaze duration. This is the ratio of the effective gaze duration within the unit sampling interval to the total duration of the sampling interval, with a value range of [0,1]. The predefined safe area referred to here is one or more spatial regions pre-defined according to the task logic in the three-dimensional coordinate system of the virtual scene. Each region is defined by its center coordinates and bounding radius. When the Euclidean distance between the gaze point coordinates and the center coordinates of a certain safe area is less than or equal to the bounding radius of that area, the gaze point is determined to fall into that safe area. The specific calculation method is as follows: within the unit sampling interval corresponding to the i-th sampling time, the three-dimensional spatial coordinates and gesture state (including grasping, clicking or rotating) of the target user's hand are obtained through the motion capture device. The hand coordinates are matched with the spatial position of the target object required to be operated in the current task stage, and it is determined whether the gesture state conforms to the operation specifications of the task stage. If the hand coordinates are within the operation tolerance range of the target object (i.e., within a spherical spatial area with the center coordinates of the target object as the center and the preset radius as the radius) and the gesture state is correct, the duration of the sampling interval is recorded as the correct operation duration. It is the ratio of the correct operation time within the unit sampling interval to the total time of the sampling interval, and the value range is [0,1]. The total fixation time of the target user at the i-th sampling time is the total time during which the target user keeps their eyes open and the eye-tracking device effectively tracks the fixation point within that unit sampling interval, regardless of whether the fixation point is located within the safe area. The total operation time of the target user at the i-th sampling time is the total duration during which the target user's hand is effectively tracked by the motion capture device and is in motion or interactive state within the unit sampling interval, regardless of whether the operation is correct. The value is determined by the current task stage type of the virtual scene. For example, an internal mapping table between task stages and weight values ​​is maintained. This mapping table is preset by the task designer during the scene configuration stage according to the operational risk level of each stage. When the task stage of the virtual scene changes, the value corresponding to the current sampling time is automatically updated. This allows for dynamic adjustment of assessment sensitivity based on task risk.

[0053] It should be understood that although the above embodiments use the instrument area and insulating glove area in power safety training as examples of predefined safe areas, this does not constitute a limitation on the scope of protection of this application. In other application scenarios, predefined safe areas can be flexibly defined according to specific task requirements. For example, in minimally invasive surgery simulation training, the safe area can be a specific anatomical structure or the range of motion of the instrument tip under the laparoscopic view; in mine emergency drills, the safe area can be an escape route marker or the surface of a gas detector. Regardless of how the scenario changes, as long as the core logic of matching and analyzing visual gaze and limb operation in the spatiotemporal dimension is followed, and the evaluation results are dynamically corrected using task stage weights, it falls within the scope of protection of this application. Furthermore, and The specific calculation method can also be adjusted according to the sensor accuracy and task characteristics. For example, binary judgment (0 or 1) or continuous probability value can be used, as long as it can characterize the effectiveness of gaze and operation.

[0054] Specifically, a predefined safe area refers to a key interactive object or risk monitoring area pre-marked according to the task logic in a virtual scene. Each safe area is defined in the three-dimensional coordinate system of the virtual scene by its center coordinates and its bounding radius; that is, a safe area is a spherical spatial region with the center coordinates as its center and the bounding radius as its radius. There can be one or more predefined safe areas. When multiple safe areas exist, any safe area being viewed is considered to satisfy the condition of "viewing a safe area".

[0055] The specific method for determining the percentage of visual gaze duration of a target user within a predefined safe area based on visual gaze data is as follows: The three-dimensional spatial coordinates of the target user's gaze point are obtained using an eye-tracking device. For the unit sampling interval (e.g., 50 milliseconds) corresponding to the i-th sampling moment, the Euclidean distance between the gaze point coordinates and the center coordinates of each predefined safe area is calculated. If this distance is less than or equal to the enclosing radius of the corresponding safe area, the gaze point is determined to fall within that safe area, and this sampling interval is recorded as the effective gaze sampling interval. The number of sampling intervals falling within any predefined safe area within a preset sliding time window (e.g., 5 seconds) is counted, multiplied by the unit sampling interval duration to obtain the effective gaze duration, and divided by the total gaze duration within the window (i.e., the number of sampling intervals within the window where the eye-tracking device effectively tracks the gaze point multiplied by the unit sampling interval duration) to obtain the percentage of visual gaze duration. Its value ranges from 0 to 1.

[0056] The specific method for determining the percentage of correct operation time for a target user performing the correct limb operation based on limb operation data is as follows: The three-dimensional spatial coordinates and gesture state (such as grasping, clicking, rotating, etc.) of the target user's hand are acquired using a motion capture device. If the Euclidean distance between the hand coordinates and the center coordinates of the target object required for the current task stage is less than or equal to a preset operation tolerance range (e.g., 5 cm), and the gesture state is consistent with the gesture state required for the current task stage, then this sampling interval is recorded as the correct operation sampling interval. The correct operation time is obtained by multiplying the number of correct operation sampling intervals within the statistical window by the unit sampling interval duration. This correct operation time is then divided by the total operation time within the window (i.e., the number of sampling intervals where the motion capture device effectively tracks the hand within the window multiplied by the unit sampling interval duration) to obtain the percentage of correct operation time. Its value ranges from 0 to 1.

[0057] The specific method for calculating spatiotemporal overlap based on the proportion of visual fixation time and the proportion of correct operation time is as follows: within a preset sliding time window, for each sampling moment... ,calculate and The product of these values ​​yields the spatiotemporal overlap value at that moment. A larger product value indicates a higher degree of coordination and consistency between visual attention and physical actions in the spatiotemporal dimension at that moment. The specific method for determining the attention allocation evaluation result based on the spatiotemporal overlap is as follows: calculate the weighted sum of the spatiotemporal overlap values ​​at each sampling moment within a preset sliding time window, and determine the attention allocation evaluation result based on this sum.

[0058] To verify the above visual attention deficit index To verify its effectiveness, the following experiments were conducted: Three states were simulated in a virtual environment: "normal collaboration" (eyes focused on a safe area while hands perform correct operations), "seeing but not seeing" (eyes focused on a safe area but hands do not perform operations or perform incorrect operations), and "blind operation" (hands perform operations but eyes are not focused on a safe area). 100 sets of data were collected for each state. The results for each state were calculated. The values ​​indicate that under normal cooperative conditions... The mean of the values ​​is 0.06, and the standard deviation is 0.03; under the "ignoring" state The mean of the values ​​is 0.58, and the standard deviation is 0.07; under the "blind operation" state The mean of the values ​​is 0.49, and the standard deviation is 0.06, indicating that... It can effectively distinguish between normal collaborative states and various latent attention deficit states.

[0059] Based on the above, this embodiment aligns the spatial orientation of visual attention with the temporal effectiveness of limb manipulation, calculating the degree of overlap between the two in the spatiotemporal dimensions. This effectively identifies latent attention deficits such as "seeing but not seeing" (gazing but not acting) or "blindly acting" (acting but not gazing at key areas), overcoming the limitations of traditional methods that can only assess single-modal behavior. Furthermore, this embodiment introduces task stage weights, allowing attention assessment to dynamically change with the task risk level. Specifically, high-risk stages require higher levels of eye-hand coordination and exhibit greater sensitivity to the deficit index. The formula structure visually reflects the inverse relationship between the proportion of effective coordinated behavior and the total amount of behavior, providing a precise quantitative basis for subsequent adaptive intervention.

[0060] Example 4 In the process of using cognitive load assessment results and attention allocation assessment results as state inputs and adjusting strategies by outputting scenario parameters through a reinforcement learning model, the specific design of the state space, action space, and reward function determines the decision-making ability and adaptive adjustment effect of the reinforcement learning model. Traditional methods typically employ discrete state switching logic or make decisions based on only a single indicator, making it difficult to achieve continuous, multi-dimensional fine-tuning. Therefore, based on Examples 1 and 3, the state space of the reinforcement learning model includes the cognitive load assessment results and attention allocation assessment results; the action space of the reinforcement learning model includes the information presentation density adjustment amount, the fault complexity adjustment amount, and the task interference intensity adjustment amount; the reward function of the reinforcement learning model is constructed based on the attention allocation assessment results, cognitive load assessment results, and task completion degree, which is used to characterize the overall progress of the current task.

[0061] Specifically, the state space design adopts a two-dimensional driving mode, which simultaneously incorporates the cognitive load assessment results (i.e., dynamic coupling coefficient) that characterize the user's internal physical and mental stress. ) and the results of attention allocation assessment that characterizes the quality of users' explicit interactions (i.e., the visual attention deficit index). The state vector is represented as follows: In another embodiment, the cognitive load assessment results and attention allocation assessment results can each be refined into multidimensional sub-states (such as refining the cognitive load assessment results into EEG power characteristics, heart rate variability characteristics, etc.) to form a higher-dimensional state vector.

[0062] The action space is defined as adjustments in three consecutive dimensions. The action vector is represented as... ,in Adjust the density of information presentation. This is the adjustment amount for the degree of fault complexity. This refers to the adjustment amount for the task interference intensity. Using "adjustment amount" as the action output means that the corresponding hardware system is performing incremental optimization based on the current scene baseline. , , ,in The density is presented for the preset baseline information. The preset baseline fault complexity, The preset baseline task interference intensity consists of dimensionless positive real numbers. This incremental adjustment ensures the continuity of the training process and the stability of the user experience.

[0063] The construction of the reward function is a key guiding mechanism for achieving adaptive control objectives. In this embodiment, the reward function comprehensively considers three dimensions: attention allocation assessment results, cognitive load assessment results, and task completion rate. The reward function is expressed as follows: ,in This encourages good eye-hand coordination behavior. The lower the score, the higher the reward. The cognitive load scoring function is designed as an inverted U-shaped curve, which aims to guide the user's cognitive load to be maintained within the optimal learning range, so as to avoid overload or boredom. This represents the task completion percentage, with a value between 0 and 1, indicating the current progress of the task (0 indicates not yet started, 1 indicates completed). Weighting coefficient. , and satisfy It can be dynamically adjusted according to the training stage; for example, it can be set during the beginner stage. , , In the advanced stage, settings can be configured. , , .

[0064] When adjusting the density of information presentation during actual implementation, the cognitive load assessment results can be used. The information is smoothly converted into an adjustment amount for presentation density through a mapping function. As an example, the Sigmoid function can be used as the mapping model to achieve a smooth transition of scene difficulty with changes in cognitive load. A feasible calculation formula for the mapping model is as follows: in, The density of target information at time t. The preset upper limit of baseline information density, The dynamic coupling coefficients calculated in Example 2 are... The threshold for cognitive load sensitivity. To adjust the slope coefficient, this formula utilizes the non-linear saturation characteristic of the sigmoid function to achieve a smooth transition of scene difficulty with changes in cognitive load. When far below When the denominator approaches 1, the density remains near the baseline level with an extremely low rate of change, ensuring stable performance under low loads; when near When the function enters the linear sensitivity region, the density decreases rapidly but continuously as the load increases, providing timely load reduction feedback; when Much higher As the denominator rapidly increases, the density approaches 0 or the minimum safe value, forming a gradual lower limit of protection. Compared to traditional piecewise linear functions or hard threshold switches, the Sigmoid function eliminates abrupt changes in the adjustment process, allowing users to experience a natural, breathing-like scaling of the scene when cognitive load fluctuates, rather than a mechanical jump. This is crucial for maintaining an immersive experience and a sense of psychological security. Parameters and Historical user data can be used for fitting and calibration to adapt to the adjustment needs of different sensitivity groups. It should be understood that although this embodiment uses the Deep Deterministic Policy Gradient (DDPG) algorithm as an example, this does not constitute a limitation on the scope of protection of this application. In other embodiments, PPO (Proximal Policy Optimization), SAC (Soft Actor-Critic), or other reinforcement learning algorithms suitable for continuous action spaces can also be used. Similarly, the specific mathematical form of the reward function can be adjusted according to the actual training objective, such as introducing a curiosity-driven mechanism or a sparse reward mechanism. As long as its core logic is based on the comprehensive feedback of the user's real-time state and task performance to guide the optimization of scene parameters, it falls within the scope of protection of this application. Furthermore, the parameters in the Sigmoid function can also be designed as variables that dynamically change with time or user proficiency to achieve a higher level of adaptive adjustment characteristics. In addition to the Sigmoid function adjustment of information presentation density mentioned above, the action space of the reinforcement learning model also includes the adjustment amount of fault complexity and the adjustment amount of task interference intensity, the specific adjustment methods of which are described below.

[0065] The dynamic adjustment of fault complexity adopts an exponential decay function as a mapping model, and its calculation formula is as follows: ,in Let be the target fault complexity at time t. The preset baseline fault complexity (i.e., the maximum number of associated fault points and complexity level in the virtual scene) is used, AFD(t) is the visual attention deficit index calculated in Example 3, and η is the preset adaptive decay coefficient (taking a positive value, such as 1.5). The physical meaning of this formula is: when the target user's attention deficit index... When higher, the exponential term As the number of faults approaches zero, the fault complexity decreases accordingly, thus reducing the number of faults that users need to handle simultaneously; conversely, when... At lower levels, the exponential term approaches 1, and the fault complexity remains near the baseline level. This exponential decay characteristic ensures that the rate of decrease in fault complexity is proportional to the severity of the attention deficit, avoiding the problem of oversimplification of the scenario in cases of minor defects, which is common with linear reduction methods.

[0066] The dynamic adjustment of the task interference intensity adopts a linear mapping method, and its calculation formula is as follows: ,in Let be the target task interference intensity at time t. The preset baseline task interference intensity (such as the baseline decibel value of background noise or the baseline trigger frequency of alarm). This is used to adjust the task interference intensity output of the reinforcement learning model, and its value ranges from [-0.5, +0.5]. When the value is negative, the task interference intensity decreases to between 50% and 100% of the baseline value; when When the value is positive, the intensity of task interference increases to between 100% and 150% of the baseline value.

[0067] To verify the effectiveness of the reinforcement learning model described above, a comparative experiment was conducted in a simulation environment. The reinforcement learning model of this embodiment (experimental group) and the fixed threshold rule controller (control group) were used to adjust scene parameters for 30 target users in the same virtual scene. Experimental results show that the experimental group maintained the average cognitive load within the optimal range for 78.3% of the time, while the control group only maintained it for 42.1%; the experimental group also maintained the average attention deficit index... The value was 0.12, while the control group was 0.29; the average task completion time for the experimental group was 85.3 seconds, while that for the control group was 112.7 seconds. These experimental data demonstrate that the reinforcement learning model in this embodiment can more effectively maintain the user's cognitive load within the optimal range, reduce the attention deficit index, and shorten the task completion time.

[0068] Based on the above, this embodiment establishes an end-to-end mapping mechanism from multi-dimensional state evaluation to multi-dimensional scene parameter adjustment through reinforcement learning algorithm. The design of the reward function takes into account three training objectives: moderate cognitive load, reasonable attention allocation, and task completion. Thus, it can automatically explore the most suitable personalized scene configuration strategy for the current user state while ensuring training effect.

[0069] Example 5 In virtual scenarios where sudden high-stress events may occur, the decision-making cycle of reinforcement learning models is typically on the order of hundreds of milliseconds to seconds, suitable for dealing with the gradual accumulation of cognitive load or the slow drift of attention. However, when sudden events such as electric arcs, sudden device drops, or sharp alarms occur in virtual scenarios, users will experience an instinctive startle reflex or acute stress response. This physiological abrupt change occurs within a timescale of tens of milliseconds. If the reinforcement learning model is allowed to complete state evaluation and policy reasoning before outputting adjustment instructions, the user may have already fallen into a state of cognitive freeze or extreme panic. Therefore, based on Examples 1 and 4, this method further includes: When the burst power of the target physiological signal in the multimodal physiological state data exceeds a preset adaptive threshold, a scenario simplification protocol is triggered; the target physiological signal includes the β-band signal in electromyography and / or electroencephalography, and the burst power is the sum of the squares of the sampled values ​​of the target physiological signal within a preset short time window; Scene simplification protocols include reducing the information presentation density, fault complexity, and / or task interference intensity of virtual scenes.

[0070] Specifically, the target physiological signal is preferably a signal mode capable of sensitively characterizing instantaneous somatic reflexes or cortical alertness levels, which may include the burst power of electromyography (EMG) signals and / or the instantaneous power of the beta band in electroencephalography (EEG) signals. A sudden increase in EMG signals typically corresponds to involuntary muscle tension or limb retraction when a user is startled, representing the most direct physical manifestation of somatic stress. The instantaneous peak power of EEG beta waves reflects the rapid alertness and emotional arousal of the cerebral cortex to a sudden threat. Burst power refers to the instantaneous energy value of the target physiological signal within a preset short-term window. Its specific calculation method is to sum the squares of the sampled values ​​of the target physiological signal within the short-term window (e.g., 200 milliseconds), i.e. ,in Let be the signal amplitude at the j-th sampling point within the short-time window, and the summation range covers all sampling points within that window. When the calculated burst power exceeds the corresponding preset adaptive threshold, it is determined that a sudden change has occurred in the physiological signal of that channel.

[0071] In practical applications, a dual-channel joint determination strategy can be adopted to improve the specificity and accuracy of triggering. This means that triggering is only confirmed when the burst power of both the electromyographic signal and the burst power of the electroencephalogram (EEG) beta wave signal simultaneously exceed their respective adaptive thresholds, thus effectively eliminating false triggers caused by noise or artifacts in a single channel. The logical expression for dual-channel joint determination is: ,in This is a trigger flag for transient stress response events (a value of 1 indicates a trigger, and a value of 0 indicates no trigger). Let be the burst power of the electromyographic signal at time t. Let be the burst power of the EEG beta wave signal at time t. The first adaptive threshold corresponding to the electromyographic signal. This is the second adaptive threshold corresponding to the EEG beta wave signal. Only when the burst power of both channels simultaneously exceeds their respective thresholds... Only when it is set to 1 can false triggering caused by noise or artifacts in a single channel be effectively eliminated.

[0072] It is particularly important to emphasize that the preset adaptive threshold is not a fixed static constant, but a variable that is dynamically adjusted based on the target user's real-time physiological baseline or historical data. This is because there are significant individual differences in the physiological response amplitude of different users to stimuli of the same intensity, and the baseline level of the same user will also fluctuate under different fatigue or emotional states. If a fixed threshold is used, it is very easy to cause false triggering for sensitive users or missed triggering for insensitive users. In specific implementation, the preset adaptive threshold can be calculated in real time using a sliding window statistical method, for example, taking the mean of physiological signals within a window over a past period plus several times the standard deviation as the dynamic threshold; or using an online learning algorithm to automatically fit a personalized stress judgment boundary based on the user's actual physiological response distribution to various emergencies in historical training. This adaptive mechanism ensures that the safety protection can both keenly capture real danger signals and adapt to the physiological changes of users as they gradually become desensitized or adapt during the training process. Specifically, the dynamic update formula for the preset adaptive threshold is: ,in The adaptive threshold at time t, This is the moving average of the burst power of the target physiological signal over a preset time window (e.g., the past 30 seconds). This represents the sliding standard deviation of the burst power within this time window. This is the sensitivity coefficient (a positive number, such as 3.0). The physical meaning of this formula is: the adaptive threshold is always maintained at a position several standard deviations above the user's current physiological baseline level. When the user is in a calm state, the baseline level is low and fluctuates less, resulting in a correspondingly low threshold and greater sensitivity to sudden events. When the user is in a state of sustained stress, the baseline level rises and fluctuates more, resulting in a correspondingly higher threshold, avoiding frequent false triggers caused by baseline drift. and The update frequency is consistent with the calculation frequency of burst power (e.g., updated every 50 milliseconds) to ensure that the threshold can track changes in the user's physiological state in real time.

[0073] When the scene simplification protocol is triggered, the operations performed to reduce information presentation density, fault complexity, and / or task interference intensity are rapid and significant, designed to instantly reduce sensory input load to disrupt the stress loop. In addition to the sigmoid-based smoothing described in Example 4, the simplification operations here tend to be discrete and large-scale degradation processing. For example, reducing information presentation density can manifest as immediately hiding all non-core auxiliary instruments, minor fault points, or decorative textures in the virtual scene, retaining only the key visual elements necessary for basic operation; reducing task interference intensity can manifest as instantly muting background noise, pausing non-critical alarms, or reducing visual sway effects. Furthermore, the scene simplification protocol may also include proactively adding visual guidance markers, such as projecting bright halos or arrows around key operational objects, to help users in a state of cognitive confusion quickly refocus their attention. The entire response latency from signal detection to scene rendering update should be controlled within milliseconds (e.g., less than 100 milliseconds). The scenario simplification protocol can automatically exit after a preset time (e.g., 5 seconds), or exit when the burst power of each target physiological signal is detected to drop below 80% of its respective threshold, gradually restoring the scenario parameters to the level before triggering.

[0074] To verify the effectiveness of the aforementioned dual-channel joint judgment strategy, the false trigger rate and missed trigger rate were tested under both single-channel and dual-channel triggering configurations. In 100 simulated sudden event tests, the false trigger rate for the single-channel configuration (using only EMG signals) was 18%, and the missed trigger rate was 6%; the false trigger rate for the single-channel configuration (using only EEG beta wave signals) was 22%, and the missed trigger rate was 4%; the false trigger rate for the dual-channel joint configuration (simultaneous judgment of EMG and EEG beta wave signals) was 3%, and the missed trigger rate was 5%. The experimental results show that the dual-channel joint judgment strategy, while maintaining a low missed trigger rate, reduced the false trigger rate from 18% to 22% to 3%, thus improving the specificity of the trigger.

[0075] Based on the above, this embodiment designs a rapid protection mechanism independent of conventional reinforcement learning decision-making for millisecond-level physiological stress responses caused by sudden events. By actively reducing the complexity of the scene at the moment of sudden physiological signal change, it prevents users from panicking or making mistakes due to instantaneous cognitive overload, thus ensuring the safety and psychological comfort of immersive training.

[0076] It should be understood that although electromyography (EMG) and electroencephalography (EEG) signals are preferred examples of target physiological signals in this embodiment, this does not constitute a limitation on the scope of protection of this application. In other embodiments, the target physiological signal may also be a momentary drop in heart rate variability, a steep upward slope in skin conductance, or even a violent jitter of the fixation point or a momentary dilation of the pupil diameter in eye-tracking data, as long as the signal can effectively characterize the user's sudden stress state. Similarly, the specific execution content of the scenario simplification protocol can also be adapted according to the type of virtual scenario. For example, in driving simulation, it may be automatically taking over vehicle control and slowing down to the side of the road; in surgical simulation, it may be pausing bleeding effects and highlighting hemostatic instruments. Regardless of how the specific signal modality or simplification method changes, as long as the core logic of "triggered by physiological mutation, with higher priority than conventional regulation, and immediate degradation of execution" is followed, it falls within the scope of protection of this application.

[0077] Example 6 Based on the same inventive concept, and building upon embodiments 1 to 5, this application also provides an apparatus for implementing the aforementioned dynamic control method for a multi-path accident simulation experience process based on a VR scene. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more apparatus embodiments provided below can be found in the above-described limitations of the dynamic control method for a multi-path accident simulation experience process based on a VR scene, and will not be repeated here.

[0078] In one embodiment, such as Figure 4 As shown, the device includes: a data acquisition module, a cognitive assessment module, an attention assessment module, and a scene adjustment module. Among them: The data acquisition module is used to acquire multimodal physiological state data of the target user in the virtual scene, as well as the current task complexity data of the virtual scene; it also acquires visual gaze data of the target user towards a predefined safe area, and limb operation data for the target action. Specifically, the data acquisition module acts as a convergence and preprocessing unit for multi-source heterogeneous data in the system architecture. Internally, it typically contains multiple sensor interface adapters, which connect to hardware terminals such as EEG acquisition devices, heart rate monitoring devices, skin conductance sensors, eye trackers, and hand motion capture devices, converting raw signals with different sampling rates and communication protocols into a standardized digital stream. Simultaneously, this module also includes a task status monitoring subunit, used to read in real-time information such as task progress, parameter count, and step count output by the virtual scene engine, and encapsulate it into a task complexity data structure. More importantly, the data acquisition module has a built-in time-series alignment buffer. Through interpolation resampling or timestamp synchronization algorithms, it aligns physiological signals, eye-tracking data, operation data, and task data on a unified time base, eliminating asynchronous deviations between multimodal data and providing a high-quality, spatiotemporally consistent input data stream for the subsequent evaluation module. This structured data aggregation design ensures that the device is compatible with various hardware configurations and provides a stable and reliable data foundation for upper-layer algorithms.

[0079] The cognitive assessment module determines the cognitive load assessment results of target users based on multimodal physiological state data and task complexity data through a dynamic coupling model. Specifically, the cognitive assessment module is a specialized computational processing unit internally loaded with pre-trained dynamic coupling model parameters. This module receives aligned data streams from the data acquisition module, first extracting key feature values ​​of physiological signals (such as β-wave power, SDNN index, and SCR index) and task complexity scores, and then substituting these feature values ​​into the calculation formula for the dynamic coupling coefficient for real-time calculation. In terms of hardware implementation, this module can be implemented using a general-purpose processor with a dedicated acceleration library, or using application-specific integrated circuits such as FPGAs or ASICs to meet millisecond-level real-time requirements. The output port of the cognitive assessment module is directly connected to the state input interface of the scene adjustment module, transmitting the calculated dynamic coupling coefficient or the cognitive load classification results derived from it as a quantitative signal characterizing the user's internal physical and mental state. By encapsulating complex nonlinear coupling calculations into an independent module, not only is the system's computational efficiency improved, but it also facilitates subsequent individual updates of model parameters for specific populations or task types without refactoring the entire system.

[0080] The attention assessment module performs spatiotemporal matching analysis on visual gaze data and limb operation data to determine the target user's attention allocation assessment results. Specifically, the attention assessment module constitutes the device's behavioral analysis engine. This module receives two parallel data streams from the data acquisition module: visual gaze trajectory and limb operation trajectory. First, it maintains a spatial mapping table of predefined safe areas internally, associating the coordinates of gaze points in three-dimensional space with key interactive objects. Subsequently, through the spatiotemporal overlap calculation unit, it calculates the cumulative sum of the product of the effective gaze duration percentage and the correct operation duration percentage within a sliding time window, and generates a visual attention deficiency index by combining it with the weight of the current task stage. The processing logic of this module focuses on the joint operation of spatial geometric relationships and time series, and its output is also transmitted to the scene adjustment module in a standardized numerical form. It should be understood that although this embodiment divides cognitive assessment and attention assessment into two independent modules, in other embodiments, they can also be integrated into a single multidimensional state assessment processor, as long as its internal logic can separately complete physiological-task coupling calculation and eye-hand spatiotemporal matching analysis. This modular behavioral analysis architecture enables the device to accurately capture users' implicit attention deficits during complex interactions, providing precise behavioral evidence for adaptive adjustment.

[0081] The scene adjustment module takes the cognitive load assessment results and attention allocation assessment results as state inputs and outputs scene parameter adjustment strategies through a reinforcement learning model. Based on the scene parameter adjustment strategies, it dynamically adjusts the information presentation density, fault complexity, and / or task interference intensity of the virtual scene. Specifically, the scene adjustment module is the decision-making and control center of the entire device. It contains two core sub-units: a reinforcement learning inference engine and a rendering instruction generator. The inference engine receives two-dimensional state signals from the cognitive assessment module and the attention assessment module, combines them with auxiliary information such as the current task completion rate, and calculates the optimal action vector through a pre-trained policy network. This action vector contains continuous control parameters such as information presentation density adjustment, fault complexity adjustment, and task interference intensity adjustment. The rendering instruction generator is responsible for converting these abstract adjustment quantities into specific rendering instructions or script parameters that the virtual scene engine can recognize, such as modifying the visibility flags of UI elements, adjusting the gain values ​​of ambient sound effects, or changing the state machine parameters of fault triggering logic. In particular, when performing information density adjustment, this module integrates a Sigmoid function mapping unit to ensure that the output instructions can achieve a smooth transition rather than a step change. The output of the scene adjustment module is directly connected to the rendering pipeline or logic controller of the virtual scene, forming a complete closed loop from state perception to environmental intervention. This modular design, which separates decision-making from execution, not only ensures the intelligence and continuity of the adjustment strategy, but also enables the device to flexibly adapt to different types of virtual simulation platforms. The core adaptive control logic can be reused across platforms simply by changing the adaptation layer of the rendering instruction generator.

[0082] Through the modular architecture described above, the device provided in this application transforms the abstract adaptive control method into a concrete structured data processing system. Signals flow between modules is achieved through standardized data interfaces, ensuring both the integrity and continuity of the overall functionality, and providing the system with good scalability and maintainability.

[0083] The modules in the aforementioned dynamic control device for multi-path accident simulation experience based on VR scenes can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module. In actual engineering implementation, the functions of multiple modules can be combined and executed by a single hardware unit, or the function of a single module can be broken down and completed collaboratively by multiple distributed units. As long as the overall device can achieve a closed loop of core functions such as data acquisition, cognitive assessment, attention assessment, and scene adjustment, it falls within the protection scope of this application.

[0084] Furthermore, in one feasible implementation, a computer device is also provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores all data required for the device's operation. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a dynamic control method for a multi-path accident simulation experience process based on a VR scene.

[0085] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0086] Example 7 This embodiment provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods in the above method embodiments.

[0087] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the methods in the above method embodiments.

[0088] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0089] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0090] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A dynamic control method for a multi-path accident simulation experience process based on VR scenes, characterized in that, The method includes: Acquire multimodal physiological state data of the target user in a virtual scene, as well as the current task complexity data of the virtual scene; based on the multimodal physiological state data and the task complexity data, determine the cognitive load assessment result of the target user through a dynamic coupling model; The system acquires visual gaze data of the target user toward a predefined safe area, as well as limb movement data of the target action; it performs spatiotemporal matching analysis on the visual gaze data and the limb movement data to determine the target user's attention allocation evaluation result. The cognitive load assessment results and the attention allocation assessment results are used as state inputs, and a scene parameter adjustment strategy is output through a reinforcement learning model. Based on the scenario parameter adjustment strategy, the information presentation density, fault complexity, and / or task interference intensity of the virtual scenario are dynamically adjusted.

2. The method according to claim 1, characterized in that, The multimodal physiological state data includes at least two of the following: electroencephalogram (EEG) signals, heart rate variability (HRV) signals, and skin conductance signals. The task complexity data includes a weighted combination of the number of device parameter reads and the number of process steps.

3. The method according to claim 2, characterized in that, The cognitive load assessment results of the target user are determined through the dynamic coupling model, including: The dynamic coupling coefficient is calculated using the dynamic coupling model described above. The formula for calculating the dynamic coupling coefficient is as follows: in, Let be the dynamic coupling coefficient at time t. The power of the brainwave β wave at time t is extracted based on the aforementioned brainwave signal. The heart rate variability index at time t is extracted based on the new path variability signal. This is a skin conductivity response index at time t based on the signal extracted from the skin point. Score the complexity of the character at time t. , and Preset weighting coefficients; Here is the regularization constant; The cognitive load assessment result is determined based on the dynamic coupling coefficient.

4. The method according to claim 1, characterized in that, Spatiotemporal matching analysis is performed on the visual gaze data and the body movement data to determine the attention allocation evaluation result of the target user, including: Based on the visual gaze data, determine the percentage of visual gaze time of the target user in the predefined safe area; Based on the limb manipulation data, determine the percentage of correct operation time for the target user when performing the correct limb manipulation; The spatiotemporal overlap is calculated based on the visual fixation duration ratio and the correct operation time ratio. The attention allocation evaluation result is determined based on the spatiotemporal overlap.

5. The method according to claim 4, characterized in that, Determining the attention allocation evaluation result based on the spatiotemporal overlap includes: The Visual Attention Deficit Index is calculated using the following formula: in, The visual attention deficit index is mentioned above. The total number of sampling times. For the first The task stage weights corresponding to each sampling time point For the first The percentage of visual gaze time of the target user in the predefined safe area at each sampling time. For the first The percentage of correct operation time for the target user at each sampling time point. For the first The total gaze duration of the target user at each sampling time point For the first The total operation time of the target user at each sampling time point; The visual attention deficit index is used as the result of the attention allocation assessment.

6. The method according to claim 1, characterized in that, The state space of the reinforcement learning model includes the cognitive load assessment results and the attention allocation assessment results; The action space of the reinforcement learning model includes information presentation density adjustment, fault complexity adjustment, and task interference intensity adjustment. The reward function of the reinforcement learning model is constructed based on the attention allocation evaluation result, the cognitive load evaluation result, and the task completion degree, which is used to characterize the overall progress of the current task.

7. The method according to claim 6, characterized in that, The method further includes: When the burst power of the target physiological signal in the multimodal physiological state data exceeds a preset adaptive threshold, a scenario simplification protocol is triggered; the target physiological signal includes the β-band signal in electromyography and / or electroencephalography, and the burst power is the sum of the squares of the sampled values ​​of the target physiological signal within a preset short time window; The scenario simplification protocol includes reducing the information presentation density and / or the task interference intensity of the virtual scenario.

8. A dynamic control device for a multi-path accident simulation experience process based on VR scenes, characterized in that, The device includes: The data acquisition module is used to acquire multimodal physiological state data of the target user in the virtual scene, as well as the current task complexity data of the virtual scene; acquire visual gaze data of the target user towards a predefined safe area, as well as limb operation data for the target action; The cognitive assessment module is used to determine the cognitive load assessment result of the target user based on the multimodal physiological state data and the task complexity data through a dynamic coupling model. An attention assessment module is used to perform spatiotemporal matching analysis on the visual gaze data and the limb operation data to determine the attention allocation assessment result of the target user. The scene adjustment module is used to take the cognitive load assessment results and the attention allocation assessment results as state inputs, and output scene parameter adjustment strategies through a reinforcement learning model; based on the scene parameter adjustment strategies, the information presentation density, fault complexity and / or task interference intensity of the virtual scene are dynamically adjusted.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the dynamic control method for multi-path accident simulation experience process based on VR scene as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the dynamic control method for multi-path accident simulation experience process based on VR scene as described in any one of claims 1 to 7.

Images