System for representing persistent attention of subject in 3D environment and working method thereof

Abstract

The invention discloses a system for representing persistent attention of a testee in a 3D environment and a working method thereof, and the method comprises the steps: obtaining subjective reports, objective behaviors and physiological data of persistent attention of the testee in a cross manner when the testee executes a three-dimensional GNG (GO / No-go) test in a VR head-mounted device; objective behaviors and physiological data of persistent attention of the testee are analyzed, and a hierarchical drift diffusion model and event-related potentials based on the decision field theory are obtained. According to the method, subjective mentality, objective performance and neuroinformatics data can be tracked in a developed 3D GNG task, the attention level of a user in a 3D environment is represented in a multi-level and all-around mode, and a quantitative and qualitative representation method is provided for more common distraction phenomenon research in a super-reality computer simulation world.

Description

Technical Field

[0001] The invention relates to a system for characterizing a subject's sustained attention in a 3D environment and a working method thereof, and belongs to the technical field of virtual reality. Background Art

[0002] Three-dimensional (3D) virtual reality (VR) is a simulation technology that goes beyond traditional flat-panel displays. It uses head tracking and binocular stereo vision to synchronously change and render visual graphics, presenting a 360-degree artificial world from a self-centered perspective. This advanced display technology for immersive human-computer interaction has been applied in education, healthcare, engineering and robotics, and occupational safety.

[0003] In this context, sustained attention, the ability to direct cognition to the external environment and amplify relevant objective cues for a longer period of time to make better behavioral decisions, plays a vital role in the training and learning efficiency of the VR ecosystem. It is well known that it is difficult for humans to focus their thoughts on current considerations, working memory, and further action decisions without interruption, because thoughts always deviate to the inner world irrelevant to the task, that is, mind wandering. In addition, visually induced motion sickness (VIMS), characterized by fatigue, headache, and nausea, usually occurs after users experience VR for a period of time, which will aggravate distraction and cause users to react slowly or impulsively. Sensory conflict is believed to be a possible cause of VIMS, because VR's motion-simulating visual stimulation causes the user's vestibular system to produce neural inputs that do not match the stored motor neural patterns.

[0004] The functions of the prefrontal cortex include execution, high-level cognition, and top-down control of other brain areas for decision-making. When performing tasks continuously, the right ventrolateral prefrontal cortex (rVLPFC) is one of the areas of the human brain that is particularly active when behavior-related events occur, used to capture exogenous cues, update motor actions, and inhibit attention shifts. Previous studies have shown that transcranial direct current stimulation (tDCS) acting on the rVLPFC can have a positive effect on the concentration of subjects when performing two-dimensional (2D) psychological paradigms displayed on paper or computer screens. This proves that activating the rVLPFC can make it easier for focused attention to win the brain resource competition with spontaneous distraction, so as to use more neurophysiological resources to stimulate the attention network and perform external tasks. However, in 3D virtual environments (VEs), it is not clear whether the rVLPFC is also related to the level of sustained 3D attention, and methods for characterizing the level of 3D attention in 3D VEs also need to be developed.

[0005] The Go / No-go (GNG) task is a classic 2D continuous psychology paradigm used to study response inhibition. In this task, subjects need to strictly control infrequent stop stimulus events and inhibit their own response tendencies in order to respond appropriately to different external stimulus events. Precise inhibition is a core element of self-control, supports flexible goal-oriented behavior, and reflects the neural resources invested by users to maintain attention in a constantly changing environment.

[0006] Based on this, how to design an effective psychological paradigm to stimulate and characterize the subject's 3D attention is a technical problem that technicians in this field urgently need to solve. This method of characterizing 3D attention patterns can promote the wider application of VR in brain-computer interfaces and other real-world fields. Summary of the invention

[0007] Purpose: In order to overcome the deficiencies in the prior art, the present invention provides a system and a working method for characterizing the sustained attention of a subject in a 3D environment. In a 3D inhibition task, the user is recorded at different angles and with different properties, and the recorded data of various properties are further modeled and analyzed, which can reflect the neural attention rhythm oscillating between the user in the hyper-realistic VR world and his inner psychology.

[0008] Technical solution: To solve the above technical problems, the technical solution adopted by the present invention is:

[0009] First, a system for characterizing a subject's sustained attention in a 3D environment includes: a VR headset, a keyboard, a conventional electroencephalograph, and a host computer.

[0010] The VR head mounted device is used to display the GNG (GO / No-go) program, and to display a questionnaire while displaying the task program in VR to obtain the subject's subjective report on sustained attention.

[0011] The keyboard is used to monitor the objective behavior of the subject.

[0012] The conventional electroencephalograph is used to measure the physiological data of the subject.

[0013] The host computer is respectively connected to the VR head-mounted device, the keyboard and the conventional electroencephalograph to obtain the subjective report of the subject's sustained attention, and to analyze the objective behavior and physiological data of the subject's sustained attention, so as to obtain the hierarchical drift diffusion model and event-related potential based on the decision field theory.

[0014] As a preferred solution, the conventional electroencephalograph adopts a 32-channel EEG device.

[0015] In the second aspect, a working method of a system for characterizing a subject's sustained attention in a 3D environment specifically includes:

[0016] Step 1: Several 3D face models are extracted from the database as sequence materials for 3D GNG testing.

[0017] Step 2: Implement the three-dimensional GNG test by rendering left and right virtual views of a degree viewing angle in two stereo glasses of the VR head-mounted device, where a is a natural number.

[0018] Step 3: While the subjects are performing the 3D GNG test in the VR headset, cross-acquire the subjects' subjective reports of sustained attention, objective behavioral and physiological data.

[0019] Step 4: Analyze the objective behavioral and physiological data of the subjects' sustained attention to obtain a hierarchical drift diffusion model and event-related potentials based on decision field theory.

[0020] As a preferred solution, the sequence material uses eight 3D face models, and the eight 3D face models include: two happy female faces, two sad female faces, two happy male faces, and two sad male faces.

[0021] As a preferred solution, the three-dimensional GNG test includes:

[0022] At the beginning of the test, the VR headset displayed task instructions reminding the subjects to press the space bar when they saw a sad expression (Go trial), but to avoid any response to happy expressions (No-go trial).

[0023] The formal test includes N sequences, in which the ratio of No-go to Go sequences is 3:7. If no response is detected, each trial is presented for the trial interval. Once the subject presses the space bar, the facial expression disappears immediately. The random uniform jitter algorithm is used to set the trial interval. During the test, the trial information and key response are recorded synchronously, and N is an integer.

[0024] As a preferred solution, the face immersion environment in the three-dimensional GNG test is set to pure black.

[0025] As a preferred solution, the subjective report of the subject's sustained attention is: after every b sequences, the test will be interrupted, and a questionnaire containing two questions will be displayed in the VR headset to record the subject's subjective attention level during the short period of time of these b sequences, where b is an integer.

[0026] As a preferred option, the first question of the questionnaire was “Where did you focus your attention?”, and the subjects selected one of the following options: (1) focused on the task, (2) not focused on the task, (3) blank, (4) don’t remember.

[0027] The second question in the questionnaire assessed the participants' attention on a 5-point scale, where 1 = "not at all focused on the task" and 5 = "completely focused on the task."

[0028] Preferably, objective measures of sustained attention of the subjects include: accuracy and reaction time.

[0029] As a preferred solution, the physiological data of the subject's sustained attention is: physiological electroencephalogram recording, wherein the ground and reference electrodes of a conventional electroencephalograph are located at FPz and Fz, respectively.

[0030] As a preferred solution, the method for obtaining the hierarchical drift diffusion model based on decision field theory specifically includes:

[0031] The results of Go and No-go trials were input into the stimulus encoding function, and the deviations in accuracy and reaction time were estimated.

[0032] The Markov Chain Monte Carlo (MCMC) method was used as the sampling strategy for the deviations of accuracy and reaction time, and the initial c samples were discarded as burn-ins to eliminate model autocorrelation, where c is an integer.

[0033] The Geweke statistic method was applied and the trace, autocorrelation, and marginal posterior of each parameter were plotted to assess the convergence of the MCMC chain.

[0034] After confirming that the model converged correctly, the speed of information extraction from the No-go and Go trial results, the time of pre-stimulus information encoding and motor planning, the amount of evidence required for decision making, and the subjective initial tendency of whether to press the space bar were used as a hierarchical drift-diffusion model based on decision field theory.

[0035] As a preferred solution, the method for acquiring event-related potentials specifically includes:

[0036] The EEG was re-referenced to the Pz channel, and the EEG data were de-noised using a band-pass filter and a notch filter, respectively.

[0037] Cut out the EEG data segment from 200ms before the 3D face display to 800ms after the display.

[0038] Artifacts and noise in EEG data segments were removed by algorithms and manually, and the retained segments were baseline corrected and averaged to obtain ERP waveforms.

[0039] The ERP waveform was unbiasedly obtained using the folded locator method to obtain the time window of the ERP component, and the average amplitude of the ERP waveform in the time window was calculated for robust quantification of the ERP component.

[0040] The P300 in the rVLPFC region was obtained from the ERP component.

[0041] As a preferred solution, P300 is obtained from ERP components, including:

[0042] The P300 amplitudes of the FC6, F4, F8, C4, and T8 channels were averaged within the time range of 270–308 ms to obtain the P300 in the rVLPFC region.

[0043] Beneficial effects: The system and working method provided by the present invention for characterizing the sustained attention of subjects in a 3D environment can track subjective mentality, objective performance and neuroinformatics data in the developed 3D GNG task, characterize the user's attention level in a 3D environment in a multi-level and all-round manner, and provide a quantitative and qualitative characterization method for the study of the more common distraction phenomenon in the hyper-realistic computer simulation world. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of the 3D Go / No-go task in VR.

[0045] Figure 2Schematic diagram of the questionnaire survey on the level of subjectivity in VR.

[0046] Figure 3 Schematic diagram of modeling behavioral performance monitoring and hierarchical drift diffusion model.

[0047] Figure 4 Schematic diagram of the analysis of the event-related potential P300 component. DETAILED DESCRIPTION

[0048] The following is a clear and complete description of the technical solutions in the examples of the present invention in conjunction with the accompanying drawings in the examples of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the protection scope of the present invention.

[0049] The present invention will be further described below in conjunction with specific embodiments.

[0050] Embodiment 1:

[0051] This embodiment introduces a system for characterizing the sustained attention of a subject in a 3D environment, including: a VR head-mounted device, a keyboard, a 32-channel conventional electroencephalograph (EEG device), a host computer, such as Figure 1 shown.

[0052] The VR headset was used to display the GNG (GO / No-go) program, and questionnaires were interspersed with the VR display of the task program to obtain the subjects' subjective reports on sustained attention.

[0053] A keyboard, such as a wireless keyboard, is used to monitor the subject's objective behavior.

[0054] A 32-channel conventional electroencephalograph was used to measure the subjects' physiological data.

[0055] The host computer, such as a desktop computer or a laptop computer, is used to control the VR head-mounted device and record the keyboard monitoring data, store the EEG signal as a data file, and analyze the keyboard monitoring data and the EEG signal data file.

[0056] Among them: the VR headset and the wireless keyboard are connected to a desktop computer to effectively control the VR headset and record the data monitored by the keyboard through the desktop computer; the 32-channel conventional electroencephalograph is connected to a laptop computer to store and write the EEG signals as data files.

[0057] Embodiment 2:

[0058] This embodiment introduces a method for characterizing a subject's sustained attention in a 3D environment, including two stages: training and formal testing of a 3DGNG task. Specifically, it includes:

[0059] Step 1: Several 3D face models are extracted from the database as sequence materials.

[0060] In one embodiment, eight 3D face models are used as sequence materials, including two female happy, two female sad, two male happy and two male sad faces.

[0061] Step 2: Implement the 3D GNG test by rendering the left and right a-degree viewing angle virtual views in the two stereo glasses of the VR headset respectively.

[0062] In one embodiment, the present implementation adopts a 60 degree viewing angle.

[0063] In one embodiment, the three-dimensional vision includes: the face immersion environment is set to pure black. A random number algorithm is used to randomize the test order each time the program is run. During the test, the computer synchronously records the test information and key response.

[0064] In one embodiment, at the beginning of the test, the VR headset displays a task instruction to remind the subject to press the space bar as quickly as possible when seeing a sad expression (Go trial), but to avoid any response to a happy expression (No-go trial).

[0065] In one embodiment, there is a brief training session before the actual test to familiarize the subject with the procedure.

[0066] In one embodiment, there are 400 sequences in the formal test, with a ratio of 3:7 between No-go and Go sequences. If no response is detected, each trial is presented for 0.5 seconds, and the facial expression disappears immediately once the subject presses the space bar. The inter-trial interval is 0.5–1.1 seconds (random uniform jitter algorithm) to avoid a fixed and rigid rhythm that reduces the user's attention level.

[0067] Step 3: While the subjects are performing the 3D GNG test in the VR headset, cross-acquire the subjects' subjective reports of sustained attention, objective behavioral and physiological data.

[0068] In one embodiment, the subject's subjective report of sustained attention is: the subject's subjective perception of his or her own attention level. After every 10 sequences, the test will be interrupted and a questionnaire containing two questions will be displayed in the VR headset to record the subject's subjective attention level in the short period of time of these 10 sequences.

[0069] In one embodiment, the first question of the questionnaire is “Where are you focusing your attention?”, and participants are required to select one of the following options: (1) focused on the task, (2) not focused on the task, (3) blank, (4) don’t remember.

[0070] The second question asked participants to rate their attention on a 5-point scale (from 1 = “not at all focused on the task” to 5 = “completely focused on the task”) to further measure subjective attention levels, e.g. Figure 2 shown.

[0071] In one embodiment, the objective behavior of sustained attention includes behavioral results including accuracy and reaction time (RT).

[0072] In one embodiment, the physiological data of the subject's sustained attention is recorded by physiological electroencephalogram (EEG), wherein the ground and reference electrodes are located at FPz and Fz, respectively.

[0073] Step 4: Analyze the objective behavioral and physiological data of the subjects' sustained attention to obtain a hierarchical drift diffusion model and event-related potentials based on decision field theory.

[0074] In order to record the objective behavioral performance of the subjects in the task, the present invention monitors whether the subjects press the space bar within 1 second after the start of each sequence. In the Go sequence, failure to press the space bar is marked as an incorrect response, while in the No-go sequence, pressing the space bar is marked as an error. RT is defined as the difference between the timestamps of the space bar press and the start of the stimulus. The hierarchical drift-diffusion model (HDDM) based on decision field theory can be applied to deeper behavioral indicator analysis to evaluate the potential dynamic cognitive process in decision-making in attention tasks.

[0075] In one embodiment, a hierarchical drift diffusion model based on decision field theory uses sequential sampling to decompose RT distribution and reaction results into HDDM parameters, and the complete posterior distribution of HDDM parameters is used to evaluate sensory information. Specifically, it includes:

[0076] The results of Go and No-go trials were input into the stimulus encoding function, and the deviations in accuracy and reaction time were estimated.

[0077] The Markov Chain Monte Carlo (MCMC) method was used as the sampling strategy for the deviations of accuracy and reaction time, and the first 500 samples were discarded as burn-in to eliminate model autocorrelation.

[0078] The Geweke statistic method was applied and the trace, autocorrelation, and marginal posterior of each parameter were plotted to assess the convergence of the MCMC chain.

[0079] After confirming that the model converges correctly, the speed of extracting information from the No-go and Go test results (No-go drift rate, Go drift rate: [v nogo ] and [v go ]), the time for pre-stimulus information encoding and motor planning (non-decision time, [NDT]), the amount of evidence required to make a decision (decision threshold, [a]), and the subjective initial tendency to press the spacebar (decision bias, [z]), such as Figure 3 shown.

[0080] In one embodiment, event-related potentials are introduced to characterize the user's attention level from a neurophysiological perspective. The present invention uses a 32-channel EEG system to record and amplify EEG signals. Specifically, it includes:

[0081] EEG data analysis was first re-referenced to the Pz channel and noise was removed using a 2–80 Hz IIR (Infinite Impulse Response) Butterworth bandpass filter and a Parks–McClellan notch filter, respectively.

[0082] Then the EEG data segment from 200 ms before the 3D face display to 800 ms after the display was cut out.

[0083] In order to filter out artifact interference such as eye movements, heartbeats, and head movements, independent component analysis was performed on the EEG data segments. The artifact components were deleted with reference to the ICLabel algorithm, and all data segments were visually inspected. The segments containing residual artifacts and noise were manually removed. The retained segments were baseline corrected and averaged to obtain the ERP waveform.

[0084] The ERP waveform was unbiasedly obtained using the folded locator method to obtain the time window of the ERP component, and the average amplitude of the ERP waveform in the time window was calculated for robust quantification of the ERP component.

[0085] The P300 in the rVLPFC region was obtained from the ERP components, such as Figure 4 shown.

[0086] One embodiment, obtaining P300 from ERP components, specifically includes:

[0087] The P300 amplitudes of FC6, F4, F8, C4, and T8 channels were averaged within the time range of 270–308 ms to obtain the P300 amplitudes of the rVLPFC region, which were used to compare the P300 amplitudes of the Go and No-go stimulation conditions.

[0088] Event-related potentials (ERPs) can be used to quantify stimulus-triggered regional neuronal excitability with fine temporal resolution. Based on the existing ERP literature, a stimulus-locked P300 component associated with the rVLPFC triggers a complete inhibition in the stop-signal response.

[0089] Therefore, P300 was used here to evaluate the functional activation and response inhibition level of rVLPFC.

[0090] According to the above operations, the present invention can track subjective mentality, objective performance and neuroinformatics data in the developed 3D GNG task, characterize the user's attention level in the 3D environment in a multi-level and all-round manner, and provide a quantitative and qualitative characterization method for the study of the more common distraction phenomenon in the hyper-realistic computer simulation world.

[0091] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A system for characterizing a subject's sustained attention in a 3D environment, characterized by: include: VR headset, keyboard, conventional EEG device, host computer; The VR head mounted device is used to display the GNG (GO / No-go) program, and to display a questionnaire while displaying the task program in VR to obtain the subject's subjective report on sustained attention; The keyboard is used to monitor the objective behavior of the subject; The conventional electroencephalograph is used to measure the physiological data of the subject; The host computer is respectively connected to the VR head-mounted device, the keyboard and the conventional electroencephalograph to obtain the subjective report of the subject's sustained attention, and to analyze the objective behavior and physiological data of the subject's sustained attention, so as to obtain the hierarchical drift diffusion model and event-related potential based on the decision field theory.

2. A system for characterizing a subject's sustained attention in a 3D environment according to claim 1, characterized in that: The conventional electroencephalograph uses a 32-channel EEG device.

3. The method of claim 1 or 2, wherein: Specifically include: Step 1: Extract several 3D face models from the database as sequence materials for 3D GNG testing; Step 2: Implement the 3D GNG test by rendering left and right virtual views of a degree viewing angle in two stereo glasses of the VR head-mounted device, where a is a natural number; Step 3: While the subject is performing the 3D GNG test in the VR headset, the subject's subjective report of sustained attention, objective behavior, and physiological data are cross-obtained; Step 4: Analyze the objective behavioral and physiological data of the subjects' sustained attention to obtain a hierarchical drift diffusion model and event-related potentials based on decision field theory.

4. The working method according to claim 3, characterized in that: The sequence material uses eight 3D face models, which include: two happy female faces, two sad female faces, two happy male faces and two sad male faces.

5. The working method according to claim 3, characterized in that: The three-dimensional GNG test includes: At the beginning of the test, the VR headset displayed task instructions reminding the subjects to press the space bar when they saw a sad expression (Go trial), but to avoid any response to a happy expression (No-go trial); The formal test includes N sequences, among which the ratio of No-go to Go sequences is 3:

7. If no reaction is detected, the presentation time of each trial is the trial interval time. Once the subject presses the space bar, the facial expression will disappear immediately. The trial interval time is set using a random uniform jitter algorithm. During the test, the trial information and key response are recorded synchronously, and N is an integer.

6. The working method according to claim 5, characterized in that: In the three-dimensional GNG test, the face immersion environment is set to pure black.

7. The working method according to claim 3, characterized in that: The subjects' subjective report of sustained attention is as follows: after every b sequences, the test will be interrupted, and a questionnaire containing two questions will be displayed in the VR headset to record the subject's subjective attention level during the short period of time of these b sequences, where b is an integer.

8. The working method according to claim 7, characterized in that: The first question of the questionnaire was "Where do you focus your attention?", and the subjects selected one of the following options: (1) focused on the task, (2) not focused on the task, (3) blank, (4) don't remember; The second question in the questionnaire assessed the subject's attention on a 5-point scale, where 1 = "not at all focused on the task" and 5 = "completely focused on the task." 9. The working method according to claim 3, characterized in that: Objective behaviors of subjects' sustained attention include: accuracy and reaction time.

10. The working method according to claim 3, characterized in that: The physiological data of the subjects' sustained attention were: physiological EEG recording, in which the ground and reference electrodes of a conventional EEG were located at FPz and Fz, respectively.

11. The working method according to claim 3, characterized in that: The method for obtaining the hierarchical drift diffusion model based on decision field theory specifically includes: The Go and No-go trial results were input into the stimulus encoding function to estimate the deviations in accuracy and reaction time; The Markov Chain Monte Carlo method was used as the sampling strategy for the deviation of accuracy and reaction time, and the initial c samples were discarded as burn-ins to eliminate model autocorrelation, where c is an integer; The Geweke statistic method was applied and the trace, autocorrelation, and marginal posterior of each parameter were plotted to assess the convergence of the MCMC chain; After confirming that the model converged correctly, the speed of information extraction from the No-go and Go trial results, the time of pre-stimulus information encoding and motor planning, the amount of evidence required for decision making, and the subjective initial tendency of whether to press the space bar were used as a hierarchical drift-diffusion model based on decision field theory.

12. The working method according to claim 3, characterized in that: The method for acquiring event-related potentials specifically includes: The EEG was re-referenced to the Pz channel, and the EEG data noise was eliminated using a bandpass filter and a notch filter, respectively; Cut out the EEG data segment from −200ms before the 3D face display to 800ms after the display; Artifacts and noise in EEG data segments were removed by algorithms and manually, and the retained segments were baseline corrected and averaged to obtain ERP waveforms; The ERP waveform was unbiasedly obtained using the folded locator method to obtain the time window of the ERP component, and the average amplitude of the ERP waveform in the time window was calculated for robust quantification of the ERP component; The P300 in the rVLPFC region was obtained from the ERP component.

13. The working method according to claim 12, characterized in that: Get P300 from ERP components, including: The P300 amplitudes of the FC6, F4, F8, C4, and T8 channels were averaged within the time range of 270–308 ms to obtain the P300 in the rVLPFC region.

Images