Method and apparatus for determining the degree of immersion

Abstract

To improve determination accuracy of determining an immersion degree of a subject on the basis of a brain activity signal from the subject.SOLUTION: An immersion degree determination method detects brain activity signals in a central region and a rear region of the brain of a subject (S1), detects the nictitation of the subject (S2), measures the functional connectivity between the central region and the rear region on the basis of the detected brain activity signals (S3), and determines an immersion degree of the subject on the basis of the presence or absence of the functional connectivity in the first period from the detection of the nictitation to the elapse of the first prescribed time, and the presence or absence of the functional connectivity in the second period from the point after the first period to the elapse of the second period (S4-S10).SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a method for determining the degree of immersion and a device for determining the degree of immersion.

Background Art

[0002] In Patent Document 1, first to third index values indicating the strengths of alpha waves, beta waves, and theta waves are calculated from the brain waves of a driver, and a driver state determination device that determines the driver's state of arousal and concentration based on these first to third index values has been proposed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, since brain waves change due to factors other than concentration and there are individual differences among subjects in the frequency intensity of brain waves, it is impossible to capture the specific brain functions of a subject only by the intensity of the alpha brain wave frequency band, and false determination is likely to occur. The present invention has been made in view of such circumstances, and an object thereof is to improve the determination accuracy for determining the degree of immersion of a subject based on a brain activity signal from the subject.

Means for Solving the Problems

[0005] In one aspect of the present invention, a method for determining the degree of immersion involves detecting brain activity signals in the central and posterior regions of the subject's brain, detecting the subject's blinking, measuring the functional connectivity between the central and posterior regions based on the detected brain activity signals, and determining the subject's degree of immersion based on the presence or absence of functional connectivity during a first period from the time of detection of blinking until a first predetermined time has elapsed, and the presence or absence of functional connectivity during a second period from the time after the first period until a second predetermined time has elapsed. [Effects of the Invention]

[0006] According to the present invention, the accuracy of determining the degree of immersion of a subject based on brain activity signals from the subject can be improved. [Brief explanation of the drawing]

[0007] [Figure 1] This figure shows an example of the hardware configuration of the immersion level determination device according to the embodiment. [Figure 2] This is a schematic diagram of the different regions of the subject's brain. [Figure 3] Figure 1 is a block diagram showing an example of the controller's functional configuration. [Figure 4] This is a schematic diagram of the period for determining whether or not functional connectivity exists. [Figure 5] This is a flowchart of an example of a method for determining the degree of immersion in an embodiment. [Modes for carrying out the invention]

[0008] Embodiments of the present invention will be described below with reference to the drawings. Note that the drawings are schematic and may differ from actual ones. Furthermore, the embodiments of the present invention described below are illustrative examples of devices and methods for realizing the technical concept of the present invention, and the technical concept of the present invention is not limited to the structure, arrangement, etc., of the components described below. The technical concept of the present invention can be modified in various ways within the technical scope defined by the claims described in the patent claims.

[0009] (composition) Figure 1 shows an example of the hardware configuration of the immersion level determination device according to the embodiment. The immersion level determination device 1 is a device that detects brain activity in various regions of the brain of a subject whose immersion level is to be determined using sensors, and determines whether or not the subject is immersed in a specific or unspecified object based on the detected brain activity. The following description will explain an example in which the immersion level determination device 1 is installed in a vehicle to determine the driver's level of immersion (i.e., an example in which the subject is the driver). However, the present invention is not limited to such uses and can be broadly applied to various applications for determining the subject's level of immersion.

[0010] The immersion level determination device 1 comprises a brain activity sensor 2, a display device 3, a speaker 4, and a controller 5. Brain activity sensor 2 is a sensor that detects the brain activity of the driver, who is the subject of the experiment. For example, brain activity sensor 2 may be an electroencephalogram (EEG) sensor that detects the driver's brain waves. Furthermore, when measuring the driver's brain activity using, for example, functional magnetic resonance imaging (fMRI), the brain activity sensor 2 may include a magnetic field application mechanism for applying a magnetic field to the driver's brain and a receiving coil for receiving the response wave. Also, when measuring the driver's brain activity using, for example, functional near-infrared spectroscopy (fNIRS), the brain activity sensor 2 may include a near-infrared light transmitter and a light receiver.

[0011] Display device 3 is an output device that is positioned in a location visible to the driver and presents visual information generated by the immersion level determination device 1. Display device 3 may be, for example, the display screen of a navigation system, or a display device located near the meter in front of the driver's seat or at another location. Speaker 4 is an output device that presents auditory information generated by the immersion level determination device 1. A buzzer may also be provided as a device that outputs auditory information. For example, if the immersion level determination device 1 determines that the driver's level of immersion is high, it may determine that the driver is not concentrating on driving and output a warning display or visual warning message from the display device 3, or output a warning sound or auditory warning message from the speaker 4 or buzzer.

[0012] Controller 5 is an electronic circuit that determines the driver's level of immersion based on detection signals output from the brain activity sensor 2. Controller 5 includes a processor 5a and peripheral components such as a memory device 5b. The processor 5a may be, for example, a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit). The storage device 5b may include semiconductor storage devices, magnetic storage devices, optical storage devices, etc. The storage device 5b may include registers, cache memory, and memory such as ROM (Read Only Memory) and RAM (Random Access Memory) used as main memory. The functions of the controller 5 described below are realized, for example, by the processor 5a executing a computer program stored in the memory device 5b. The controller 5 may be formed by dedicated hardware for performing the information processing described below. For example, the controller 5 may include functional logic circuits set in a general-purpose semiconductor integrated circuit. The controller 5 may also have a programmable logic device (PLD) such as a field-programmable gate array (FPGA).

[0013] When determining the degree of driver immersion, the controller 5 analyzes the detection signal output from the brain activity sensor 2 to detect brain activity signals from a plurality of sub-regions of the driver's brain. For example, as brain activity signals, EEG (ElectroEncephaloGraphy) data, MRI data, and NIRS data from a plurality of sub-regions are detected. Specifically, the controller 5 detects brain activity signals in the central region and the posterior region of the brain. Figure 2 is a schematic diagram of each region of the subject's brain. For example, the controller 5 may detect the brain activity signal of the posterior cingulate cortex (PCC) in the central region of the brain as the brain activity signal. Also, for example, the controller 5 may detect at least one brain activity signal of the left angular gyrus (LAG), right angular gyrus (RAG), left posterior inferior parietal sulcus (LplPS), or right posterior inferior parietal sulcus (RplPS) as the brain activity signal in the posterior region of the brain.

[0014] Then, the controller 5 measures the functional brain connectivity between the central region and the posterior region of the brain based on the brain activity signals in these regions. Functional connectivity is a pattern of statistical dependence relationships between different regions within the nervous system. In the example of Figure 2, the controller 5 measures any one or more of the first connectivity C1, second connectivity C2, third connectivity C3, or fourth connectivity C4 described below as the functional connectivity between the central region and the posterior region.

[0015] The first connectivity C1 is the functional connectivity between the posterior cingulate cortex PCC and the left angular gyrus LAG, the second connectivity C2 is the functional connectivity between the posterior cingulate cortex PCC and the right angular gyrus RAG, the third connectivity C3 is the functional connectivity between the posterior cingulate cortex PCC and the left posterior inferior parietal sulcus LplPS, and the fourth connectivity C4 is the functional connectivity between the posterior cingulate cortex PCC and the right posterior inferior parietal sulcus RplPS.

[0016] The inventors of the present invention have discovered that when a subject is in a state of high immersion in some object, the functional connectivity between the central region and the posterior region of the brain after the subject's blink continues longer than when the immersion level is low. Further, it has been discovered that when the subject is not immersed in anything and is in an unconscious state or a state of low arousal, no functional connectivity occurs between the central region and the posterior region after the subject's blink.

[0017] Therefore, the immersion degree determination device 1 according to the embodiment detects the driver's blink, and based on the presence or absence of functional connectivity between the central region and the posterior region in a first period until a first predetermined time elapses from the time when the driver's blink is detected, and the presence or absence of functional connectivity between the central region and the posterior region in a second period until a second predetermined time elapses from a time after the first period, determines the immersion degree of the subject. Thus, by determining the immersion degree of the subject based on the functional connectivity of specific brain internal positions that occur after the subject's blink, it is possible to reduce misjudgment more than determination based on simply the frequency intensity of brain waves as in Patent Document 1 above. Thereby, the determination accuracy for determining the immersion degree of the subject can be improved.

[0018] FIG. 3 is a block diagram of an example of the functional configuration of the controller 5 in FIG. 1. The controller 5 includes a brain activity analysis unit 10, a blink detection unit 11, a subject state determination unit 12, and an alarm generation unit 13. The brain activity analysis unit 10 analyzes the detection signal output from the brain activity sensor 2, and detects brain activity signals (such as EEG data, MRI data, NIRS data, etc.) from the central region (for example, posterior cingulate cortex PCC) of the driver's brain and brain activity signals from the posterior region (for example, left angular gyrus LAG, right angular gyrus RAG, left posterior inferior parietal sulcus LplPS, right posterior inferior parietal sulcus RplPS), respectively.

[0019] The brain activity analysis unit 10 calculates the functional connectivity between the central region and the posterior region based on the brain activity signal from the central region and the brain activity signal from the posterior region. For example, the brain activity analysis unit 10 calculates the correlation coefficient between brain activity signals from the central region and brain activity signals from the posterior region. For example, the brain activity analysis unit 10 may calculate the correlation coefficient of the normal correlation between time-series data of brain activity signals as the correlation coefficient between brain activity signals.

[0020] Alternatively, the brain activity analysis unit 10 may calculate the coherence between time-series data of brain activity signals as a correlation coefficient between brain activity signals. For example, if the self-spectrums of the time-series data x and y of brain activity signals in the target region for which the correlation coefficient is calculated are denoted as Pxx(f) and Pyy(f), and the cross-spectrum is denoted as Pxy(f), then the coherence Cxy(f) of the brain activity signals in these regions can be calculated using the following formula. Cxy(f)=|Pxy(f)| 2 / (|Pxx(f)|×|Pyy(f)|) Alternatively, the brain activity analysis unit 10 may calculate the autoregressive coefficient of partial-directed coherence (PDC) as the correlation coefficient between brain activity signals.

[0021] The brain activity analysis unit 10 may determine whether or not there is functional connectivity between the central region and the posterior region based on the calculated correlation coefficient. For example, it may determine that there is functional connectivity if the correlation coefficient is above a threshold, and that there is no functional connectivity if the correlation coefficient is below a threshold. For example, a correlation coefficient can be calculated between the brain activity signal from one of the following regions: the left angular gyrus (LAG), the right angular gyrus (RAG), the left posterior inferior parietal sulcus (LplPS), or the right posterior inferior parietal sulcus (RplPS), and the brain activity signal from the central region. If the correlation coefficient is above a threshold, it can be determined that there is functional connectivity between the central region and the posterior region; if the correlation coefficient is below the threshold, it can be determined that there is no functional connectivity.

[0022] Alternatively, for example, the correlation coefficients can be calculated between the brain activity signals from two or more regions among the left angular gyrus LAG, right angular gyrus RAG, left posterior inferior parietal sulcus LplPS, or right posterior inferior parietal sulcus RplPS and the brain activity signals from the central region. If any of these correlation coefficients is above a threshold, functional connectivity between the central region and the posterior region can be determined. If all of these correlation coefficients are below a threshold, functional connectivity can be deemed absent.

[0023] Alternatively, for example, the correlation coefficients may be calculated between the brain activity signals from two or more regions among the left angular gyrus LAG, right angular gyrus RAG, left posterior inferior parietal sulcus LplPS, or right posterior inferior parietal sulcus RplPS and the brain activity signals from the central region. If the statistical values ​​of these correlation coefficients (e.g., mean, median, maximum, or minimum) are above a threshold, functional connectivity between the central region and the posterior region may be determined. If the statistical values ​​of these correlation coefficients are below a threshold, functional connectivity may be determined to be absent. The brain activity analysis unit 10 outputs the measurement results of functional connectivity to the subject state determination unit 12.

[0024] The blink detection unit 11 detects the driver's blinks based on the detection signal output from the brain activity sensor 2. For example, the blink detection unit 11 may detect the driver's blinks by detecting changes in muscle potential detected by the electroencephalogram (EEG) sensor. Alternatively, the blink detection unit 11 may detect the driver's blinks based on electrooculography (EOG) or by analyzing the driver's facial image. The immersion level determination device 1 may include an electrooculography measuring device or a camera that captures the driver's facial image. The blink detection unit 11 outputs the blink detection result to the subject state determination unit 12. The subject state determination unit 12 determines the driver's level of immersion based on the measurement results of functional connectivity between the central and posterior regions of the brain by the brain activity analysis unit 10 and the blink detection results by the blink detection unit 11.

[0025] Specifically, the degree of driver immersion is determined based on the presence or absence of functional connectivity during the first period T1, from the time blinking is detected until the first predetermined time has elapsed, and the presence or absence of functional connectivity during the second period T2, from the time after the first period T1 until the second predetermined time has elapsed. Figure 4 is a schematic diagram of the first period T1 and the second period T2. The first predetermined time and the second predetermined time may be the same length or they may be different lengths.

[0026] For example, the first predetermined time and the second predetermined time may be 150 milliseconds. That is, the first period T1 may be a period that starts from the time a blink is detected and ends 150 milliseconds after the blink is detected. The second period T2 may be a period that starts 150 milliseconds after the blink is detected and ends 300 milliseconds after the blink is detected.

[0027] For example, if the subject state determination unit 12 determines that there is functional connectivity in the second period T2, it may determine that the driver's level of immersion in a particular object is relatively high. For example, the subject state determination unit 12 may determine that the driver's level of immersion is higher than the level of immersion that is permissible while driving (i.e., it may determine that the driver's level of immersion is excessively high). For example, if the subject state determination unit 12 determines that there is functional connectivity not only in the first period T1 but also in the second period T2, it may determine that the driver's level of immersion is relatively high.

[0028] Furthermore, for example, if the subject state determination unit 12 determines that there is functional connectivity in the first period T1 and no functional connectivity in the second period T2, it determines that the degree of immersion in a particular object by the driver is relatively low. For example, the subject state determination unit 12 may determine that the driver's degree of immersion is within the range of acceptable immersion while driving. For example, it may determine that the degree of immersion while driving is appropriate.

[0029] For example, if functional connectivity does not occur between the central region and the rear region after the subject blinks, the subject state determination unit 12 may determine that the driver is not engrossed in anything and is in an unconscious or low-alert state. For example, if there is no functional connectivity during the first period T1 and the second period T2, the subject may determine that the driver is in an unconscious or low-alert state. Refer to Figure 3. The alarm generation unit 13 presents an alarm to the driver via the display device 3, speaker 4, or buzzer, based on the driver's level of immersion determination result by the subject state determination unit 12. At this time, the alarm generation unit 13 may present different alarms to the driver depending on the driver's level of immersion.

[0030] Furthermore, if the subject state determination unit 12 determines that the driver's level of immersion is relatively high, for example, the alarm generation unit 13 generates a first alarm and outputs it from the display device 3, speaker 4, or buzzer. This encourages the driver to concentrate on driving. For example, if the subject state determination unit 12 determines that the driver is unconscious or in a low state of alertness, the alarm generation unit 13 generates a second alarm and outputs it from the display device 3, speaker 4, and buzzer. This helps to restore the driver to an unconscious or low state of alertness. For example, the first and second alarms may be alarm displays or visual alarm messages shown on the display device 3, or alarm sounds or auditory alarm messages output from the speaker 4 or buzzer.

[0031] For example, if the alarm generation unit 13 determines that the driver's level of immersion is relatively high, it may present a stronger alarm (e.g., an emergency alarm) as the first alarm to inform the driver of the urgency. Furthermore, by generating a first alarm that is stronger than the second alarm, the alarm can be noticed even if the driver is highly immersed in the vehicle. For example, the second alarm may generate only an alarm display or visual alarm message to be shown on the display device 3, while the first alarm may generate both an alarm display or visual alarm message to be shown on the display device 3 and an alarm sound or auditory alarm message output from the speaker 4 or buzzer.

[0032] For example, the alarm display or visual alarm message shown on the display device 3 as the first alarm may be made more conspicuous or attention-grabbing than the alarm display or visual alarm message shown as the second alarm. For example, the alarm sound or audible alarm message output from speaker 4 or buzzer as the first alarm may be set to a louder volume or more attention-grabbing nature than the alarm sound or audible alarm message output as the second alarm. For example, the frequency of the sound of the alarm sound or audible alarm message output from speaker 4 or buzzer as the first alarm may be set to a higher frequency than that of the second alarm. On the other hand, if the subject state determination unit 12 determines that the driver's level of immersion is relatively low, the driver can quickly return to a state of concentration on driving, and therefore the alarm generation unit 13 does not generate an alarm signal. In other words, it stops presenting the first alarm signal and the second alarm signal.

[0033] (operation) Figure 5 is a flowchart of an example of a method for determining the degree of immersion according to the embodiment. In step S1, the brain activity analysis unit 10 measures brain activity signals from the central and posterior regions of the driver's brain by analyzing the detection signals output from the brain activity sensor 2. In step S2, the blink detection unit 11 detects the driver's blinking based on the detection signal output from the brain activity sensor 2. In step S3, the brain activity analysis unit 10 calculates the functional connectivity between the central and posterior regions of the driver's brain during the period following the detection of the driver's blink.

[0034] In step S4, the subject state determination unit 12 determines whether or not there is functional connectivity within the first period T1. If there is functional connectivity within the first period T1 (step S4: Y), the process proceeds to step S5. If there is no functional connectivity within the first period T1 (step S4: N), the process proceeds to step S9. In step S5, the subject state determination unit 12 determines whether or not there is functional connectivity within the second period T2. If there is functional connectivity within the second period T2 (step S5: Y), the process proceeds to step S6. If there is no functional connectivity within the second period T2 (step S5: N), the process proceeds to step S8.

[0035] In step S6, the subject state determination unit 12 determines that the driver's level of immersion is relatively high. In step S7, the alarm generation unit 13 generates a first alarm and presents it to the driver. The process then proceeds to step S11.

[0036] If it is determined in step S5 that there is no functional connectivity within the second period T2 (step S5:N), then in step S8 the subject state determination unit 12 determines that the driver's level of immersion is relatively high. In this case, the alarm generation unit 13 does not generate an alarm, and the process then proceeds to step S11. If it is determined in step S4 that there is no functional connectivity within the first period T1 (step S4:N), then in step S9 the subject state determination unit 12 determines that the driver is in an unconscious state or a state of low alertness.

[0037] In step S10, the alarm generation unit 13 generates a second alarm and presents it to the driver. The process then proceeds to step S11. In step S11, the controller 5 determines whether the vehicle's ignition key has been switched to the off position. If the ignition key has not been switched to the off position (step S11:N), the process returns to step S1. If the ignition key has been switched to the off position (step S11:Y), the process ends.

[0038] (Effects of the embodiment) (1) The brain activity sensor 2 and controller 5 detect brain activity signals from the central and posterior regions of the subject's brain. The controller 5 detects the subject's blinking, measures the functional connectivity between the central and posterior regions based on the detected brain activity signals, and determines the subject's level of immersion based on the presence or absence of functional connectivity in the first period T1, from the time of detection of blinking until a first predetermined time has elapsed, and the presence or absence of functional connectivity in the second period T2, from the time after the first period T1 until a second predetermined time has elapsed. This allows for the accurate, real-time determination of the subject's level of immersion based on the functional connectivity between the central and posterior regions of the subject's brain during a predetermined period of blinking.

[0039] (2) Controller 5 may determine that the subject's level of immersion is relatively high if it determines that there is functional connectivity in the second period T2. This allows us to determine if a subject is immersed in a particular object to a relatively high degree of immersion.

[0040] (3) Controller 5 may determine that the subject's level of immersion is relatively low if it determines that there is functional connectivity in the first period T1 but not in the second period T2. This allows us to determine that the subject is immersed in a particular object, but the degree of that immersion is relatively low.

[0041] (4) Controller 5 may determine that the subject is in an unconscious state if it determines that there is no functional connectivity during the first period T1 and the second period T2. This allows us to determine if the subject is in an unconscious state.

[0042] (5) The controller 5 may present different alarms to the subject depending on the degree of immersion. This allows the controller to generate the necessary alarms according to the subject's degree of immersion.

[0043] (6) If the controller 5 determines that functional connectivity exists in the second period T2, it may present a stronger alarm to the subject than if it determined that there was no functional connectivity in the first period T1 and the second period T2. This allows the subject to be alerted to the high level of urgency. Furthermore, it ensures that the driver notices the warning even if they are highly immersed in the driving experience.

[0044] (7) The controller 5 may issue an alert to the subject if it determines that there is functional connectivity in the second period T2, or if it determines that there is no functional connectivity in the first period T1 and the second period T2, but may not issue an alert to the subject if it determines that there is functional connectivity in the first period T1 and no functional connectivity in the second period T2. This allows the system to alert subjects if they become too immersed in a particular object or are unconscious, prompting them to recover from these states, while simultaneously suppressing unnecessary alerts if the subject can quickly recover from a relatively low level of immersion.

[0045] (8) The controller 5 may detect brain activity signals from the posterior cingulate cortex of the subject's brain as central brain activity signals, and may detect brain activity signals from at least one of the following locations in the posterior brain region: the left angular gyrus, the right angular gyrus, the left posterior inferior parietal sulcus, or the right posterior inferior parietal sulcus. In this way, the accuracy of the determination can be improved by assessing the degree of immersion based on brain activity in specific brain regions. [Explanation of Symbols]

[0046] 1…Immersion level determination device, 2…Brain activity sensor, 3…Display device, 4…Speaker, 5…Controller, 5a…Processor, 5b…Storage device, 10…Brain activity analysis unit, 11…Blink detection unit, 12…Subject state determination unit, 13…Alarm generation unit

Claims

1. The brain activity signals of the central and posterior regions of the subject's brain are detected. The blinking of the subject is detected, Based on the detected brain activity signals, the functional connectivity between the central region and the posterior region is measured. The degree of immersion of the subject is determined based on the presence or absence of the functional connectivity during a first period from the time the blink is detected until a first predetermined time has elapsed, and the presence or absence of the functional connectivity during a second period from the time the first period ends until a second predetermined time has elapsed. A method for determining the degree of immersion, characterized by the following:

2. The method for determining the degree of immersion according to claim 1, characterized in that if it is determined that the functional connectivity exists during the second period, it is determined that the degree of immersion of the subject is relatively high.

3. The method for determining the degree of immersion according to claim 1, characterized in that if it is determined that the functional connectivity exists during the first period and that the functional connectivity does not exist during the second period, it is determined that the subject's degree of immersion is relatively low.

4. The method for determining the degree of immersion according to claim 1, characterized in that if it is determined that there is no functional connectivity during the first period and the second period, it is determined that the subject is in an unconscious state.

5. The method for determining the degree of immersion according to claim 1, characterized in that different alarms are presented to the subject according to the difference in the degree of immersion.

6. The method for determining the degree of immersion according to claim 5, characterized in that if it is determined that the functional connectivity exists during the second period, a stronger alarm is presented to the subject than when it is determined that the functional connectivity does not exist during the first and second periods.

7. The method for determining the degree of immersion according to claim 1, characterized in that an alarm is presented to the subject if it is determined that the functional connectivity exists during the second period, or if it is determined that the functional connectivity does not exist during the first and second periods, and no alarm is presented to the subject if it is determined that the functional connectivity exists during the first period and does not exist during the second period.

8. The brain activity signal of the posterior cingulate cortex of the subject's brain is detected as the brain activity signal of the central region. The brain activity signals of the posterior region include detecting at least one brain activity signal from the left angular gyrus, right angular gyrus, left posterior inferior parietal sulcus, or right posterior inferior parietal sulcus of the subject's brain. A method for determining the degree of immersion according to any one of claims 1 to 7.

9. A sensor that detects the brain activity of the subject, An electronic circuit that detects brain activity signals in the central and posterior regions of the subject's brain based on the detection signal of the sensor, detects the subject's blinking based on the detection signal, measures the functional connectivity between the central region and the posterior region based on the detected brain activity signals, and determines the subject's degree of immersion based on the presence or absence of the functional connectivity during a first period from the time of detection of the blink until a first predetermined time has elapsed, and the presence or absence of the functional connectivity during a second period from the end of the first period until a second predetermined time has elapsed. An immersion level determination device characterized by comprising the following:

Images