Systems and methods for determining the perceptual load and stimulus perception level of the human brain

Abstract

The present invention relates to a method for determining the perceptual load of a human brain. The perceptual load is caused by a predetermined task performed by the human. The method comprises the following steps: · measuring the metabolic activity of the human brain using a functional near infrared spectroscopy (fNIRS) sensor device while performing the task, · determining the perceptual load based on the measured changes in the metabolic activity. The present invention also relates to a method for determining the level of a system corresponding to a stimulus perception.

Description

Technical Field

[0001] The present disclosure relates to a system and method for determining the perceptual load of a human brain, in particular the perceptual load caused by a predetermined task performed by the human (e.g., the task of driving a car). The present disclosure also relates to a system and method for determining the perceptual level of the human brain to a second type of stimulus, in particular the perceptual level of a stimulus that is not noticed when the human performs the predetermined task (e.g., driving a car). Background Art

[0002] The ability to identify the perceptual load of an uncontrollable dynamic visual environment and its changes (including unnoticed events) can significantly improve safety (e.g., driving safety) by providing a basis for new designs and strategies for using warning signals. It can also set a standard against which the effectiveness of other metrics (e.g., physiological metrics such as pupil or EEG (electroencephalogram) measurements) for detecting human attention and engagement can be evaluated. In addition, it can also be applied to design optimal strategies for human-machine (e.g., vehicle-driver) interactions for highly automated systems (e.g., vehicles), such as but not limited to so-called takeover request situations, in which the automated control system requests the driver to regain control of vehicle operation.

[0003] Not all information that hits the retina of the eye can be perceived. Whether or not the information is perceived will depend on the strength of the neural signals in the visual cortex (striate and extrastriate cortex). It can also be determined that if the information being attended to has a high perceptual load (i.e., during the performance of a demanding task), then people may not perceive information outside of their focus ("blindness"). These effects of "blindness" are believed to be a major cause of accidents (e.g., car crashes and airplane crashes, etc.).

[0004] Humans have a limited capacity to process perceptual information. Therefore, the level of perceptual load in task processing is an important factor in determining what stimuli can be perceived. When perceptual load is high (e.g., when visual scenes are crowded, or when more demanding perceptual decisions must be made), stimuli outside the focus of attention are less likely to be noticed because there is no spare attentional capacity available for their processing. In extreme cases, this can lead to the phenomenon of inattentional blindness (or deafness), in which participants are completely unable to perceive stimuli due to a lack of perceptual resources, see: Cartwright-Finch U, Lavie N (2007) The role of perceptual load in inattentional blindness, Cognition 102: 321-340.

[0005] On the other hand, under conditions of low load, when there is available capacity, it automatically “spills over” onto the perception of task-irrelevant stimuli, increasing their potential as distractors, see: Forster S, Lavie N (2008) Failures to ignore entirely irrelevant distractors: The role of load in J Exp Psychol Appl 14: 73–83.

[0006] fMRI has been used to measure the effects of the perceptual load of attended task processing on the vascular response to unattended stimuli (BOLD response), see e.g.

[0007] Rees G, Frith CD, Lavie N (1997) Modulating irrelevant motion perception by varying attentional load in an unrelated task. Science 278: 1616–1619.

[0008] Schwartz S, Vuilleumier P, Hutton C, Maravita A, Dolan RJ, Driver J (2005) Attentional load and sensory competition in human vision: modulation of fMRI responses by load at fixation during task-irrelevant stimulation in the peripheral visual field. Cerebral Cortex 15: 770–786, and

[0009] Torralbo A, Kelley TA, Rees G, Lavie N (2016) Attention induced neural response trade-off in retinotopic cortex under load. Sci Rep 6:33041.

[0010] However, fMRI is not a portable wearable technology and therefore does not allow the same applications as fNIRS, which exists as a wearable system. EEG systems have been used to measure the effects of perceptual load on the brain's response to unattended stimuli (see Parks NA, Beck DM, Kramer AF (2013) Enhancement and suppression in the visual field under perceptual load in Psychol 4:275). However, EEG is highly sensitive to the movement of the participant and, although portable, its setup is rather cumbersome (e.g., in some system configurations, conductive glue needs to be used on the head). Therefore, it is not well suited for practical applications outside of a laboratory setting.

[0011] In response to cognitive factors that are indirectly related to perceptual load, some studies have used commercial fNIRS to measure blood oxygenation and deoxygenation levels (HbO2 and HHb). HbO2 and HHb have been shown to be related to the level of visuomotor demand (see: Hosseini SMH, Bruno JL, Baker JM, Gundran A, Harbott LK, Gerdes JC, Reiss AL (2017) Neural, physiological, and behavioral correlates of visuomotor cognitive load in Sci Rep 7:8866) and to cognitive load (see: Fishburn FA, Norr ME, Medvedev A V., Vaidya CJ (2014) Sensitivity of fNIRS to cognitive state and load in Front Hum Neurosci 8:76).

[0012] Neither type of load is related to perceptual load that can lead to effects such as “seeing nothing” (often referred to as “inattentional blindness”). Zhang et al. (2016) and Kojima and Suzuki (2010) correlated levels of HbO2 and HHb with attentional demands in a task, but their attentional demand manipulations were not shown to constitute perceptual load that would be expected to lead to inattentional blindness (as has been shown in cognitive neuroscience studies of perceptual load). They also did not attempt to measure metabolism.

[0013] Harasawa and Shiori (2011) found differences between conditions where HbO2 was tracked in one disc and two discs, but only when presented unilaterally and not bilaterally. This study also did not address the issue of perceptual load leading to inattentional blindness, see: Harasawa M, Shioiri S (2011) Asymmetrical brain activity induced by voluntary spatial attention depends on the visual hemifield: A functional near-infrared spectroscopy study, Brain Cognition 75: 292–298.

[0014] Both fMRI and commercially available fNIR infer neural activity indirectly from measurements of blood oxygenation levels. One study used fMRI (functional magnetic resonance imaging) to measure blood oxygenation levels and blood flow to estimate (rather than measure) the metabolic rate of oxygen in the visual cortex based on a model that approximates other physiological variables with constants for attended and unattended stimulus conditions, see: Moradi F, GT, Buxton RB (2012) Attention strongly increases oxygen metabolic response to stimulus in primary visual cortex. NeuroImage 59: 601–607.

[0015] Heekeren et al. (1999) and Phan et al. (2016a) used developed broadband fNIR to more directly measure neural activity by detecting the oxidation level of cytochrome C oxidase (OxCCO) in mitochondria, which is ubiquitous in neural cells and indicates the level of neural oxygen metabolism. They correlated relative OxCCO levels with the presence of visual stimuli, see:

[0016] Heekeren HR, Kohl M, Obrig H, Wenzel R, von Pannwitz W, Matcher SJ, Dirnagl U, Cooper CE, Villringer A (1999) Noninvasive assessment of changes in cytochrome-c oxidase oxidation in human subjects during visual stimulation. J Cereb Blood Metab 19: 592–603, and

[0017] Phan P, Highton D, Brigadoi S, Tachtsidis I, Smith M, Elwell CE (2016a) Spatial Distribution of Changes in Oxidised Cytochrome C Oxidase During Visual Stimulation Using Broadband Near Infrared Spectroscopy Imaging, pp. 195–201. Advances in Experimental Medicine and Biology.

[0018] Kolyva et al. (2012) and Phan et al. (2016b) showed that levels of oxCCO in the prefrontal cortex are affected by cognitive tasks (e.g., performance on a crossword puzzle solving task versus rest), see: Kolyva C, Tachtsidis I, Ghosh A, Moroz T, Cooper CE, Smith M, Elwell CE (2012) Systematic investigation of changes in oxidized cerebral cytochrome coxidase concentration during frontal lobe activation in healthy adults, Biomed Opt Express 3: 2550–2566, and

[0019] Phan P, Highton D, Lai J, Smith M, Elwell C, Tachtsidis I (2016b) Multi-channel multi-distance broadband near-infrared spectroscopy system to measure the spatial response of cellular oxygen metabolism and tissue oxygenation. Biomed Opt Express 7: 4424.

[0020] Although the neural correlates of perceptual load have been investigated in many studies using a range of neuroimaging techniques, the sources of capacity limitation remain to be understood. Summary of the invention

[0021] Currently, it is still desirable to provide a system and method for determining the perceptual load of the human brain. In addition, it is still desirable to provide a system and method for determining the level of stimulation perception to the human brain, particularly when the human brain perceives (i.e., is exposed to and / or receives signals from) unnoticed stimulation when the human brain performs a predetermined task (e.g., a driving task).

[0022] More specifically, it would be desirable to have a portable, non-invasive system that could detect the perceived level of unnoticed stimulation during driving (or performing another intended task), particularly one that is less sensitive to motion than current EEG-based systems. Such a system could provide a better understanding of the driver's state while driving than current systems using EEG, and could therefore better inform the development of assistive automotive technologies.

[0023] As already noted, Heekeren et al. (1999) and Phan et al. (2016a) correlated the levels of oxCCO in the visual cortex with the presence of visual stimuli. Kolyva et al. (2012) and Phan et al. (2016b) showed that the levels of oxCCO in the prefrontal cortex were affected by cognitive tasks (e.g., performance on a crossword puzzle solving task versus rest). However, none of these publications deals with the cognitive neuroscience of attention, perception, and (more specifically) perceptual load, and cannot be applied to measure and detect perceptual load: this requires expertise in the cognitive neuroscience of attention. On the other hand, the techniques used in cognitive neuroscience cannot measure neurometabolism (instead, they are limited to more indirect measurements of blood oxygenation and deoxygenation levels (HbO2, HHb). However, in existing fNIRS sensor devices that rely on delayed vascular hemodynamic responses, the benefit of measuring neurometabolism compared to oxygenation and deoxygenation (HbO2 and HHb) measures is that it is a more direct indication of neural activity and can therefore provide a faster response to any changes in neural activity.

[0024] Therefore, according to an embodiment of the present disclosure, the present disclosure relates to a method for determining a perceptual load of a human brain. The perceptual load is caused by a predetermined (or predefined) task performed by a human (e.g., driving a car in an autonomous driving mode or other non-driving tasks, such as watching a movie). The method comprises the following steps: while performing the task, using a functional near-infrared spectroscopy (fNIRS) sensor device to measure the metabolic activity of the human brain, and determining the perceptual load based on the measured changes in the metabolic activity.

[0025] By providing such a method, a mobile, kinesthetically flexible (in particular real-time or quasi-real-time) measurement of cognitive conditions including so-called perceptual load can be performed.

[0026] The predetermined tasks may include varying complexities resulting in corresponding varying perceived loads.

[0027] Said complexity may also be referred to as perceptual complexity.In particular, a predetermined task may comprise a predetermined (or predefined) level of variation of perceptual load.

[0028] Typically, in the case where the metabolic activity changes by more than a predetermined amount, with respect to changes in perceived load in a task, it can be determined that the human brain is still able to perceive the task and / or maintain control of the task. In other words, significant changes associated with task changes in load can indicate that the human brain is still able to perceive the task and / or maintain control of the task.

[0029] In the event, for example, that complexity increases and at the same time the measured change in metabolic activity does not exceed a predetermined (or predefined) first upper threshold, it may be determined that the perceptual load caused by the predetermined task exceeds the perceptual load threshold and / or the human brain is not paying attention to the task.

[0030] The threshold is expected to be predetermined in the sense that the threshold is represented by a predetermined value or is predetermined by the brain capacity itself. On the other hand, the predetermined threshold may be predetermined by the brain's perceived load capacity, but the predetermined threshold is ideally lower than the maximum load capacity of the human brain.

[0031] Thus, in the event that the perceptual load is determined to exceed a predetermined threshold, the perceptual load may exceed the capacity of the human brain and / or even be unintentional blindness / deafness to the task.

[0032] For example, in this case, preventive measures can be taken, such as triggering any warning signals or activating an autonomous driving system that takes over driving control.

[0033] If the complexity decreases and at the same time the measured change in metabolic activity does not exceed a predetermined (or predefined) first lower threshold, it can be determined that the human brain is not paying attention to the task.

[0034] Thus, even in the case where the task has a reduced complexity, it can be determined whether the human is paying attention to the task (rather than being sleepy). Performing such a low-complexity task can also be referred to as a baseline activity. However, if the human is not paying attention to the task, there is no metabolic response corresponding to the task. In other words, the metabolic activity does not exceed a predetermined (or predefined) first lower threshold.

[0035] An example may be a vehicle passenger (intended to be the driver) who is viewing a driving scene during automated driving. The driving scene may provide an empty or relatively empty road, i.e., a task with reduced complexity. Using the present disclosure, it may be checked whether the passenger is paying attention to an event in the driving scene, such as a road sign. If it is not determined that metabolic activity exceeds a predetermined (or predefined) first lower threshold when an event occurs (e.g., passing a road sign), it may be determined that the passenger is not paying attention to the driving scene.

[0036] If the complexity decreases and at the same time the measured change in metabolic activity does not decrease below a predetermined (or predefined) first upper threshold, it may be determined that the human brain is not paying attention to the task.

[0037] If the complexity is reduced and the human (e.g., driver) brain is engaged in the task, then the human brain metabolic activity is also expected to be reduced. If it remains at a higher level, it may mean that the human brain is engaged in another task, for example, the driver's brain is engaged in a non-driving task. In general, if the change in complexity exceeds a certain threshold, and the change in metabolic activity also does not exceed a certain threshold, it can be determined that the human brain is not paying attention to the task.

[0038] Therefore, a situation where the perceptual load is too low may lead to automatic processing of stimuli that are not relevant to the task (i.e., automatic processing of stimuli that are not relevant to the task). For example, the human brain may be focused on anything else and no longer pay enough attention to the task. The result may again lead to inadvertent blindness / deafness on the task.

[0039] For example, in this case, preventive measures can be taken, such as triggering any alarm signal to "wake up" the human, i.e., to draw his / her attention back to the task.

[0040] The present disclosure also relates to a method for determining the level of perception of a second type of stimulus by a human brain. The method comprises the following steps:

[0041] During a first time period, exposing the human brain to a predetermined (or predefined) first type of stimulus, in response to which the human performs a predetermined (or predefined) task, thereby inducing a perceptual load in the human brain,

[0042] During the second time period, the human brain is exposed to a predetermined (or predefined) first type of stimulus and a predetermined (or predefined) second type of stimulus perceptible to the human brain,

[0043] measuring metabolic activity of the human brain during the first time period and the second time period using a functional near-infrared spectroscopy (fNIRS) sensor device, and

[0044] A perceived level of the second type of stimulation is determined based on a measured change in metabolic activity during the second time period relative to the first time period.

[0045] The determined perception level may imply both a (positive) case where the second type of stimulus is actually perceived, and a (blind) case where, on the contrary, nothing of the second stimulus is perceived despite being in the field of view (blind case). For example, in the case where the perception level is, for example, "0", it may be determined that there is no perception. In the case where, for example, the perception level is "100", perception may be determined.

[0046] The first time and the second time can have any order and length. They can also be repeated multiple times.

[0047] The load measurement may be understood to be continuous as long as there are times with stimulation of the second type and times without stimulation of the second type.

[0048] In general, the method of determining the perceptual level of the second type of stimulation may be combined with any aspects of the method of determining the perceptual load of the human brain.

[0049] The predetermined task may be a driving task. Note that according to the present disclosure, a driving task is understood to be predetermined (or predefined) in the sense that it includes a type of predetermined task (e.g., steering, accelerating, braking, etc.). In addition, the stimulus to which a human must respond in order to perform the driving task, i.e., the expected stimulus, such as a turn in the road, other vehicles, etc., is also predetermined.

[0050] The first type may be a primary task type and the second type may be a secondary task type.

[0051] In other words, the task to be performed may depend primarily on the stimulus of the first type (i.e., dictated by the stimulus of the primary task type). The stimulus of the second type may have no effect on the task to be performed and / or may require only a limited reaction that is sudden but timely. In one example, the task may be driving (e.g., "driving home after get off work"), and the secondary stimulus type may be the sudden appearance of a pedestrian. For example, in the case where the pedestrian is not crossing the road, his appearance may not affect the driving task. Regardless, it is a relevant precaution for the driver to pay attention to the pedestrian.

[0052] In another example, in the case of autonomous driving, the task may be watching a movie in a vehicle, the first type of stimulus may be the movie being shown, and the second type of stimulus may be the sudden appearance of a pedestrian or any other event in the driving scene outside the vehicle. For example, the vehicle's autonomous driving system may react to the event autonomously. Regardless, checking whether the vehicle occupants (and / or the intended driver) also noticed the event may be a precautionary measure.

[0053] If the measured change in metabolic activity in the second time period relative to the first time period exceeds a predetermined (or predefined) (second upper) threshold, it may be determined that the perception level of the second type of stimulation exceeds a predetermined minimum perception level.

[0054] For example, in this case, no preventive action is required, such as triggering any warning signals or activating an automated driving system that takes over driving control.

[0055] If the measured change in metabolic activity in the second time period relative to the first time period does not exceed a predetermined (or predefined) threshold, it can be determined that the human brain does not perceive the second type of stimulation and / or the human brain does not notice the second type of stimulation.

[0056] For example, in this case, preventive actions can be taken, such as triggering any warning signals or activating an autonomous driving system that takes over driving control.

[0057] The method may further include the steps of generating a predetermined task to be performed by a human being and / or sensing a predetermined task to be performed by a human being.

[0058] For example, in the case of experimental testing, the predetermined task can be completely generated. Likewise, in the case of a driving task, the task may be partially generated / influenced by the movement of the vehicle (e.g., speed and direction). In any case, in the case of a driving task, the task can be sensed, for example, using different sensors (e.g., optical sensors, speed sensors, etc.).

[0059] The method may further comprise the step of exposing the human brain to a first type of predetermined (or predefined) stimulus, in particular comprising a driving scenario, in response to which the human performs a predetermined (or predefined) task, in particular comprising a driving task.

[0060] The first type of stimuli can be noticed by humans when performing a task.

[0061] For example, in case the task to be performed is a driving task, the first type of stimulus may be an attended stimulus of the driving scene, such as a turn in the road, in response to which the human has to perform a steering task.

[0062] The second type of stimulus may not be predetermined by the task so that it is an unnoticed stimulus to a human being when performing the task.

[0063] For example, where the task to be performed is a driving task, the second type of stimulus may be an unnoticed stimulus of a driving scene, such as a soccer ball rolling across the road.

[0064] Given that the total energy expenditure of the brain is essentially constant and that the bioenergetic cost of neural impulses is high, there is a need to efficiently utilize available resources. The inventors have found that the increase in energy expenditure associated with the additional processing demands of high-load (relative to low-load) tasks leaves fewer metabolic resources for processing unnoticed stimuli.

[0065] The first type of stimulus may be variable (ie, may be varied) such that the complexity of the predetermined task varies accordingly.

[0066] The present disclosure further relates to a system for determining the perceptual load of a human brain. The system comprises:

[0067] a functional near-infrared spectroscopy (fNIRS) sensor device configured to measure changes in metabolic activity of the human brain, and

[0068] A control device is configured to determine perceived load based on the measured change in metabolic activity.

[0069] In general, the described system may include one or more system features corresponding to the method features described above, in particular a method of determining the perceptual load of a human brain.

[0070] Perceptual load may be caused by a predetermined (or predefined) task performed by a human.

[0071] The system may further include a generating and / or sensing device configured to generate a predetermined (or predefined) task to be performed by a human being and / or sense a predetermined (or predefined) task to be performed by a human being.

[0072] The present disclosure further relates to a system for determining the level of perception of a stimulus by a human brain. The system comprises:

[0073] A generating and / or sensing device configured to: generate a predetermined (or predefined) task to be performed by a human being and / or sense a predetermined (or predefined) task to be performed by a human being in a first time period and a second time period, and to generate and / or sense a predetermined (or predefined) stimulus perceptible to a human brain in a second time period,

[0074] a functional near infrared spectroscopy (fNIRS) sensor device configured to measure metabolic activity of a human brain during a first time period and a second time period, and

[0075] A control device is configured to determine a perceived level of stimulation based on a measured change in metabolic activity during the second time period relative to the first time period.

[0076] The stimulus ideally corresponds to the "second type of stimulus" in the above cited method. In general, the described system may include one or more system features corresponding to the above method features, in particular methods for determining the level of perception of the second type of stimulus by the human brain.

[0077] Furthermore, the system may also be the described system for determining the perceptual load of the human brain or may be combined with the latter.

[0078] The generating and / or sensing device may be configured to generate and expose the human brain to a first type of predetermined stimulus, in response to which the human has to perform a predetermined (or predefined) task.

[0079] Alternatively or additionally, the generating and / or sensing device may be configured to sense predetermined (or predefined) tasks performed by humans.

[0080] Broadband functional near-infrared spectroscopy (fNIRS) sensor devices can be used to test changes in metabolic activity. The sensor device can be configured to monitor changes in the redox state of cytochrome C oxidase (oxCCO), a mitochondrial enzyme that indicates cellular oxygen metabolism, in addition to changes in the concentrations of oxyhemoglobin (HbO2) and deoxyhemoglobin (HHb).

[0081] The increase in energy demand during neuronal activation is largely covered by oxidative phosphorylation in mitochondria. Therefore, measurement of oxCCO activity enables monitoring of changes in metabolic activity that are associated with increased energy expenditure during neural responses. The present disclosure enables direct comparison of metabolic activity induced by optically identical stimuli at different levels of perceptual load.

[0082] Functional near-infrared spectroscopy (fNIRS) sensors can be configured to measure changes in the redox state of mitochondrial enzymes that are indicative of cellular oxygen metabolism, particularly cytochrome C oxidase (oxCCO).

[0083] The control device may be configured to determine the sensed load based on measured changes in the redox state of measured mitochondrial enzymes indicative of cellular oxygen metabolism, in particular cytochrome C oxidase (oxCCO).

[0084] The measured changes in metabolic activity may indicate increased energy expenditure during neural responses in the human brain.

[0085] The functional near-infrared spectroscopy (fNIRS) sensor may be further configured to measure changes in the concentration of oxyhemoglobin (HbO2) and / or deoxyhemoglobin (HHb).

[0086] The control device may be configured to preselect a region of interest of the brain based on measured changes in concentrations of oxyhemoglobin (HbO2) and / or deoxyhemoglobin (HHb), and to determine the perceived load based on measured changes in metabolic activity in the preselected region of interest of the brain.

[0087] A functional near-infrared spectroscopy (fNIRS) sensor device may be configured to measure changes in metabolic activity in the medial occipital lobe and / or the right medial occipital gyrus of the human brain.

[0088] The control device may be configured to determine the perceptual load based on measured changes in metabolic activity in the medial occipital lobe and / or the right medial occipital gyrus.

[0089] The sensor device may be a broadband functional near infrared spectroscopy (fNIRS) sensor device, for example, configured to sense in the range of 650 nm to 900 nm.

[0090] For example, broadband fNIRS sensor devices for the head can be used to establish new methods to measure and detect the strength of brain signals (HbO2, HHb, oxCCO) to identify the level of perceptual load on the visual cortex during attended tasks and in response to unattended stimuli.

[0091] Broadband fNIRS can measure brain metabolism through cytochrome C oxidase redox state (oxCCO) and blood oxygenation and deoxygenation (HbO2 and HHb) levels.

[0092] Metabolic measures are directly related to neural responses, whereas HbO2 and HHb as measures of vascular responses can only lead to indirect conclusions about neural responses.

[0093] To date, the desired metabolic responses cannot be obtained in any conventional fNIRS because they use only 2-3 wavelengths instead of broadband.

[0094] Measures of reduced metabolism ideally indicate unattended stimulation during performance of the task, implying increased load compared to the attended task condition of reduced load.Preliminary analysis of the new data also indicated increased metabolism in response to attended stimulation.

[0095] The sensor device may include multiple input sources and multiple detectors configured to provide multiple measurement channels across the occipital lobe of the brain.

[0096] The measurement channels may be distributed over the outer extent of the brain's primary visual cortex (V1) and / or across the extrastriate regions of the brain (V2, V3, and V4).

[0097] The measurement channels may be grouped into multiple regions of interest of the brain, particularly including the left / right inferior occipital gyrus (IOG), the left / right middle occipital gyrus (MOG) and / or the medial occipital lobe.

[0098] It is generally noted that determinations according to the present disclosure (eg, determination of perceived load) may also be considered as estimates, since the determined results may not be accurate (eg, in view of actual driving scenarios).

[0099] The present disclosure also relates to a vehicle comprising the control device as described above.

[0100] The vehicle may further comprise a sensor configured to sense the visual driving scene and / or the driver's face, in particular the eyes / pupils, the sensor in particular being an optical sensor, more in particular at least one digital camera.The driving scene may be a first type of stimulus.

[0101] The vehicle may be configured for at least partially automated driving control.The control device may be used in a pre-processing step of driving control to identify situations that imply an increased perceptual load on the driver.

[0102] A vehicle may include hardware and / or software resources configured for at least partial automated driving control.

[0103] The vehicle may include a human-machine interface configured to notify a human of a critical situation, such as an increased sensory load. Thus, the human-machine interface may assist the driver in driving safely.

[0104] Additionally or alternatively, the human-machine interface may be configured for so-called takeover requests, in which the automated control system requests the driver to retake control of the vehicle operation. In this case, it is important that the automated control system is able to reliably determine the perceptual load of the driving scenario in relation to the driving task. Another exemplary configuration would be for the automated control system to take over driving control, for example, if the system identifies that the determined perceptual load exceeds a certain threshold.

[0105] Combinations of elements described above and within the specification are intended to be carried forward unless otherwise contradicted.

[0106] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention.

[0107] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] Figure 1 A block diagram of a system having a control device according to an embodiment of the present disclosure is shown;

[0109] Figure 2 A schematic flow chart illustrating an exemplary method of determining a perceptual load according to an embodiment of the present disclosure is shown;

[0110] Figure 3 A schematic flow chart illustrating an exemplary method of determining a perception level according to an embodiment of the present disclosure is shown;

[0111] Figure 4 A schematic diagram showing an exemplary stimulus according to an embodiment of the present disclosure;

[0112] Figure 5 shows a graph with exemplary levels of metabolic activity measured according to an embodiment of the present disclosure;

[0113] Figure 6 a graph showing an exemplary change in measured metabolic activity due to an additional stimulus of a second type according to an embodiment of the present disclosure, and

[0114] Figure 7 A graph showing exemplary changes in measured metabolic activity due to increasing complexity of a task according to an embodiment of the disclosure. DETAILED DESCRIPTION

[0115] Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Throughout the drawings, the same reference numerals will be used wherever possible to refer to the same or like parts.

[0116] Figure 1 A block diagram of a system 10 having a control device (or electronic device) 1 according to an embodiment of the present disclosure is shown.

[0117] The control device 1 may be connected to or comprise a data storage device (not shown). As described below, the data storage device may be used to store an algorithm for determining the perceptual load or level of stimulus perception.

[0118] The control device 1 may additionally perform other functions in the system 10. For example, the control device may also act as a general ECU (electronic control unit) of the system. The control device 1 may include: electronic circuits, processors (shared, dedicated or grouped), combinational logic circuits, memory for executing one or more software programs, and / or other suitable components that provide the described functionality.

[0119] The control device 1 is also connected to the fNIR sensor device 2. The control device 1 and the sensor device 2 may consist of a system 10. The fNIR sensor device may be a functional near infrared spectrometer. It is desirable to further have a broadband sensor, for example in the range of 650nm to 900nm (i.e., from red light to near infrared light). In this case, the band is the wavelength of light, so broadband refers to a wide (large number of) wavelengths of light. While, for example, the Shimadzu LightNIRS system uses three light bands, the sensor device 10 preferably uses hundreds of light bands.

[0120] The fNIR sensor device 2 can measure changes in the redox state of mitochondrial enzymes, particularly cytochrome C oxidase (oxCCO), which are indicative of cellular oxygen metabolism.

[0121] The fNIR sensor device 2 is configured such that it measures metabolic activity of the brain of a human h. The sensor device can make measurements in particular in the visual cortex, in particular the outer extent of the primary visual cortex (V1) and across the extrastriate areas including V2, V3 and V4.

[0122] The fNIRS device may include 4 input sources and 10 detectors (arranged in a row including 5 detectors, 4 input sources, and 5 detectors), thereby forming a total of 16 measurement channels across the occipital lobe of the entire human. The photodiodes can be placed according to a 10-20 electrode system, and all positioned locations can be digitized to allow channel locations to be drawn on a standardized brain. Based on the anatomical regions associated with the average MNI coordinates, the measurement channels can be divided into 5 regions of interest: left / right inferior occipital gyrus (IOG), left / right middle occipital gyrus (MOG), and medial occipital lobe (based on the AAL digital brain atlas). Functionally, this means that the channels are ideally distributed in the outer range of the primary visual cortex (V1) and across the outer stripe area (including V2, V3, and V4). The setting can also be any other configuration above the occipital cortex or other cortical areas associated with attention and perception (e.g., parietal cortex and auditory temporal cortex).

[0123] Preferred regions of interest (ROIs) of the occipital cortex are the left middle occipital gyrus (MOG); the medial occipital lobe; the right MOG; and / or the right inferior occipital gyrus (IOG).

[0124] The following table indicates examples of suitable anatomical locations for the mean MNI coordinates for each channel:

[0125]

[0126]

[0127] The system 10 is also connected to, or even comprises, a generating and / or sensing device 20. The device 20 is configured to generate and / or sense stimuli to which the human h is exposed. Thus, in case the device 20 is configured as a stimulus generator, the system 10 (in particular the control device 1) controls the device 20. In case the device 20 is configured as a stimulus sensor, the system 10 (in particular the control device 1) receives sensor data of the device 20. A combined configuration of the device 20 as a generator and a sensor is also possible.

[0128] Regarding stimuli, any kind of stimuli are possible, as long as they are perceptible to the human brain. Figure 4 Examples of basic stimulation areas are described, in which the system 10 is primarily used as an experimental test system.

[0129] However, the system can also be used in real-life applications, such as in vehicles. In this case, the first type of stimulus (as explained in more detail later) can, for example, consist of visual and dynamic driving scenes perceivable by the driver (ie, human h).

[0130] The generating and / or sensing device 20 may comprise a sensor configured to sense the visual driving scene, in particular an optical sensor, more particularly at least one digital camera.

[0131] Therefore, multiple sensors can also be used to sense (ie, perceive) the driving scene. For example, two sensors can be used to obtain three-dimensional information of the driving scene and a surround-view type sensor configuration and any combination thereof.

[0132] The sensor data of device 20 can then be made available to system 10 , in particular control device 1 .

[0133] In addition, the second type of stimulus (as described in more detail later) may, for example, consist of a predetermined event in the visual and dynamic driving scene, such as a human passing or approaching the driving lane. In this case, the second type of stimulus may be sensed (e.g., in the same manner as the first type of stimulus) and recognized by the device 20, and corresponding information may be provided to the system 10. Alternatively, the control device may receive the sensor output of the device 20 and recognize the second type of stimulus on its own, for example based on machine learning techniques.

[0134] The second type of stimulus may also include a stimulus generated by the device 20 , for example a predetermined sound or light signal, and which may be perceived by the driver. In this case, the system 10 , in particular the control device 1 , may control the output of the device 20 .

[0135] The generating or sensing device 20 may further be configured to instruct a human to perform a specific task in response to the perception of the first type of stimulus. This is particularly relevant in experimental scenarios, e.g. Figure 4 The scene in.

[0136] Alternatively, the device may determine a task that is suggested (i.e., provided) by the first type of stimulus. This situation is particularly relevant for real-life applications, such as in a moving vehicle. In this example, the first type of stimulus may be a driving scene, and therefore, the relevant task may include driving the vehicle.

[0137] For example, a driving scenario may imply braking (a task). In this case, the task performed by the driver depends on the driving scenario, i.e., the (sensory) stimulus of the first type. In another example, the device 20 may determine, based on the planned destination, that the driver must leave the current road and must make a left or right turn, crossing a lane with vehicles (higher sensory load) or an empty lane (lower sensory load) to reach a new road. In this case, the task that the driver will perform depends on the driving scenario, i.e., the (sensory) stimulus of the first type, and on the external instructions provided to the driver. In summary, the task may be context-dependent.

[0138] Below Figure 2 and Figure 3 The functionality of the control device 1 , in particular how the sensor data received from the fNIR sensor device 2 and the generating or sensing device 20 proceed, is explained in more detail in the context of FIG.

[0139] Figure 2 A schematic flow chart is shown, which illustrates an exemplary method of determining a perceived load according to an embodiment of the present disclosure.

[0140] In step S21, the metabolic activity ma11 of the human brain is measured during a task with a first complexity level c1 performed by the human. In more detail, the human may be exposed to a first type of stimulus and may be instructed to perceive the first type of stimulus and to perform a predetermined task in response thereto. Due to the task to be performed, a perceptual load is induced in the human brain, which results in a measurable metabolic activity.

[0141] In step S22, the process of step S21 is repeated, but the complexity c2 of the task is increased. The increased complexity can be controlled by a correspondingly more complex stimulus of the first type. Again, in step S22, the metabolic activity ma12 is measured.

[0142] Steps S21 and S22 can also have another order. Steps S21 and S22 can also be repeated a number of times and, for example, an average value of the corresponding measurements is determined.

[0143] In step S23, the perceptual load (pl) of the human brain is determined (or estimated) based on the change in metabolic activity between the metabolic activity ma11 measured in step S21 and the metabolic activity ma12 measured in step S22. Typically, a significant change indicates that the human brain is still able to perceive and / or maintain control of the task.

[0144] In step S24 it may be determined whether the (measured) change exceeds a predetermined first threshold.

[0145] If not, then in step S25 it is determined that the perceived load exceeds a predetermined load threshold, such as a predetermined value or a maximum brain capacity. It may also be found that the human brain is not paying attention to the task (e.g., no longer paying attention due to increased complexity). For example, in the case where the method is used in a vehicle and the relevant task is a driving task, the autonomous driving system may take over driving control.

[0146] Figure 3 A schematic flow chart illustrating an exemplary method of determining a perception level of a second type of (unnoticed) stimulus according to an embodiment of the present disclosure is shown.

[0147] In step S31, metabolic activity ma21 is measured during a first type of stimulation. In more detail, a human is exposed to a first type of stimulation s1 and is guided to perceive the first type of stimulation and to perform a predetermined task in response thereto. Due to the stimulation perception and the task to be performed, a perceptual load is induced in the human brain, which results in a measurable metabolic activity.

[0148] In step S32, the process of step S31 is repeated, but additionally exposing the human being to stimuli of the second type s2, preferably simultaneously with the stimuli of the first type s1.

[0149] The second type of stimulus is ideally not related to any predetermined task to be performed by a human being, or may be only weakly related, or may be related to the primary task stimulus as a secondary stimulus. In other words, the second type of stimulus can be perceived by a human being and therefore noticed by a human being without implying that any additional action will be performed relative to the task associated with the first stimulus. The fNIRS sensor can be used to determine whether the stimulus has been noticed. The method can be applied to stimuli that also require attention but have a secondary level of importance to the task-related stimulus.

[0150] Again, in step S32, the metabolic activity ma22 is measured, which, however, is not only influenced by the first type of stimulus (such as ma21 in step S31), but also by the second type of stimulus s2.

[0151] Steps S31 and S32 can also have another order. Steps S31 and S32 can also be repeated a number of times and, for example, an average value of the corresponding measurements is determined.

[0152] In step S33, the perceived level of the stimulus of the second type (s2) is determined (or estimated) based on the change in metabolic activity between the metabolic activity ma21 measured in step S31 and the metabolic activity ma22 measured in step S32. Typically, a reduced change indicates a reduced perceived level of the stimulus of the second type. In other words, an increase in the change indicates that the human brain is still able to perceive otherwise unnoticed stimuli of the second type.

[0153] In step S34 it may be determined whether the (measured) change exceeds a predetermined second threshold.

[0154] If not, it is determined in step S35 that the second type of stimulus is not perceived, or at least not sufficiently perceived. For example, in the case where the method is used in a vehicle and the second type of stimulus is the identification of a potentially dangerous event in a driving scenario, a warning signal may be activated, or as a precaution, the autonomous driving system may take over driving control.

[0155] Figure 4 Schematic diagrams of exemplary stimuli according to embodiments of the present disclosure are shown. In particular, Figure 4 The stimuli used in the experiment are shown, in which participants were exposed to a first type of different stimuli in the form of subsequently appearing crosses of different colors and shapes, and a second type of stimuli in the form of appearing randomly or, for example, on the nth out of every n (n>1) stimulus checkerboard.

[0156] In more detail, participants were presented with a rapid stream of colored crosses with 2 different orientations (vertical / reverse). During high-load blocks (see blocks indicated with an upward arrow), the target was defined by a combination of orientation and color, while during low-load blocks (see blocks indicated by a downward arrow), the target was defined only by color. Note that the visual stimulus could therefore be identical in both conditions, and only the target instruction could vary.

[0157] Each block had a duration of, for example, 25 seconds, followed by a 25-second rest period. On half of the trials, a full-field flickering checkerboard appeared in the periphery, which the participants were primed to ignore.

[0158] To test the hypothesis that changes in oxCCO signal in response to unattended stimuli are modulated by load level, a repeated measures ANOVA (analysis of variance) with two factors Loading and Presence of a distractor (i.e., presence of a second type of stimulus) was performed for each region of interest (ROI).

[0159] Figure 5 A graph with exemplary levels of metabolic activity measured according to an embodiment of the present disclosure is shown. In particular, Figure 5 Four averages (i.e., test results) are shown for each brain region. These correspond to a high-load condition versus a low-load condition, each with and without an additional stimulus of the second type (i.e., a checkerboard). In two regions (medial occipital and right MOG), the relative increase from the checkerboard-absent state to the checkerboard-present state was significantly greater under low perceptual load than under high perceptual load (i.e., the checkerboard was perceived more strongly during the easy task).

[0160] Figure 6 FIG. 1 is a diagram showing an exemplary change in measured metabolic activity due to an additional stimulus of a second type according to an embodiment of the present disclosure. Figure 5 compared to, Figure 6 Only the relative difference between the checkerboard present and absent conditions for high and low loads is shown (ie, checkerboard absent values ​​subtracted from checkerboard present values).

[0161] In particular, Figure 6 Comparison of oxCCO concentration changes associated with the presence of a second type of stimulus (presence minus absence) under high and low perceptual load conditions is shown: Significantly higher signal increases in response to the presence of the checkerboard during low load than high load were found in the medial occipital lobe and right medial occipital gyrus. Error bars represent standard error of the mean.

[0162] The interaction between load and distractor presence (i.e., first-type versus second-type stimulation) was significant in the medial occipital lobe (F(1,12)=9.18, p=.010) and right MOG (F(1,9)=10.39, p=.010). The direction of this interaction was as expected: for both ROIs, the difference in signal change between presence and absence conditions was greater at low load (medial occipital lobe: M=0.029 μM, SD=0.019; right MOG: M=0.029 μM, SD=0.018) than at high load (medial occipital lobe: M=0.018 μM, SD=0.019; right MOG: M=0.011 μM, SD=0.011). Post hoc paired t tests determined that in both ROIs, there was a simple main effect of distractor presence at low load (medial occipital lobe: t(12) = 5.52, p < .001; right MOG: t(9) = 5.29, p = .001) as well as high load (medial occipital lobe: t(12) = 3.40, p = .005; right MOG: t(9) = 3.20, p = .011). The trend of oxCCO signal attenuation for distractor presence versus absence was consistent across all ROIs except left IOG (see Figure 5). Figure 5 ). This indicates that metabolic activity associated with the presence of distractors is reduced under conditions of high perceptual load compared to low load. To the best of the inventors' knowledge, this is the first direct evidence that load-dependent attention modulation affects brain metabolism in the visual cortex.

[0163] This result is consistent with previous fMRI findings that used the blood oxygen level-dependent (BOLD) signal to measure load regulation of neural responses in visual cortical responses (Rees et al., 1997; Schwartz et al., 2005; Torralbo et al., 2016). However, these previous findings are not directly related to brain energy expenditure: the BOLD signal exploits the paramagnetic properties of HHb. During neural activity, the concentration of HHb decreases due to a complex interaction of changes in cerebral blood flow, cerebral blood volume, and cellular metabolism. The significantly larger changes in blood flow are not a direct result of increased energy demand, but are triggered by glutamatergic signals. Importantly, these two processes should not be equated with each other, as the coupling between them is variable in space and time and is affected by neural activation. Therefore, the BOLD signal, which measures HHb levels, is not suitable for testing hypotheses specific to brain metabolism.

[0164] The results described in this disclosure include recent findings by Moradi et al. (2012), who used fMRI to measure the BOLD signal as well as cerebral blood flow (CBF) to mathematically estimate the cerebral metabolic rate of oxygen (CMRO2) when a peripheral chessboard stimulus was attended and unattended. The unattended condition consisted of a central n-back task, while the attended condition required participants to covertly shift their attention to the peripheral chessboard and detect changes in brightness while fixating on the center of the screen. However, in this experiment, eye movements were not controlled, and the difference between the attended and unattended conditions may be due to the stronger visual signal caused by fixating on the chessboard in the attended condition (as opposed to a covert shift of attention).

[0165] In the present disclosure, as mentioned above Figures 4 to 6 In the context of the , participants had to perform a task in the center of the screen in both high-load (“unattended”) and low-load (“attended”) conditions, and in both cases the checkerboard distractor was irrelevant. Furthermore, the fast stimulus presentation rate ensured that fixation was supported.

[0166] As mentioned above Figures 4 to 6The limited capacity model of perceptual processing of load theory is supported by results that focus on metabolism as a possible physiological correlate of capacity limitations for perceptual processing, as described in the context of . The total energy expenditure of the brain remains relatively constant, regardless of mental state. However, neural impulses are metabolically very expensive. This has led to the proposal of a sparse coding model in which resources are saved by using only a few active neurons to represent relevant information. The present disclosure supports such a model because it shows that metabolic activity is significantly reduced under conditions where processing of other stimuli needs to be enhanced due to task demands. In particular, metabolic resources saved by reducing the representation of task-irrelevant stimuli under high load conditions are indeed used to improve the representation of attended, task-relevant stimuli.

[0167] In short, as mentioned above Figures 4 to 6The results described in the context of show that brain metabolism in the striate and extrastriate cortex is significantly attenuated for task-irrelevant stimuli under conditions of high perceptual load compared to low perceptual load. These attention-induced, load-dependent changes in cellular metabolism have been explained within the framework of perceptual load theory (see, e.g., Lavie, Nilli; Tsal, Yehoshua (1994). Lavie, N. (1995) “Perceptual load as a major determinant of the locus of selection in visual attention”, Perception and Psychophysics, 56(2): 183–197. Lavie, N. (2005) Perceptual load as a necessary condition for selective attention, Journal of Experimental Psychology: Human Perception and Performance, 21, 451–468. Lavie, N., Beck, D. M. & Konstantinou, N. (2014) Distracted and confused?: selective attention under cognitive bias, Trends in Cognitive Sciences, 9, 75–82. load (Distracted and confused?: Selective attention under load). Blinded by the load: attention, awareness and the role of perceptual load in Philosophical Transactions of the Royal Society B, 369:20130205, and accommodates a large number of neuroimaging findings that reduce BOLD responses (Rees et al., 1997; Schwartz et al., 2005; Torralbo et al., 2016), neural excitability (Muggleton et al., 2008), and neural field potentials to task-irrelevant stimuli under high load conditions. Importantly, this suggests that metabolism is directly affected by load and may therefore be neurally related to capacity limitations outlined in load theory.Furthermore, this disclosure demonstrates how multi-channel broadband fNIRS may be an advantageous neuroimaging tool as it allows for the distinction between vascular and metabolic factors, which is not possible with measurements such as the BOLD signal.

[0168] Figure 7 A graph showing exemplary changes in measured metabolic activity due to changes in the perceived load of a task performed by a human. Task vs. Figure 4 The task is the same as in , but the cross is larger and patterned. In particular, Figure 7 Shown is a comparison of the change in oxCCO concentration between a low-load task and a high-load task in the absence of a flickering checkerboard (second type of stimulus): significantly higher signal increases were found in response to increased perceived load of the task. Error bars represent standard error of the mean.

[0169] In this example, channel 8 is used for measurement. The measurement channels of the fNIRS system are located between each input source and the detector. In the current exemplary case, channel 8 refers to a location in the occipital lobe (left medial occipital gyrus) where the measurement change between high and low loads reaches statistical significance. Therefore, it is desirable to measure at the occipital lobe.

[0170] Throughout the specification, including the claims, unless otherwise stated, the term "comprising a" should be understood as being synonymous with "comprising at least one". In addition, unless otherwise stated, any ranges set forth in the specification, including the claims, should be understood to include the values ​​at the ends thereof. The specific values ​​of the elements described should be understood to be within acceptable manufacturing or industrial tolerances known to those skilled in the art, and any use of the terms "substantially" and / or "approximately" and / or "typically" should be understood to be within such acceptable tolerances.

[0171] Although the present disclosure has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the present disclosure.

[0172] It is intended that the specification and examples be considered as exemplary only, with the true scope of the disclosure being indicated by the following claims.

Claims

1. A method of determining a perceptual load on the brain of a human being, the perceptual load being caused by a predetermined task performed by the human being, the method comprising: using a functional near infrared spectroscopy sensor device (2) to measure the metabolic activity of the brain of the human being while performing the predetermined task, wherein the predetermined task includes a varying complexity of the perceptual load causing a corresponding variation, determining the perceived load based on the measured changes in metabolic activity, and In case the complexity decreases and at the same time the change in the measured metabolic activity does not fall below a predetermined first upper threshold or does not exceed a predetermined first lower threshold, it is determined that the brain of the human is not paying attention to the predetermined task.

2. The method according to claim 1, further comprising: determining that the human brain is still able to perceive and / or maintain control of the predetermined task if the metabolic activity changes by more than a predetermined amount; as well as In case the complexity increases and at the same time the change in the measured metabolic activity does not exceed the predetermined first upper threshold, it is determined that the perceptual load caused by the predetermined task exceeds a perceptual load threshold and / or the human brain is not paying attention to the predetermined task.

3. The method according to claim 1, comprising at least one of the following: generating the predetermined task to be performed by the human; and The predetermined task performed by the human is sensed.

4. The method according to claim 1, comprising: The human's brain is exposed to a first type of predetermined stimulus, in response to which the human performs the predetermined task.

5. The method according to claim 4, wherein: The first type of predetermined stimulus includes a driving scene, and the predetermined task includes a driving task.

6. A system for determining a perceptual load of a human brain, the system comprising: a functional near infrared spectroscopy sensor device (2) configured to measure changes in metabolic activity of the brain of said human being, and a control device (1) configured to determine the perceived load based on the measured changes in metabolic activity, wherein the perceptual load is caused by a predetermined task performed by the human being, wherein the predetermined task includes a varying complexity causing a corresponding varying perceptual load, and The control device is further configured to determine that the brain of the human is not paying attention to the predetermined task if the complexity decreases and at the same time the change in the measured metabolic activity does not fall below a predetermined first upper threshold or does not exceed a predetermined first lower threshold.

7. The system according to claim 6, Also includes: A generating and / or sensing device (20) configured to: A predetermined task to be performed by the human is generated and / or a predetermined task to be performed by the human is sensed.

Images