Method and system for monitoring a human state on an arousal scale

Abstract

An arousal estimation system is proposed herein that estimates a human psychophysiological state on an arousal scale, using an arousal detection model that outputs a single arousal score characterizing arousal, rather than a particular target state, such as drowsiness, stress, or cognitive load. Based on an estimated arousal score, the human state can be characterized as either within or outside a desired range of arousal levels associated with optimal performance of relevant tasks. The arousal detection model is trained on a wider set of sensor data than may be applicable to a specific mental state. By training the arousal detection model on the wider set of sensor data associated with different arousal levels, an arousal score generated by the arousal detection model may generate a more accurate estimation of the human state than the conventional single state detectors, even when combined.

Description

METHOD AND SYSTEM FOR MONITORING A HUMAN STATE ON AN AROUSAL SCALEFIELD

[0001] The disclosure relates generally to increasing the performance of driver monitoring systems of vehicles.BACKGROUND

[0002] Current driver monitoring systems (DMS) may be used to estimate a psychophysiological state (e.g., drowsiness, stress) of a driver of a vehicle. The estimation may be based on sensors that analyze various aspects or parameters of driver physiology and behavior, such as, for example, head movement, eye gaze, heart rate, and / or other similar aspects or parameters of driver behavior that may indicate or relate to the psychophysiological state of the driver. If the psychophysiological state of the driver can be accurately assessed or predicted, under certain circumstances, a controller of the vehicle may intervene to assist the driver in operating the vehicle. For example, if the DMS detects that the driver is drowsy, a controller of the vehicle may alert the driver via an audio recording, issue a visual or audio notification, or intervene in a different manner.

[0003] However, different psychophysiological states of the driver may be predicted or estimated based on similar sensor data, where sensor data may not adequately distinguish between the different psychophysiological states. For example, in a drowsy state, the driver may yawn frequently, which is accompanied by breathing-related increases in heart rate that could be misinterpreted as an episode of acute stress. As a result, different types of single state detectors working independently may result in competing interpretations of the driver’s psychophysiological state.SUMMARY

[0004] In various embodiments, the issues described above may be addressed by an arousal estimation system, comprising a processor, and a memory storing instructions that when executed, cause the processor to estimate a level of arousal of a human, on a predefined arousal scale, using a model, based on information received from sensors; generate an arousal score of the human based on the estimated level of arousal; detect that the arousal score exceeds a threshold value, and in response, notify and / or perform an intervention with the human.

[0005] It should be understood that the summary above is provided to introduce in simplified form a selection of concepts that are further described in the detailed description. It is not meant to identify key or essential features of the claimed subject matter, the scope of which is defined uniquely by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to implementations that solve any disadvantages noted above or in any part of this disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The disclosure may be better understood from reading the following description of non-limiting embodiments, with reference to the attached drawings, wherein below:

[0007] FIG. 1 is a schematic block diagram of an exemplary vehicle control system;

[0008] FIG. 2 is a flowchart illustrating an exemplary high-level method for estimating an arousal level of a driver of a vehicle on an arousal scale;

[0009] FIG. 3 is a schematic block diagram of an exemplary training system for an arousal estimation model;

[0010] FIG. 4 is a schematic block diagram that shows an exemplary procedure for training the arousal estimation model;

[0011] FIG. 5 A is a flowchart illustrating a first exemplary method for training the arousal estimation model;

[0012] FIG. 5B is a flowchart illustrating a second exemplary method for training the arousal estimation model;

[0013] FIG. 6 shows an exemplary dashboard of a vehicle including a plurality of controls;

[0014] FIG. 7 is a schematic block diagram that shows an exemplary in-vehicle computing system and a control system of a vehicle; and

[0015] FIG. 8 is a graph shown an experimentally-derived human arousal scale, based on performance of a task, as prior art.DETAILED DESCRIPTION

[0016] The following detailed description relates to monitoring a state or level of arousal of a human and intervening with a view to delivering one or more appropriate interventions. For example, the human may be a driver of a vehicle, and the appropriate interventions may be aimed at maintaining or increasing the driver’s performance of driving tasks. Alternatively, the human may’ be a pilot, or an operator of controls of a different system. It should be appreciated that while the disclosed arousal estimation system is described herein with respect to a driverAttorney Docket No. P240449WOof a vehicle, in other embodiments, the arousal estimation system may be used to determine the state of arousal of a human in non-automotive contexts without departing from the scope of this disclosure.

[0017] The disclosure is based on the principle that the arousal level can be determined using externally observable criteria captured in biosignals (e.g., physiological data) acquired from the human. The biosignals may be interpreted in conjunction with other sensor data, such as images acquired via a camera of the vehicle (e.g., a driver monitoring system (DMS) camera of the vehicle). The arousal level may be determined with respect to human arousal data that has been experimentally validated. In particular, the arousal level of the human may be expressed on an arousal scale based on the Yerks-Dodson Law arousal curve, known in the art. The Yerks-Dodson Law arousal curve shows a relationship between a quality’ of a subject's performance of a task with an arousal level of the human subject. An example of the Yerks-Dodson Law arousal curve is shown in FIG. 8, and described in greater detail below.

[0018] For example, a first set of biosignals may indicate drowsiness; a second set of biosignals may indicate a high cognitive load; a third set of biosignals may indicate stress; and so on. A model, such as an Al model, may be trained to estimate or predict the arousal level of the driver based on the biosignals and / or other sensor data. In some examples, the model may be a machine learning (ML) model, such as a convolutional neural network (CNN). In other examples, the model may be a rules-based model, a statistical model, or a different kind of model. The model may output an arousal score representing the arousal level of the human. Based on the arousal score, a suitable intervention may be performed, for example, if the score is above or below threshold scores associated with acceptable performance of a particular task, such as a driving task.

[0019] Various approaches have been taken to detecting psychophysiological states of drivers from biosignals and / or images. A typical psychophysiological state detector (also referred to herein as a state detector) output is normally associated with a particular aspect of the human state, e.g. drowsiness, cognitive load, stress, etc. The state detector may output a number or logical value (1 / 0, Yes / No, True / False) associated with a particular state or state level. For example, for sleepiness detectors. 0 indicates ‘‘not sleepy” (fully awake), and I indicates “drowsy” (falling asleep). In some examples, the state detector may output scores that attempt to quantify a particular psychophysiological state. For example, a drowsiness detector may output a 0.1 to indicate a low estimated level of drowsiness, and a 0.9 to indicate a high estimated level of drowsiness. If the estimated level of drowsiness is above a first threshold, a first intervention may be performed (e.g., an audio alert), and if the estimated level ofAttorney Docket No. P240449WOdrowsiness is below a second threshold, a second intervention may be performed (e.g., a suggestion to listen to calming music).

[0020] Typically, different psychophysiological states like drowsiness and stress are assessed separately, and a conditional algorithm may be applied to determine an overall psychophysiological state of the driver. However, when the overall psychophysiological state of the driver relies on assessments made by independent systems using a same set of sensor data and images, the outputs of the independent systems may not be easily synthesized. For example, in a drowsy state, the driver may yawn frequently, which is accompanied by breathing-related increases in heart rate that could be misinterpreted as acute stress. As a result, conflicting assessments of a driver’s psychophysiological state may be generated by the independent systems, which may recommend or result in different, conflicting interventions.

[0021] To address this issue, an arousal estimation system is proposed herein that estimates an overall psychophysiological state of a driver on an arousal scale, using an arousal detection model that outputs a single arousal score characterizing driver arousal, rather than a particular target state, such as drowsiness or stress. Based on the arousal value, the driver’s psychophysiological state can be characterized as either within or outside a desired range of arousal levels. The arousal detection model may be trained on a wider set of sensor data than may be applicable to a specific psychophysiological state. That is, a first conventional psychophysiological state detector may rely on a first set of sensor data relating to a first target state, and may not consider sensor data outside the first set; while a second conventional psychophysiological state detector may rely on a second set of sensor data relating to a second target state, and may not consider sensor data outside the second set. By training the arousal detection model on the wider set of sensor data associated with different arousal levels, an arousal score generated by the arousal detection model may be a more accurate estimation of the driver’s psychophysiological state than the conventional single state detectors, even when combined.

[0022] State representation based on continuous arousal may be more precise and informative than representations of other state detectors known in the art. Further, an additional advantage of the arousal estimation system described herein is that when the driver’s arousal score deviates from the desired range, interventions can be determined based on a quantifiable amount by which arousal levels are decreased (e.g., fatigue, drowsiness etc.) or increased (e.g., high cognitive load, mental distraction, stress etc.). That is, conventional state prediction / estimation models may output an indication of whether the driver is experiencing drowsiness, but not a quantification of the drowsiness. Even when a quantification of thedrowsiness is generated, the quantification may be different from, or on a different scale from a quantification of a different state, such as stress. In other words, the quantification of the drowsiness may not be correlated with the quantification of the stress, and as a result, determining a suitable intervention may be difficult.

[0023] In contrast, the proposed arousal estimation system generates an arousal score that is a single quantification of the driver’s psychophysiological state (also referred to herein as the driver’s state), encompassing both the drowsiness and the stress. For example, a low arousal score may indicate a high level of drowsiness and a low level of stress, while a high arousal score may indicate a low level of drowsiness and a high level of stress. By generating the single quantification rather than various individual classifications and / or assessments of disparate systems, the assessment of the driver's state (e.g., arousal level) may be more accurate, and the driver’s state may be more accurately distinguished from other states.

[0024] As a result, an intensity of the interventions can vary depending on a deviation of the driver’s arousal score from acceptable thresholds. If the driver’s arousal score is lower than the acceptable thresholds by a first amount, a first intervention may be performed to increase the driver’s arousal. If the driver’s arousal score is lower than the acceptable thresholds by a second, larger amount, a second intervention may be performed to increase the driver’s arousal. For example, the first intervention may include increasing a flow of air of a heating, ventilation, and air conditioning (HVAC) system of the vehicle by a first amount, and the second intervention may include increasing the flow of air of the HVAC system by a second, larger amount.

[0025] Similarly, if the driver’s arousal score is higher than the acceptable thresholds by a first amount, a first intervention may be performed to decrease the driver’s arousal. If the driver’s arousal score is higher than the acceptable thresholds by a second, larger amount, a second intervention may be performed to decrease the driver’s arousal. For example, the first intervention may include decreasing a volume of music playing in a cabin of the vehicle by a first amount, and the second intervention may include decreasing the volume by a second, larger amount. In this way, interventions to maintain the driver with a desired range of psychophysiological states (e.g.. arousal states) may be more successful than in an alternative system that relies on various individual cognitive assessment models.

[0026] Referring now to FIG. 1, a simplified vehicle control system 100 of a vehicle is shown, including a controller 102, a plurality of sensors 120, and a plurality of output devices 130. Controller 102 may include a processor 104. which may execute instructions stored on aAttorney Docket No. P240449WOmemory 106 to generate output signals, such as audio, video, or other electronic control signals, at output devices 130 based at least partly on an output of sensors 120.

[0027] As discussed herein, the memory 106 may include any non-transitory computer readable medium in which programming instructions are stored. For the purposes of this disclosure, the term “tangible computer readable medium” is expressly defined to include any type of computer readable storage. The example methods and systems may be implemented using coded instruction (e.g., computer readable instructions) stored on a non-transitory computer readable medium such as a flash memory, a read-only memory (ROM), a random-access memory (RAM), a cache, or any other storage media in which information is stored for any duration (e.g. for extended period time periods, permanently, brief instances, for temporarily buffering, and / or for caching of the information). Computer memory of computer readable storage mediums as referenced herein may include volatile and non-volatile or removable and non-removable media for a storage of electronic-formatted information such as computer readable program instructions or modules of computer readable program instructions, data, and so on that may be stand-alone or as part of a computing device. Examples of computer memory may include any other medium which can be used to store the desired electronic format of information and which can be accessed by the processor or processors or at least a portion of a computing device.

[0028] The one or more sensors 120 of the vehicle may include one or more vehicle system sensors 150. Data outputted by vehicle system sensors 150 may be an input into controller 102. Vehicle system sensors 150 may include, for example, engine speed and / or wheel speed sensors, which may indicate a speed of the vehicle or used to calculate acceleration of the vehicle; pedal position sensors that indicate, for example, a position of an accelerator pedal or brake pedal of the vehicle; gear sensors that indicate a selected gear of the vehicle; and / or other types of sensors used to detect operating conditions of the vehicle.

[0029] Vehicle system sensors 150 may also include one or more in-cabin sensors 152 arranged within a cabin of the vehicle. The one or more in cabin sensors 152 may include one or more cameras 154, such as a dashboard camera, which may be used to collect images of the driver of the vehicle for further processing. In particular, the images may be used to estimate a level of arousal of the driver, as described in greater detail below. The one or more in-cabin sensors 152 may include ambient temperature sensors that indicate a temperature of the cabin. The one or more in-cabin sensors 152 may include one or more biosensors 156 arranged and / or positioned to be in contact with a skin of the driver, which may receive biosignals from the driver. In various embodiments, the one or more biosensors 156 may be arranged on a steeringAttorney Docket No. P240449WOwheel where hands of the driver may be positioned. The one or more biosensors 156 may include photoplethysmogram (PPG) sensors that sense a flow of blood of the driver, which can be used to measure a heart rate of the driver. The one or more biosensors 156 may include galvanic skin response and / or other skin potential sensors, which may indicate heartbeat information, glandular activity, etc.

[0030] The one or more in-cabin sensors 152 may include one or more microphones 158 arranged on a dashboard of the vehicle and / or a different part of the cabin of the vehicle, which may be used to receive audio input from the driver and / or to determine a level of noise within the cabin and / or generate contextual data based on audio signals detected within the cabin.

[0031] It should be appreciated that the example sensors described herein are for illustration, not limitation, and the one or more in-cabin sensors 152 may include other types of sensors (seat sensors, humidity sensors, vibration sensors, etc.) without departing from the scope of this disclosure. The in-cabin sensors and environment of the vehicle is described in greater detail in reference to FIGS. 6 and 7 below.

[0032] The one or more sensors 120 of the vehicle may include one or more external sensors 159. and sensor data of external sensors 159 may be an input into controller 102. External sensors 159 may include, for example, one or more external cameras, such as a front end camera and a rear end camera; radar, lidar, and / or proximity sensors of the vehicle, which may detect a proximity of objects (e.g., other vehicles) to the vehicle; sensors of a windshield wiper, lights, and / or a sunroof, which may be used to determine an environmental context of the vehicle; and / or sensors of one or more indicator lights to the vehicle, which may be used to determine a traffic scenario of the vehicle. The environmental context and the traffic scenario may include information used to estimate the level of arousal (e g., cognitive load, stress) of the driver.

[0033] The one or more sensors 120 may include a DMS 171. DMS 171 may monitor the driver to detect or measure aspects of a psychophysiological state of the driver, for example, via a dashboard camera of the vehicle, or via one or more sensors arranged in the cabin of the vehicle. Biometric data of the driver (e.g.. vital signs, galvanic skin response, and so on) may be collected from a sensor of a driver’s seat of the vehicle, or a sensor on a steering wheel of the vehicle, or a different sensor in the cabin. DMS 171 may analyze dashboard camera data, biometric data, and other data of the driver to generate an output.

[0034] Controller 102 includes an arousal estimation system 108, which may monitor an arousal level of the driver based on data received from sensors 120. Arousal estimation system 108 may include an arousal estimation model 110, which may take as input data received fromAttorney Docket No. P240449WOsensors 120, and output an arousal score of the driver, as described in greater detail below in reference to FIG. 2. In various embodiments, arousal estimation model 110 is a ML model trained on training data, and is described accordingly herein. However, it should be appreciated that in some examples, arousal estimation model 110 may be a rules-based model, such as a decision tree, or a different type of model (e.g., statistical model, Bayesian model, etc.). Arousal estimation system 108 may include an arousal threshold detector 112, which may detect when an arousal score estimated by arousal estimation model 110 decreases below a first threshold arousal level (e g., a lower arousal level) or increases above a second threshold arousal level (e.g., a higher arousal level). The first and second threshold arousal levels may define a desired range of a driver’s arousal score, as explained in greater detail below. Arousal estimation system 108 may include an intervention model 114, which may take as input an arousal score of a driver, and output one or more actions to take to increase or decrease the arousal of the driver, such that the driver may be maintained within the desired range of arousal.

[0035] Arousal estimation system 108 may also take as input route data of the vehicle received from navigational system 160, such as, for example, a type of road or roads on a route of the vehicle, an estimated time to reach a destination of the vehicle, and / or a driving environment of the vehicle (e.g., rural, city, highway, etc.). In some embodiments, the inputs into arousal estimation model 110 may include elements of the route data.

[0036] Output devices 130 include at least one or more speakers 132, a display of an in-vehicle infotainment (IVI) system 134, and a plurality of cabin controls 136. Output devices 130 may be relied on by intervention model 114 to intervene when the arousal level of the driver decreases below the first threshold arousal level or increases above the second threshold arousal level. For example, arousal estimation system 108 may actuate one or more cabin controls 136, such as a temperature control, an ambient lighting control, etc., to reduce or increase the driver’s arousal level.

[0037] As an example, the driver may have a low arousal level when operating the vehicle. At the low arousal level, the driver may experience drowsiness. Based on data received from sensors 120, arousal estimation model 110 may output an arousal score of the driver. Arousal threshold detector 112 may determine that the arousal score is below the first threshold arousal level. As a result of being below the first threshold arousal level, a suitable intervention may be determined by intervention model 114. For example, cabin controls 136 may be adjusted to decrease a temperature of the cabin. The adjustment of the temperature may be proportional to a difference between the arousal score outputted by the arousal estimation model and the first threshold arousal level. By decreasing the temperature of the cabin, the arousal level of theAttorney Docket No. P240449WOdriver may increase above the first threshold arousal level, reducing the drowsiness of the driver and increasing a safety of the driver and the vehicle.

[0038] Referring now to FIG. 2, an example high-level method 200 is shown for monitoring an arousal level of a driver of a vehicle, and intervening if the arousal increases or decreases to a level at which driver safety may be affected. Method 200 may be performed by an arousal estimation system of the vehicle, such as arousal estimation system 108 of FIG. 1. Instructions for performing method 200 may be stored in a memory of the vehicle and executed on a processor of the controller, such as memory 106 and processor 104, respectively.

[0039] At 202, method 200 includes estimating and / or measuring vehicle operating conditions. For example, the vehicle operating conditions may include, but are not limited to, a status of an engine of the vehicle (e.g., whether the engine is switched on), and an engagement of one or more gears of a transmission of the vehicle (e.g., whether the vehicle is moving). Vehicle operating conditions may include engine speed and load, vehicle speed, and other operational parameters of the vehicle. In one example, the vehicle is a hybrid electric vehicle, and estimating and / or measuring vehicle operating conditions includes determining whether the vehicle is being powered by an engine or an electric motor. In another example, the vehicle can be operated autonomously, and estimating and / or measuring vehicle operating conditions includes determining whether the vehicle is being operated by a driver or being operated in an autonomous mode. Estimating and / or measuring vehicle operating conditions may also include detecting whether there are passengers in the vehicle. A number of passengers in the vehicle may be detected by sensors of vehicle (e.g., vehicle system sensors 150), such as weight sensors of passenger seats of the vehicle, or other sensors.

[0040] At 204, method 200 includes collecting arousal estimation data of the driver, where the arousal estimation data is data used by arousal estimation model to estimate the level of arousal of the driver. The arousal estimation data may include data received from sensors 120 of FIG. 1, including vehicle system sensors 150, in-cabin sensors 152, external sensors 159, and DMS 171. Specifically, at 206, collecting the arousal estimation data of the driver includes acquiring biosignal (e.g.. physiological) data of the driver. The biosignal data may be collected via biosensors (e.g.. biosensors 156) arranged on a steering wheel of the vehicle, or in a different location. The biosensors may be positioned in skin-contact w ith the driver, or in some examples, physiological biosensor data such as sonograms may be acquired without direct skin contact. For example, remote PPG data may be acquired from images of skin captured via incabin cameras.

[0041] At 208, collecting the arousal estimation data of the driver may include acquiring camera images of the driver. The images may be captured of the driver by an in-cabin camera such as a dashboard camera (e.g., cameras 154). In some embodiments, the images may be captured by a camera of or received from a DMS system of the vehicle. In some examples, the camera images may be processed by the DMS system or a different processing component or model of the vehicle. For example, the images may be processed to detect a heartbeat of the driver. The images may be processed by an eye-gaze detector that determines a focus of the driver.

[0042] At 210, collecting the arousal estimation data of the driver may include collecting other t pes of data that may be related to the arousal of the driver. Behavioral events, or performance metrics may be collected and / or calculated. Behavioral events may include when a head of the driver falls when the head is rolled over in either direction; voluntary blinking, when the driver blinks one or several times with effort for no longer than a threshold time (e.g., three seconds); reseating or movements unrelated to driving (subject warms up, stretches, massages some parts of the body etc ); rubbing eyes, eyelids, and an area around the eyes; headshaking right to left to wake up; mouth movements including nudging, licking, stretching, chewing, curling the lips sideways, biting the lips, jaw movements etc ); and so on. These can be considered as signs of drowsiness or behavior directed toward awakening. Detection of these behavioral events and their calculation can be useful for arousal level evaluation because they are primary signs of decreasing arousal.

[0043] Additionally, different driving performance metrics can be sensitive to different non-optimal states of the driver, such as drowsiness, cognitive load and stress, that are represent different levels of arousal. Lower arousal levels associated with drowsiness and fatigue may produce more changes related to lateral control of the vehicle, while higher arousal levels triggered by high CL and stress may have greater effects on longitudinal control and speed regulation.

[0044] At 212, method 200 includes extracting relevant parameters from the collected arousal estimation data, within a predetermined analysis window. The analysis window may correspond to a period of time during which parameters may be aggregated for submission to the arousal estimation model. For example, the analysis window may have a duration of 30 seconds, where relevant parameters collected from the arousal estimation data during 30 seconds may be inputted into an arousal estimation model during one feedforward cycle, to generate an output relevant to the analysis window. The extracted parameters may include data generated during processing of raw biosignal or camera data. For example, the extractedAttorney Docket No. P240449WOparameters may include a heart rate of the driver extracted from PPG sensor data, eye gaze data extracted from camera images, etc.

[0045] At 214, method 200 includes estimating the arousal level of the driver based on the extracted relevant parameters of the arousal estimation data, using an arousal estimation model (e.g., arousal estimation model 110 of FIG. 1). The arousal estimation model may output an arousal score of the driver. The arousal score may indicate or summarize the arousal of the driver. The arousal score may be a value expressed within a range of values, e.g.. a predefined arousal scale. For example, an arousal of 10 may indicate a maximum arousal of the driver. An arousal score of zero may indicate a minimum arousal of the driver. An arousal score of 5 may indicate an optimal level of arousal for performing driving tasks. In various embodiments, the arousal scale may be based on the Yerks-Dodson Law arousal curve.

[0046] FIG. 8 shows an exemplary arousal scale 800 based on the Yerks-Dodson Law arousal curve, as prior art. Arousal characterizes an overall level of psychophysiological activation and alertness. The Yerkes-Dodson law describes the relationship between arousal and performance: best performance corresponds to optimal levels of arousal, performance declines as arousal increases or decreases, compared to the optimal levels. Drowsiness, fatigue, mental workload / cognitive load, and stress are among the states that have been of particular interest in the field of human state detection during activities requiring attention and engagement, such as driving a car, because they are known to reduce performance. These and other states can be aligned along the arousal scale from low to high, with a predicted dynamics of performance corresponding to each state. Detecting ranges of non-optimal states on the arousal scale may facilitate creating appropriate interventions aimed at maintaining acceptable driving performance and promoting road safety.

[0047] Arousal scale 800 includes a plot 802 of a performance quality of a human subject on a task, on the y axis, against an arousal level of the human subject, on the x axis. The arousal values of the x-axis are divided into five categories, and each of the arousal categories has a corresponding range of arousal values. A first arousal category 810 corresponds to drowsiness, which encompasses a first arousal range 820 of arousal levels / scores. A second category 812 corresponds to fatigue, which encompasses a second arousal range 822. A third arousal category 814 corresponds to an optimal arousal state, which encompasses a third arousal range 824. A fourth arousal category 816 corresponds to cognitive load, which encompasses a fourth arousal range 826. A fifth arousal category 818 corresponds to stress, which encompasses a fifth arousal range 828.Attorney Docket No. P240449WO

[0048] Thus, the human subj ect may be categorized into one of the arousal categories based on a detected, estimated, or predicted arousal level of the human expert. In the depicted example, the arousal scale classifies arousal levels into a scale from 0.0 to 10.0. Each of the arousal ranges 820-828 may encompass ranges of arousal levels of different sizes. For example, first arousal range 820 may encompass arousal levels from 0.0 to 2.0; second arousal range 822 may encompass arousal levels from 2.0 to 4.5; third arousal range 824 may encompass arousal levels from 4.5 to 6.0; fourth arousal range 826 may encompass arousal levels from 6.0 to 7.5; and fifth arousal range 828 may encompass arousal levels from 7.5 to 10.0. In other embodiments, the thresholds of each arousal range may differ.

[0049] It should be appreciated that arousal ranges and categories / classifications determined by the arousal ranges may differ in other embodiments. Herein, states of drowsiness, cognitive load and stress are described; however, in different embodiments, different descriptors or states may be described, for example in terms of cognitive or mental distraction, etc. These categories may be considered qualitative terms used to describe variation in a human psychophysiological state along the arousal scale that is our tool for state quantification.

[0050] As shown by plot 802, the quality of performance of the task by the human subject is highest when the arousal level of the human subject is within third arousal range 824 (e.g., the optimal arousal state). As the arousal level of the human subject decreases, the human subject experiences fatigue (e.g., in second arousal range 822), and subsequently drowsiness (e g., in first arousal range 820), which compromises their performance of the task. As the arousal level of the human subject increases, the human subject experiences cognitive load (e.g., in fourth arousal range 826), and subsequently stress (e.g., in fifth arousal range 828), which similarly compromises their performance of the task.

[0051] Thus, the Yerks-Dodson Law arousal curve demonstrates both how arousal level can be a predictor of performance on driving tasks, and how the performance of the driving tasks can be increased by manipulating a driver’s arousal level. When drowsiness or fatigue are detected in a driver, increasing the driver’s arousal level may result in a higher performance of the driving tasks. Similarly, when stress or cognitive load are detected in the driver, decreasing the driver’s arousal level may result in a higher performance of the driving tasks. Put another way, declines in performance observed due to drowsiness may be preceded by fatigue, and declines in performance observed due to stress may be preceded by cognitive load. Thus, detecting fatigue and cognitive load may allow interventions to be performed earlier, prior to affecting driver safety.

[0052] Returning to method 200, at 216, method 200 includes determining whether the arousal is less than a first threshold. The first threshold may be a value at which the driver may begin to experience fatigue, which may reduce a safety of the driver and the vehicle. For example, the first threshold may be 4.5, where if the arousal score of the driver decreases below 4.5, then the driver begins to experience fatigue, and a further decline may indicate drowsiness.

[0053] If at 216 it is determined that the arousal score of the driver is less than the first threshold, method 200 proceeds to 218. At 218. method 200 includes performing an intervention to increase the arousal level (and score) of the driver. By performing the intervention when the arousal score of the driver is less than the first threshold, the intervention may be initiated when the driver is fatigued, and prior to detection of drowsiness. Thus, an advantage of the arousal estimation system described herein is that state-based interventions may be performed earlier than in other competing state detectors that detect drowsiness. By performing the interventions earlier, a safety of the driver may be increased.

[0054] The intervention may include various components. For example, the intervention may include notifying the driver via an audio signal, an audio alert, and / or visual notifications via a speaker or IVI system of the vehicle (e.g.. speakers 132 and IVI system 134, respectively). The controller may further provide to the driver one or more suggested adjustments to the cabin controls for selection by the driver. For example, the controller may suggest a different type of audio content that may be associated with a more energetic mood.

[0055] Additionally or alternatively, the intervention may include automatically adjusting one or more settings of the vehicle. For example, a sensitivity of a braking control system may be adjusted. The intervention may be selected using an intervention model, such as intervention model 114 of FIG. 1. The intervention model may take the arousal score of the driver as input, and may output a type of intervention.

[0056] For example, the intervention may include adjusting cabin controls of the vehicle (e.g., cabin controls 136) to increase the arousal of the driver. In some examples, adjusting the cabin controls may include adjusting an interior lighting of the cabin. For example, the interior lighting may be increased (e.g., brightened). Further, light with a frequency closer to a blue spectrum may induce focus and concentration in the driver to a higher degree than light with a frequency further from the blue spectrum. Therefore, the frequency of the interior lighting of the cabin may be adjusted towards the blue spectrum to increase the focus and concentration of the driver.

[0057] Adjusting the cabin controls to increase the arousal of the driver may also include adjusting a temperature of the cabin. For example, the temperature of the cabin may be reduced,Attorney Docket No. P240449WOwhich may raise the arousal score of the driver and mitigate drowsiness or fatigue that the driver may be experiencing. Similarly, an adjustment may be made to an HVAC setting, such as increasing a fan speed to increase a flow of air into and / or around the cabin.

[0058] In some embodiments, adjusting the cabin controls to increase the arousal of the driver may include altering one or more settings of an audio system of the vehicle, or prompting the driver to alter the one or more settings of the audio system. Certain types of audio content may be more calming and conducive to increasing the focus of the driver than other types of audio content. For example, the arousal estimation system may adjust the cabin controls to decrease a volume of the music, and / or prompt the driver to change the music to a different type of music.

[0059] Additionally, the estimated arousal score of the driver may be used to determine an amount of the intervention. That is, a difference between the estimated arousal score and a reference arousal score may be calculated, and the intervention may be applied based on the difference. In some examples, the intervention may be applied proportionally to the difference.

[0060] For example, the arousal estimation model may output a first arousal score of 4.0 for the driver at a first time, based on the sensor data at the first time. As a result of the driver having an estimated arousal score of 4.0, it may be inferred that the driver is experiencing fatigue. The estimated arousal score of 4.0 may be subtracted from a reference arousal score of 5.0, associated with a middle of the arousal scale (e.g., within the optimal state of arousal). Based on the difference of 1.0, a flow of cool air into the cabin may be increased by a first amount.

[0061] At a second, later time, the arousal estimation model may output a second arousal score of 2.0 for the driver, based on the sensor data at the second time. As a result of the driver having an estimated arousal score of 2.0, it may be inferred that the driver is beginning to experience drowsiness. The estimated arousal score of 2.0 may be subtracted from the reference arousal score of 5.0. Based on the difference of 3.0, the flow of cool air into the cabin may be increased by a second amount, where the second amount may be three times the first amount. In this way, an adjustment may be made to one or more vehicle or cabin settings based on the estimated arousal score.

[0062] In still other examples, the interventions described herein may include adjusting one or more operational controls of the vehicle. For example, if drowsiness is detected, in some examples, a sensitivity of an accelerator pedal or brake pedal of the vehicle may be altered, a gear of the vehicle may be adjusted, an optional system may be engaged, and so on. As other examples, an automatic lane keeping assistance functionality may be activated to ensure theAttorney Docket No. P240449WOvehicle stays in a current lane, and / or an intelligent cruise control may be activated to increase a distance between the vehicle and a leading vehicle to allow for more time to react. If the vehicle is an autonomous vehicle, a switch to the self-driving mode may be appropriate, or in cases when drowsiness is severe, the vehicle may be brought to a stop in a safe place.

[0063] If at 216 it is determined that the arousal score is not less than the first threshold arousal, method 200 proceeds to 220. At 220, method 200 includes determining whether the arousal is greater than a second threshold. The second threshold may be a value at which the driver may begin to experience cognitive load, which may reduce a safety of the driver and the vehicle. For example, the first threshold may be 6.0, where if the arousal score of the driver increases above 6.0, then the driver begins to experience the cognitive load.

[0064] If at 220 it is determined that the arousal score of the driver is greater than the second threshold, method 200 proceeds to 222. At 222, method 200 includes performing an intervention to decrease the arousal level (and score) of the driver. The intervention performed to decrease the arousal level may be similar to, or an inverse of the intervention to increase the arousal level. For example, to decrease the arousal level, the temperature of the cabin may be increased; the flow of air may be decreased; the lighting intensity may be reduced and / or frequency may be adjusted; etc. The amount of the intervention may be based on or proportional to the difference between the arousal score and the reference arousal score. Other types of interventions may include, for example, reducing a number of visual elements on a display screen of the cabin, suggesting a less complex or busy route of the vehicle, reducing a volume of audio output in the cabin or selecting a different musical choice, suggesting that the driver perform breathing exercises, etc.

[0065] By performing the intervention when the arousal score of the driver is greater than the second threshold, the intervention may be initiated when the driver is experiencing cognitive load, and prior to detection of stress. This is in contrast to other state detectors that may detect stress, but not earlier signs of stress.

[0066] If at 220 it is determined that the arousal score is not greater than the first threshold arousal, then it may be inferred that the arousal level of the driver is within the optimal arousal range for performing driving tasks, whereby method 200 proceeds to 224. At 224, method 200 includes continuing to monitor the driver arousal score as described above, and method 200 ends. It should be appreciated that in various examples, method 200 may be performed repeatedly and / or in a continuous or cyclical manner, to continuously monitor the arousal level of the driver.

[0067] Referring now to FIG. 3, an arousal estimation model training system 302 is shown that may be used to train an arousal estimation model 309. which may be the same as or similar to arousal estimation model 110 of FIG. 1. In particular, arousal estimation model 309 may be trained using arousal estimation model training system 302, and then a trained arousal estimation model 309 may be installed at a vehicle as arousal estimation model 110. In some embodiments, arousal estimation model training system 302 may be installed on a cloud-based server. In other embodiments, arousal estimation model training system 302 may be installed on a server of a driving simulation system (e.g., simulation system 336).

[0068] Arousal estimation model training system 302 includes a processor 304 configured to execute machine readable instructions stored in non-transitory memory 306. Processor 304 may be single core or multi-core, and the programs executed thereon may be configured for parallel or distributed processing. In some embodiments, the processor 304 may optionally include individual components that are distributed throughout two or more devices, which may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 304 may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration.

[0069] Non-transitory memory 306 may store a neural network module 308, a network training module 310, an inference module 312, and simulation data 314. Neural network module 308 may include one or more machine learning (ML) models and instructions for implementing the ML models to generate an arousal score of a driver of a vehicle, as described in greater detail below. Neural network module 308 may include one or more trained and / or untrained neural networks and may further include various data, or metadata pertaining to the one or more neural networks stored therein. In particular, neural network module may store arousal estimation model 309, described in greater detail below.

[0070] Training module 310 may comprise instructions for training one or more of the neural networks implementing the arousal estimation model 309 and / or other ML models stored in neural network module 308. In particular, training module 310 may include instructions that, when executed by the processor 304. cause arousal estimation model training system 302 to conduct one or more of the steps of method 500 for training the arousal estimation model 309 in a training stage, discussed in more detail below in reference to FIGS.4, 5 A, and 5B. In some embodiments, training module 310 includes instructions for implementing one or more gradient descent algorithms, applying one or more loss functions, and / or training routines, for use in adjusting parameters of the one or more neural networks ofAttorney Docket No. P240449WOneural network module 308. Inference module 312 may comprise instructions for generating an arousal score using a trained arousal estimation model 309.

[0071] In some embodiments, the non-transitory memory 306 may include components disposed at two or more devices, which may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the non-transitory memory 306 may include remotely-accessible networked storage devices configured in a cloud computing configuration.

[0072] Arousal estimation model training system 302 may be operably / communicatively coupled to a user input device 332 and a display device 334. User input device 332 may comprise one or more of a touchscreen, a keyboard, a mouse, a trackpad, or other device configured to enable a user to interact with and manipulate data within arousal estimation model training system 302. Display device 334 may include one or more display devices utilizing virtually any type of technology. In some embodiments, display device 334 may comprise a computer monitor, and may display medical images. Display device 334 may be combined with processor 304. non-transitory memory 306. and / or user input device 332 in a shared enclosure, or may be peripheral display devices and may comprise a monitor, touchscreen, projector, or other display device known in the art, which may enable a user to view simulation data produced by driving simulation system 336, and / or interact with various data stored in non-transitory memory 306 (including simulation data 314). In some examples, display device 334 and user input device 332 may be at least part of an operating console of driving simulation system 336.

[0073] Non-transitory memory 306 further stores simulation data 314. Simulation data 314 may include driving simulation data received from simulation system 336. Arousal estimation model training system 302 may be operably / communicatively coupled to simulation system 336. Arousal estimation model training system 302 may receive simulation data from simulation system 336, process the received simulation data via processor 304 based on instructions stored in one or more modules of non-transitory memory 306, and / or store the received simulation data in simulation data 314.

[0074] It should be understood that arousal estimation model training system 302 shown in FIG. 1 is for illustration, not for limitation. Another appropriate training system may include more, fewer, or different components.

[0075] Referring to FIG. 4, a schematic diagram of an arousal estimation procedure 400 for training arousal estimation model 309 of arousal estimation model training system 302 is shown. Arousal estimation model 309 may be trained to estimate an arousal score of a driverof a vehicle in accordance with one or more operations described in reference to method 200 of FIG. 2.

[0076] In some embodiments, arousal estimation model 309 may be a deep neural network with a plurality of hidden layers. In one embodiment, arousal estimation model 309 is a convolutional neural network (CNN). Arousal estimation model 309 may be stored within neural netw ork module 308 of arousal estimation model training system 302 of FIG. 3.

[0077] Arousal estimation procedure 400 includes collecting driving data from a plurality of drivers 402 within driving simulation system 336. During the collection of the driving data, each driver 402 may perform typical driving tasks in an immersive driving simulator. The driving simulator may include a screen as a simulated windshield, and controls similar to a typical vehicle. Each driver 402 may interact with the driving simulator in accordance with one or more driving programs, which may present various driving scenarios to the driver 402. Each driver 402 may operate the driving simulator via the same one or more driving programs.

[0078] In an embodiment, the driving scenarios may include a drowsiness scenario 406, a fatigue scenario 408, a cognitive load scenario 410, and a stress scenario 412. In each of the driving scenarios the drivers 402 may be subjected to different conditions or asked to perform different tasks. For example, drivers may be asked to perform tasks within drow siness scenario 406 at a time when they are experiencing drowsiness, such as in a moderately sleep-deprived state; drivers may be asked to perform demanding cognitive tasks during driving when subjected to the cognitive load scenario 410; and so on.

[0079] For each of the driving scenarios, sensor data may be collected from the drivers 402 by the driving simulator. The sensor data may be divided into two categories: production sensor data 414 used as input data for training arousal estimation model 309, and research sensor data 416 that may be used to generate ground truth data for training arousal estimation model 309. The production sensor data may be data collected from sensors installed in driving simulation system 336, which may be identical to or similar to sensors 120 of FIG. 1. The research sensor data may be acquired from sensors not typically used in a vehicle setting, such as electroencephalogram (EEG) and / or electrocardiogram (ECG) data.

[0080] The same types of production sensor data may be collected during each scenario. However, different types of research sensor data 416 may be collected during each scenario, where the different types of research sensor data 416 may be best suited to measuring, predicting, or detecting an arousal level of the driver in a respective scenario. Specifically, drowsiness sensor data 420 may be collected during the drowsiness scenario 406, fatigue sensor data 422 may be collected during the fatigue scenario 408, cognitive load sensor data 424 maybe collected during the cognitive load scenario 410, and stress sensor data 426 may be collected during the stress scenario 412. For example, drowsiness sensor data 420 and fatigue sensor data 422 may include EEG data acquired from EEG sensors arranged on a body of the driver, and images of a posture of the driver acquired from a camera of the driving simulator. On the other hand, cognitive load sensor data 424 may include performance data on tasks assigned to the driver, e.g., response time and response accuracy in a driving reaction task, or eye gaze data acquired using eye tracking tools arranged in front of the driver's face. Stress sensor data 426 may include heart beat frequency or heart rate data acquired via ECG sensors arranged on a body of the driver. Drowsiness sensor data 420, fatigue sensor data 422, cognitive load sensor data 424, and stress sensor data 426 may be used to generate the ground truth data.

[0081] A set of training data 440 used to train arousal estimation model 309 may be generated from the sensor data acquired from the driving simulator from each of the drivers 402 during a training data generation stage 430. To generate the ground truth data, portions of the production sensor data 414 collected during each of the driving scenarios may be manually labeled based on concurrently collected research sensor data 416. That is. production sensor data 414 acquired during each of drowsiness scenario 406, fatigue scenario 408, cognitive load scenario 410, and stress scenario 412 may be paired with ground truth data generated from research sensor data 416 of each of the respective sensor data categories. For example, a first training pair may include production sensor data 414 collected during the drowsiness scenario 406 as input data, and a first manually assigned label based on drowsiness sensor data 420 as ground truth data; a second training pair may include production sensor data 414 collected during the fatigue scenario 408 as input data, and a second manually assigned label based on fatigue sensor data 422 as ground truth data; and so on. The generation of the training data is described in greater detail below in reference to FIGS. 5A and 5B.

[0082] After training, a validator 451 may validate the performance of the arousal estimation model 309 against a portion of the training data 440 (e.g., test pairs) not used for training. The validator 451 may take as input a partially trained arousal estimation model 309 and a dataset of test pairs, and may output an assessment of the performance of the partially trained arousal estimation model 309 on the dataset of test pairs.

[0083] Once the arousal estimation model 309 has been validated, a trained arousal estimation model 110 (e.g., the validated arousal estimation model 309) may be transmitted to and installed in controllers (e.g., controller 102 of FIG. 1) of a plurality of vehicles 450. At each vehicle of the plurality of vehicles, a respective arousal estimation model 110 may be used to estimate arousal scores 454 for a driver of the vehicle, as described above in method 200.Attorney Docket No. P240449WO

[0084] Turning now to FIG. 5A, a flowchart illustrating a first method 500 for training arousal estimation model 309 is shown. Method 500 may be executed by a processor of an arousal estimation model training system, such as arousal estimation model training system 302 of FIG. 3. In an embodiment, some operations of method 500 may be stored in non-transitory memory of the arousal estimation model training system (e.g., in a training module such as the training module 310 of FIG. 3) and executed by a processor of the arousal estimation model training system (e.g., processor 304).

[0085] Method 500 begins at 502, where method 500 includes receiving production sensor data (e.g., production sensor data 414 of FIG. 4) and research sensor data (e.g., research sensor data 416) collected from a plurality of drivers (e.g., users) of a driving simulation system (e.g., driving simulation system 336 of FIGS. 3 and 4) for training the arousal estimation model. The production sensor data and research sensor data may be generated as described above in reference to FIG. 4.

[0086] At 504. method 500 includes dividing the received production sensor data and research sensor data into a plurality of analysis windows, where each analysis window corresponds to a period of time during which parameters may be aggregated for inputting into the arousal estimation model, as described above in method 200. For example, the analysis window may have a duration of one second, where relevant parameters collected from the production sensor data during one second may be inputted into the arousal estimation model during one feedforward cycle, to generate an output relevant to the analysis window.

[0087] At 506, method 500 includes generating a training dataset of triplets (e.g., as opposed to training pairs). Each triplet may include a first set of production sensor data, a second set of production sensor data, and a manually generated annotation indicating which of the first and second sets of production sensor data is associated with a higher arousal score. Generating the training dataset of triples includes, at 506, creating pairs of portions of production sensor data, and at 508, generating annotations for each pair as ground truth, where the annotations are generated from the research sensor data collected at the same time as the production sensor data. The first set of production sensor data and the second set of production sensor data may be collected from a same driver of the plurality of drivers. The annotation may be a Boolean value, comprising a zero when the first set of production sensor data is associated with a higher arousal score than the second set of production sensor data, and a one when the second set of production sensor data is associated with a higher arousal score than the first set of production sensor data. The annotations may be manually generated based on the research sensor data collected concurrently w ith the first and second sets of production sensor data.Attorney Docket No. P240449WO

[0088] As an example of how a training triplet may be generated, a driver may perform a series of typical driving tasks in the driving simulation system, during various scenarios. The driver may perform the typical driving tasks during a drowsiness scenario, at a time when the driver is experiencing drowsiness. The driver may perform the typical driving tasks during a fatigue scenario, at a time when the driver is experiencing fatigue. The driver may perform the typical driving tasks during a cognitive load scenario, during which the driver is asked to perform cognitively demanding tasks such as the n-back task or a similar continuous performance task used to assess working memory. The driver may perform the typical driving tasks during a stress scenario, during which stressful conditions are imposed on the driver by the driving simulation system. During each of the scenarios, production sensor data and research sensor data are collected concurrently, and then divided into a plurality of analysis windows.

[0089] A first portion of the production sensor data corresponding to a first analysis window' during the drow siness scenario may be selected. A first portion of the research sensor data collected over the same duration as the first portion of production sensor data may then be analyzed by a human expert. For example, the first portion of research sensor data may include EEG data of the driver, and camera images of the driver’s posture. The human expert may review the EEG data and the camera images, and may estimate a first arousal score for the driver for the first analysis window based on the EEG data and the camera images. The first portion of the production sensor data may be labeled with the estimated first arousal score.

[0090] A second portion of the production sensor data corresponding to a second analysis window during the cognitive load scenario may then be selected. A second portion of the research sensor data collected over the same duration as the second portion of production sensor data may then be analyzed by the human expert. The second portion of research sensor data may include assessments of the driver’s performance on one or more driving tasks collected by the driving simulation system. For example, the driving simulation system may record measured reaction times of the driver in response to situations presented in the driving simulator. The human expert may review the reaction time data, and may estimate a second arousal score for the driver for the second analysis window based on the reaction time data. The second portion of the production sensor data may be labeled with the estimated second arousal score.

[0091] A third portion of the production sensor data corresponding to a third analysis window during the stress scenario may then be selected. A third portion of the research sensor data collected over the same duration as the third portion of production sensor data may thenAttorney Docket No. P240449WObe analyzed by the human expert. The third portion of research sensor data may include ECG data and camera images of the driver. The ECG data may be processed to generate a heart rate of the driver. The camera images may also be processed to determine the heart rate. The human expert may review the heart rate data, and may estimate a third arousal score for the driver for the third analysis window based on the heart rate data. The third portion of the production sensor data may be labeled with the estimated third arousal score.

[0092] In this way, arousal scores may be manually generated for production sensor data in a plurality of analysis windows extending over a plurality of different scenarios. Additionally, the first, second, and third arousal scores may be assigned based on a same scale. For example, the first arousal score for the drowsiness scenario may be a low score; the second arousal score for the cognitive load scenario may be a higher score than the first arousal score; and the third arousal score for the stress scenario may be a higher score than the second arousal score.

[0093] The portions of production data may then be paired, in different ordered pair configurations. For example, a first pair may include the first portion of production data and the second portion of production data; a second pair may include the third portion of production data and the first portion of production data; and a third pair may include the second portion of production data and the third portion of production data; and so on. Then, for each ordered pair, an annotation is assigned to the pair indicating which of the portions has a higher arousal score. For example, for the first pair, a first annotation of zero may be assigned, indicating that the first arousal score of the first portion of production data is lower than the second arousal score of the second portion of production data. For the second pair, a second annotation of one may be assigned, indicating that the third arousal score of the third portion of production data is higher than the first arousal score of the first portion of production data. For the third pair, a third annotation of zero may be assigned, indicating that the first arousal score of the first portion of production data is lower than the third arousal score of the third portion of production data, and so on for each pair configuration. The annotations may be assigned programmatically based on the manually labeled arousal scores.

[0094] Training triplets are then formed from the annotated, paired portions of production sensor data. A first training triplet may be formed, comprising the first portion of production data, the second portion of production data, and the first annotation; a second training triplet may be formed, comprising the third portion of production data, the first portion of production data, and the second annotation; a third training triplet may be formed, comprising the secondAttorney Docket No. P240449WOportion of production data, the third portion of production data, and the third annotation; and so on.

[0095] At 512, method 500 includes training the arousal estimation model on the training dataset. The arousal estimation model may include one or more convolutional layers, which in turn comprise one or more convolutional filters (e.g., a convoluted neural network architecture). The convolutional filters of the arousal estimation model may comprise a plurality of weights, wherein the values of the weights are learned during a training procedure.

[0096] Training the arousal estimation model on the training triplets may include iteratively inputting, at each cycle of training, both of the paired portions of production sensor data of each training triplet into an input layer of the arousal estimation model. The arousal estimation model input may be propagated from the input layer, through a plurality of hidden layers, to an output layer of the arousal estimation model. The output layer may include two output nodes, one for each set of inputs (e.g., one for each portion of production sensor data included in a respective pair). The output signals at each output node may then be compared, to obtain a Boolean value indicating which of the output signals is greater. The Boolean value is then compared with the annotation included in the triplet.

[0097] If the Boolean value is different from the annotation, the error is backpropagated through the arousal estimation model. The arousal estimation model may be configured to iteratively adjust one or more of a plurality of weights of the arousal estimation model during backpropagation, in order to minimize a loss function, based on the difference between the Boolean value and the annotation for each of the training triplets. For example, the loss function minimize a mean squared error between the Boolean value and the annotation. The loss may be backpropagated through each layer of the arousal estimation model in a reverse order of the layers. In some embodiments, back propagation of the loss may occur according to a gradient descent algorithm, wherein a gradient of the loss function (a first derivative, or approximation of the first derivative) is determined for each weight and bias of the deep neural network. Each weight (and bias) of the arousal estimation model is then updated by adding the negative of the product of the gradient determined (or approximated) for the weight (or bias) with a predetermined step size. Updating of the weights and biases may be repeated until the weights and biases of the arousal estimation model converge, or the rate of change of the weights and / or biases of the deep neural network for each iteration of weight adjustment are under a threshold.

[0098] In order to avoid overfitting, training of the arousal estimation model may be periodically interrupted to validate a performance of the arousal estimation model on the test training triplets. In an embodiment, training of the arousal estimation model may end when aAttorney Docket No. P240449WOperformance of the arousal estimation model on the test training triplets converges (e.g., when an error rate on the test set converges on or to within a threshold of a minimum value).

[0099] In this way, the arousal estimation model may be trained to estimate an arousal score for a driver based on the production sensor data. That is, as a result of using binary ground truth data, a quantitative determination of the arousal of the driver may be obtained by the trained arousal estimation model, where after training, each output of the trained arousal estimation model may correspond to an arousal score for a respective portion of the production sensor data. Thus, after training, the arousal estimation model can take a single set of production sensor data corresponding to an analysis window, and output an arousal score for the single set of production sensor data.

[0100] In other embodiments, the manually assigned arousal scores associated with each portion of the production sensor data may be used as ground truth data, and the arousal estimation model may be trained using training pairs of production sensor data and the manually assigned arousal scores.

[0101] In some examples, a different, automated process may be used for generating labeled ground truth data. Referring again to FIG. 8. the foundation for the arousal scale comes from the Yerks-Dodson Law curve. which shows a qualitative performance of a task varying as a function of arousal level of a person performing the task. This curve attributes different mental states to ranges of arousal scores on the arousal scale. Stress is defined by a range of arousal scores at which performance of the task is the worst at one extreme (e.g., extreme arousal), and drowsiness is defined by a range of arousal scores at which performance is the worst at the other extreme (e.g., low arousal). Cognitive load is defined by a range of arousal scores at which performance of the task declines linearly towards stress, and fatigue is defined by a range of arousal scores at which performance of the task declines linearly towards drowsiness. Thus, cognitive load may be a precursor to stress, and fatigue may be a precursor to drowsiness.

[0102] Because these mental states are defined in accordance with the performance of a specific task, a driving task can be defined, and the performance of a user of a driving simulator at the driving task can be quantitatively assessed. The quantitative assessments can then be used to generate arousal scores in a way that does not rely on manual labeling. This approach is described in reference to FIG. 5B.

[0103] Turning now to FIG. 5B, a flowchart illustrating a second method 550 for training arousal estimation model 309 is shown. Method 550 may be executed by a processor of an arousal estimation model training system, such as arousal estimation model training systemAttorney Docket No. P240449WO302 of FIG. 3. In an embodiment, some operations of method 500 may be stored in non-transitory memory of the arousal estimation model training system (e.g., in a training module such as the training module 310 of FIG. 3) and executed by a processor of the arousal estimation model training system (e.g., processor 304).

[0104] Method 550 begins at 552, where method 550 includes receiving portions of production sensor data (e.g., production sensor data 414 of FIG. 4) from a plurality of drivers (e.g.. users) of a driving simulation system. The production sensor data may be generated as described in FIG. 4, but with respect to a specific task performed during typical driving. For example, the specific task may be braking to a stop in response to a sudden simulated obstacle. When the stimulated obstacle is presented, a reaction time of the driver may be measured. Production sensor data may be collected at the time of the simulated obstacle. The production sensor data may be collected from each driver of the plurality of drivers under the different driving scenarios described in FIG. 4. Each portion of the production sensor data collected may be labeled with the reaction time.

[0105] At 554. method 550 includes mapping each portions of the received production sensor data to an arousal score based on the task assessment (e.g., reaction time), using the arousal curve (e g., plot 802 of FIG. 8). That is, the reaction time associated with each portion of the production sensor data may be associated with a point on the arousal curve, using the y axis as a reference. An x-coordinate of the point on the x axis indicates an arousal score associated with the portion of the received production sensor data. The portion of the received production sensor data may then be labeled with the arousal score. Arousal scores associated with low arousal may be disambiguated from arousal scores associated with high arousal based on the driving scenarios during which the portions of received production sensor data are collected. In some examples, the arousal scores may be confirmed or verified using the research sensor data described in method 500.

[0106] At 556, method 550 includes generating a training dataset of training pairs, where each training pair includes a portion of the received production sensor data as input data, and the mapped arousal score associated with the received production sensor data as ground truth data.

[0107] At 558, method 550 includes training the arousal estimation model on the training pairs. Training the arousal estimation model on the training pairs may include iteratively inputting, at each cycle of training, the portion of production sensor data of each training pair into an input layer of the arousal estimation model. The arousal estimation model input may be propagated from the input layer, through a plurality of hidden layers, to an output layer of theAttorney Docket No. P240449WOarousal estimation model. The output layer may include one output node. An output signal generated at the output node may be compared with the mapped arousal score, and a difference between the output signal and the mapped arousal score (e.g., the error) may be backpropagated through the arousal estimation model. The arousal estimation model may be configured to iteratively adjust one or more of a plurality of weights of the arousal estimation model during backpropagation, in order to minimize a loss function, as with method 500.

[0108] In other words, driver participants can be subjected to drowsiness, fatigue, cognitive load, and stress under experimental conditions within a driving simulator, with physiological, camera, and other sensor data recorded, and be prompted to perform the selected task. Their performance on the task (e.g., reaction time) can be measured and recorded. Based on the assumption that their performance on the task follows the Yerks-Dodson Law. the reaction times will follow a rough Gaussian curve. Thus, the reaction time data can be fit to the curve, where an x coordinate of each sample point corresponds to an estimated arousal scores on the arousal scale. The boundaries of each arousal state (e.g., drowsiness, fatigue, cognitive load, and stress) can be determined to establish threshold arousal values for each category. The arousal scores can be used to label the production sensor data associated with each sample point. The threshold arousal values can additionally be used to classify the sample data. The result is a training set of biosensor input data and ground truth classification into an appropriate mental state, as well as the ground truth arousal values corresponding to the production sensor (input) data. The model can then be trained using supervised learning to estimate arousal scores for drivers of vehicles, under the assumption that the performance of drivers on the selected task is consistent with the performance of other driving tasks.

[0109] FIG. 6 shows an interior of a cabin 600 of a vehicle 602, in which a driver and / or one or more passengers may be seated. Vehicle 602 of FIG. 6 may be a motor vehicle including drive wheels (not shown) and an internal combustion engine 604. Internal combustion engine 604 may include one or more combustion chambers which may receive intake air via an intake passage and exhaust combustion gases via an exhaust passage. Vehicle 602 may be a road automobile, among other types of vehicles. In some examples, vehicle 602 may include a hybrid propulsion system including an energy conversion device operable to absorb energy from vehicle motion and / or the engine and convert the absorbed energy to an energy form suitable for storage by an energy storage device. Vehicle 602 may include a fully electric vehicle, incorporating fuel cells, solar energy capturing elements, and / or other energy storage systems for powering the vehicle.Attorney Docket No. P240449WO

[0110] Vehicle 602 may include a plurality of vehicle systems, including a braking system for providing braking, an engine system for providing motive power to wheels of the vehicle, a steering system for adjusting a direction of the vehicle, a transmission system for controlling a gear selection for the engine, an exhaust system for processing exhaust gases, and the like. Further, the vehicle 602 includes an in-vehicle computing system 609, which may include elements of vehicle control system 100 of FIG. 1. The in-vehicle computing system 609 is described in greater detail below in reference to FIG. 7. The in-vehicle computing system 609 may include an arousal estimation system such as arousal estimation system 108, which may monitor an arousal level of a driver of vehicle 602, as described above.[OHl] As shown, an instrument panel 606 may include various displays and controls accessible to a human user (e.g., a driver or a passenger) of vehicle 602. For example, instrument panel 606 may include a touch screen 608 of an in-vehicle computing system or infotainment system 609 (e.g., IVI system 134), an audio system control panel, and an instrument cluster 610. Touch screen 608 may receive user input to the in-vehicle computing system or infotainment system 609 for controlling audio output, visual display output, user preferences, control parameter selection, and so on.

[0112] While the example system shown in FIG. 6 includes audio system controls that may be performed via a user interface of in-vehicle computing system or infotainment system 609, such as touch screen 608 without a separate audio system control panel, in other embodiments, the vehicle may include an audio system control panel, which may include controls for a conventional vehicle audio system such as a radio, compact disc player, MP3 player, and so on. The audio system controls may include features for controlling one or more aspects of audio output via one or more speakers 612 (e.g.. speakers 132) of a vehicle speaker system. For example, the in-vehicle computing system or the audio system controls may control a volume of audio output, a distribution of sound among the individual speakers of the vehicle speaker system, an equalization of audio signals, and / or any other aspect of the audio output. In further examples, in-vehicle computing system or infotainment system 609 may adjust a radio station selection, a playlist selection, a source of audio input (e.g., from radio or MP3), and so on, based on user input received directly via touch screen 608, or based on data regarding the user (such as an arousal level of the driver) received via one or more external devices 650 and / or a mobile device 628. The audio system of the vehicle may include an amplifier (not shown) coupled to plurality of loudspeakers (not shown). In some embodiments, one or more hardware elements of in-vehicle computing system or infotainment system 609, such as touch screen 608, a display screen 611, various control dials, knobs and buttons, memory, processor(s), andany interface elements (e.g., connectors or ports) may form an integrated head unit that is installed in instrument panel 606 of the vehicle. The head unit may be fixedly or removably attached in instrument panel 606. In additional or alternative embodiments, one or more hardware elements of the in-vehicle computing system or infotainment system 609 may be modular and may be installed in multiple locations of the vehicle.

[0113] The cabin 600 may include one or more sensors for monitoring the vehicle, the user, and / or the environment. For example, the cabin 600 may include one or more seat-mounted pressure sensors configured to measure the pressure applied to the seat to determine the presence of a user, door sensors configured to monitor door activity, humidity sensors to measure the humidity content of the cabin, microphones to receive user input in the form of voice commands, to enable a user to conduct telephone calls, and / or to measure ambient noise in the cabin 600, and so on. It is to be understood that the above-described sensors and / or one or more additional or alternative sensors may be positioned in any suitable location of the vehicle. For example, sensors may be positioned in an engine compartment, on an external surface of the vehicle, and / or in other suitable locations for providing information regarding the operation of the vehicle, ambient conditions of the vehicle, a user of the vehicle, and so on. Information regarding ambient conditions of the vehicle, vehicle status, or vehicle driver may also be received from sensors external to / separate from the vehicle (that is, not part of the vehicle system), such as sensors coupled to external devices 650 and / or mobile device 628. Sensor data of various sensors of the vehicle may be transmitted to and / or accessed by the in-vehicle computing system 609 via a bus of the vehicle, such as a CAN bus.

[0114] Cabin 600 may also include one or more user objects, such as mobile device 628, that are stored in the vehicle before, during, and / or after travelling. The mobile device 628 may include a smart phone, a tablet, a laptop computer, a portable media player, and / or any suitable mobile computing device. The mobile device 628 may be connected to the in-vehicle computing system via a communication link 630. The communication link 630 may be wired (e.g., via Universal Serial Bus (USB), Mobile High-Definition Link (MHL), High-Definition Multimedia Interface (HDMI). Ethernet, and so on) or wireless (e.g., via Bluetooth®. Wi-Fi®, Wi-Fi Direct®, Near-Field Communication (NFC), cellular connectivity, and so on) and configured to provide two-way communication between the mobile device and the in-vehicle computing system. (Wi-Fi® and Wi-Fi Direct® are registered trademarks of Wi-Fi Alliance, Austin, Texas.) The mobile device 628 may include one or more wireless communication interfaces for connecting to one or more communication links (e.g., one or more of the example communication links described above). The wireless communication interface may include oneAttorney Docket No. P240449WOor more physical devices, such as antenna(s) or port(s) coupled to data lines for carrying transmitted or received data, as well as one or more modules / drivers for operating the physical devices in accordance with other devices in the mobile device. For example, the communication link 630 may provide sensor and / or control signals from various vehicle systems (such as vehicle audio system, climate control system, and so on) and the touch screen 608 to the mobile device 628 and may provide control and / or display signals from the mobile device 628 to the in-vehicle systems and the touch screen 608. The communication link 630 may also provide power to the mobile device 628 from an in-vehicle power source in order to charge an internal battery of the mobile device.

[0115] In-vehicle computing system or infotainment system 609 may also be communicatively coupled to additional devices operated and / or accessed by the user but located external to vehicle 602, such as one or more external devices 650. In the depicted embodiment, external devices are located outside of vehicle 602 though it will be appreciated that in alternate embodiments, external devices may be located inside cabin 600. The external devices may include a server computing system, personal computing system, portable electronic device, electronic wrist band, electronic head band, portable music player, electronic activity tracking device, pedometer, smart-watch, GPS system, wearable PPG, EEG, and / or ECG sensor systems, and so on. External devices 650 may be connected to the in-vehicle computing system via a communication link 636 which may be wired or wireless, as discussed with reference to communication link 630. and configured to provide two-way communication between the external devices and the in-vehicle computing system. For example, external devices 650 may include one or more sensors and communication link 636 may transmit sensor output from external devices 650 to in-vehicle computing system or infotainment system 609 and touch screen 608. External devices 650 may also store and / or receive information regarding contextual data, user behavior / preferences, operating rules, and so on and may transmit such information from the external devices 650 to in-vehicle computing system or infotainment system 609 and touch screen 608.

[0116] In-vehicle computing system or infotainment system 609 may analyze the input received from external devices 650. mobile device 628, and / or other input sources and select settings for various in-vehicle systems (such as climate control system or audio system), provide output via touch screen 608 and / or speakers 612, communicate with mobile device 628 and / or external devices 650. and / or perform other actions based on the assessment. In some embodiments, all or a portion of the assessment may be performed by the mobile device 628 and / or the external devices 650. In particular, the communication link 630 may be used toAttorney Docket No. P240449WOreceive or retrieve, from the mobile device 628, information of the driver stored on the mobile device 628.

[0117] In some embodiments, one or more of the external devices 650 may be communicatively coupled to in-vehicle computing system or infotainment system 609 indirectly, via mobile device 628 and / or another of the external devices 650. For example, communication link 636 may communicatively couple external devices 650 to mobile device 628 such that output from external devices 650 is relayed to mobile device 628. Data received from external devices 650 may then be aggregated at mobile device 628 with data collected by mobile device 628, the aggregated data then transmitted to in-vehicle computing system or infotainment system 609 and touch screen 608 via communication link 630. Similar data aggregation may occur at a server system and then transmitted to in-vehicle computing system or infotainment system 609 and touch screen 608 via communication link 636 and / or communication link 630.

[0118] FIG. 7 shows a block diagram of an in-vehicle computing system or infotainment system 609 configured and / or integrated inside vehicle 602. In-vehicle computing system or infotainment system 609 may perform one or more of the methods described herein in some embodiments. In some examples, the in-vehicle computing system or infotainment system 609 may be a vehicle infotainment system configured to provide information-based media content (audio and / or visual media content, including entertainment content, navigational services, automated conversational services, and so on) to a vehicle user to enhance the operator's in-vehicle experience. The in-vehicle computing system or infotainment system 609 may include, or be coupled to, various vehicle systems, sub-systems, hardware components, as well as software applications and systems that are integrated in, or integratable into, vehicle 602 in order to enhance an in-vehicle experience for a driver and / or a passenger.

[0119] In-vehicle computing system or infotainment system 609 may include one or more processors including an operating system processor 714 (e.g., processor 104) and an interface processor 720. Operating system processor 714 may execute an operating system on the in-vehicle computing system, and control input / output. display, playback, and other operations of the in-vehicle computing system. Interface processor 720 may interface with a vehicle control system 730 via an inter-vehicle system communication module 722.

[0120] Inter-vehicle system communication module 722 may output data to one or more other vehicle systems 731 and / or one or more other vehicle control elements 761, while also receiving data input from other vehicle systems 731 and other vehicle control elements 761, e.g., by way of vehicle control system 730. When outputting data, inter-vehicle systemAttorney Docket No. P240449WOcommunication module 722 may provide a signal via a bus corresponding to any status of the vehicle, the vehicle surroundings, or the output of any other information source connected to the vehicle. Vehicle data outputs may include, for example, analog signals (such as current velocity), digital signals provided by individual information sources (such as clocks, thermometers, location sensors such as GPS sensors, and so on), digital signals propagated through vehicle data networks (such as an engine CAN bus through which engine related information may be communicated, a climate control CAN bus through which climate control related information may be communicated, and a multimedia data network through which multimedia data is communicated between multimedia components in the vehicle). For example, vehicle data outputs may be output to vehicle control system 730, and vehicle control system 730 may adjust vehicle controls 761 based on the vehicle data outputs. As another example, the in-vehicle computing system or infotainment system 609 may retrieve from the engine CAN bus the current speed of the vehicle estimated by the wheel sensors, a power state of the vehicle via a battery and / or power distribution system of the vehicle, an ignition state of the vehicle, and so on. In addition, other interfacing means such as Ethernet may be used as well without departing from the scope of this disclosure.

[0121] A storage device 708 may be included in in-vehicle computing system or infotainment system 609 to store data such as instructions executable by operating system processor 714 and / or interface processor 720 in non-volatile form. The storage device 708 may store application data, including prerecorded sounds, to enable the in-vehicle computing system or infotainment system 609 to run an application for connecting to a cloud-based server and / or collecting information for transmission to the cloud-based server. The application may retrieve information gathered by vehicle systems / sensors, input devices (e.g., a user interface 718), data stored in one or more storage devices, such as a volatile memory 719A or a non-volatile memory 719B, devices in communication with the in-vehicle computing system (e.g., a mobile device connected via a Bluetooth® link), and so on. In-vehicle computing system or infotainment system 609 may further include a volatile memory 719A. Volatile memory 719A may be RAM. Non-transitory storage devices, such as non-volatile storage device 708 (e.g., memory 106) and / or non-volatile memory 719B, may store instructions and / or code that, when executed by a processor (e g., operating system processor 714 and / or interface processor 720), controls the in-vehicle computing system or infotainment system 609 to perform one or more of the actions described in the disclosure.

[0122] A microphone 702 (e.g., microphone 158) may be included in the in-vehicle computing system or infotainment system 609 to receive voice commands from a user, toAttorney Docket No. P240449WOmeasure ambient noise in the vehicle, to determine whether audio from speakers of the vehicle is tuned in accordance with an acoustic environment of the vehicle, and so on. In particular, microphone 702 may be used by an arousal estimation system 723 (e.g., arousal estimation system 108), to determine a level of arousal of a driver. A speech processing unit 704 may process voice commands, such as the voice commands received from the microphone 702. The speech processing unit 704 may include a text-to-voice converter, which may convert text generated by the arousal estimation system 723 into speech that is played at an audio system 732 of the vehicle 602. In some embodiments, in-vehicle computing system or infotainment system 609 may also be able to receive voice commands and sample ambient vehicle noise using a microphone included in the audio system 732.

[0123] One or more additional sensors may be included in a sensor subsystem 710 of the in-vehicle computing system or infotainment system 609. For example, the sensor subsystem 710 may include a plurality of cameras 725, such as a rear view camera for assisting a user in parking the vehicle and / or other external cameras, radars, lidars, ultrasonic sensors, and the like. The sensor subsystem 710 may include an in-cabin camera (e.g.. a dashboard cam) for identifying a user (e.g.. using facial recognition and / or user gestures). Additionally, an in-cabin camera 725 may be used to detect or estimate a level of arousal of the one or more users, which may manifest as drowsiness, fatigue, cognitive load, stress, etc. Sensor subsystem 710 of in-vehicle computing system or infotainment system 609 may communicate with and receive inputs from various vehicle sensors and may further receive user inputs. For example, the inputs received by sensor subsystem 710 may include transmission gear position, transmission clutch position, gas pedal input, brake input, transmission selector position, vehicle speed, engine speed, mass airflow through the engine, ambient temperature, intake air temperature, and so on, as well as inputs from climate control system sensors (such as heat transfer fluid temperature, antifreeze temperature, fan speed, passenger compartment temperature, desired passenger compartment temperature, ambient humidity, and so on), an audio sensor detecting voice commands issued by a user, a fob sensor receiving commands from and optionally tracking the geographic location / proximity of a fob of the vehicle, and so on.

[0124] One or more additional sensors may be included in and / or communicatively coupled to a sensor subsystem 710 of the in-vehicle computing system 609. For example, the sensor subsystem 710 may include and / or be communicatively coupled to a camera, such as a rear view camera for assisting a user in parking the vehicle, a cabin camera for identifying a user, and / or a front view camera to assess quality of the route segment ahead. The abovedescribed cameras may also be used to provide images to a computer vision-based facialrecognition and / or facial analysis module. For example, the facial analysis module may be used to determine an emotional or psychological (e.g., arousal) state of users of the vehicle, based on facial expressions, eye gaze data, etc. Sensor subsystem 710 of in-vehicle computing system 609 may communicate with and receive inputs from various vehicle sensors and may further receive user inputs.

[0125] While certain vehicle system sensors may communicate with sensor subsystem 710 alone, other sensors may communicate with both sensor subsystem 710 and vehicle control system 730, or may communicate with sensor subsystem 710 indirectly via vehicle control system 730. Sensor subsystem 710 may serve as an interface (e.g., a hardware interface) and / or processing unit for receiving and / or processing received signals from one or more of the sensors described in the disclosure.

[0126] A navigation subsystem 711 (e.g., navigational system 160) of in-vehicle computing system or infotainment system 609 may generate and / or receive navigation information such as location information (e.g., via a GPS sensor and / or other sensors from sensor subsystem 710), route information, destination information, traffic information, point-of-interest (POI) identification, and / or provide other navigational services for the user. Navigation sub-system 711 may include inputs / outputs including analog to digital converters, digital inputs, digital outputs, network outputs, radio frequency transmitting devices, and so on. In some examples, navigation sub-system 711 may interface with vehicle control system 730.

[0127] An external device interface 712 of in-vehicle computing system or infotainment system 609 may be coupleable to and / or communicate with one or more external devices 650 located external to vehicle 602. While the external devices are illustrated as being located external to vehicle 602, it is to be understood that they may be temporarily housed in vehicle 602, such as when the user is operating the external devices while operating vehicle 602. In other words, the external devices 650 are not integral to vehicle 602. The external devices 650 may include a mobile device 628 (e.g., connected via a Bluetooth®, NFC, Wi-Fi Direct®, or other wireless connection) or an alternate Bluetooth®-enabled device 752.

[0128] Mobile device 628 may be a mobile phone, smart phone, wearable devices / sensors that may communicate with the in-vehicle computing system via wired and / or wireless communication, or other portable electronic device(s). Other external devices include one or more external services 746. For example, the external devices may include extra-vehicular devices that are separate from and located externally to the vehicle. Still other external devices include one or more external storage devices 754, such as solid-state drives, pen drives, USBdrives, and so on. External devices 650 may communicate with in-vehicle computing system or infotainment system 609 either wirelessly or via connectors without departing from the scope of this disclosure. For example, external devices 650 may communicate with in-vehicle computing system or infotainment system 609 through the external device interface 712 over a network 760, a USB connection, a direct wired connection, a direct wireless connection, and / or other communication link.

[0129] One or more applications 744 may be operable on mobile device 628. In addition, specific user data requests may be received at mobile device 628 from in-vehicle computing system or infotainment system 609 via the external device interface 712. The specific data requests may include requests for information of the user that may be used by the arousal estimation system 723 to estimate an arousal of the user, in one example. An application 744 may send control instructions to components of mobile device 628 to enable the requested data to be collected on the mobile device. The application 744 may then relay the collected information back to in-vehicle computing system or infotainment system 609.

[0130] Likewise, one or more applications 748 may be operable on external services 746. As an example, external services applications 748 may be operated to aggregate and / or analyze data from multiple data sources. For example, external services applications 748 may aggregate data from one or more social media accounts of the user, data from the in-vehicle computing system (e.g., sensor data, log files, user input, and so on), data from an internet query (e.g., weather data, POI data), and so on. The collected data may be transmitted to another device and / or analyzed by the application to determine a context of the driver, vehicle, and environment and perform an action based on the context (e.g., requesting / sending data to other devices).

[0131] Vehicle control system 730 may include controls for controlling aspects of various vehicle systems 731 involved in different in-vehicle functions. These may include, for example, controlling aspects of vehicle audio system 732 for providing audio entertainment to the vehicle occupants, aspects of a climate control system 734 for meeting the cabin cooling or heating demands of the vehicle occupants, as well as aspects of a telecommunication system 736 for enabling vehicle occupants to establish telecommunication linkage with others.

[0132] Audio system 732 may include one or more acoustic reproduction devices including electromagnetic transducers such as one or more speakers 735. Vehicle audio system 732 may be passive or active such as by including a power amplifier. In some examples, in-vehicle computing system or infotainment system 609 may be a sole audio source for the acoustic reproduction device or there may be other audio sources that are connected to the audioAttorney Docket No. P240449WOreproduction system (e.g., external devices such as a mobile phone). The connection of any such external devices to the audio reproduction device may be analog, digital, or any combination of analog and digital technologies.

[0133] Climate control system 734 may be configured to provide a comfortable environment within the cabin or passenger compartment of vehicle 602. Climate control system 734 includes components enabling controlled ventilation such as air vents, a heater, an air conditioner, an integrated heater and air-conditioner (e.g., HVAC) system, and so on. Other components linked to the heating and air-conditioning setup may include a windshield defrosting and defogging system capable of clearing the windshield and a ventilation-air filter for cleaning outside air that enters the passenger compartment through a fresh-air inlet.

[0134] Vehicle control system 730 may also include controls for adjusting the settings of various vehicle control elements 761 (or vehicle controls, or vehicle system control elements) related to the engine and / or auxiliary elements within a cabin of the vehicle, such as one or more steering wheel controls 762 (e g., steering wheel-mounted audio system controls, cruise controls, windshield wiper controls, headlight controls, turn signal controls, and so on), instrument panel controls, microphone(s), accelerator / brake / clutch pedals, a gear shift, door / window controls positioned in a driver or passenger door, seat controls, cabin light controls, audio system controls, cabin temperature controls, HVAC controls, and so on. Vehicle control elements 761 may also include internal engine and vehicle operation controls (e.g., engine controller module, actuators, valves, and so on) that are configured to receive instructions via the CAN bus of the vehicle to change operation of one or more of the engine, exhaust system, transmission, and / or other vehicle system. The control signals may also control audio output at one or more speakers 735 of the vehicle’s audio system 732. For example, the control signals may adjust audio output characteristics such as volume, equalization, audio image (e.g., the configuration of the audio signals to produce audio output that appears to a user to originate from one or more defined locations), audio distribution among a plurality of speakers, and so on. Likewise, the control signals may control vents, air conditioner, and / or heater of climate control system 734. For example, the control signals may increase delivery of cooled air to a specific section of the cabin to increase an arousal level of a user.

[0135] Vehicle controls 761 may include a steering control system 762, a braking control system 763, and an acceleration control system 764. Vehicle controls 761 may include additional control systems. In some example, vehicle controls 761 may be operated autonomously. In other examples, vehicle controls 761 may be controlled by a user. Further, in some examples, a user may primarily control vehicle controls 761, while one or more ADASAttorney Docket No. P240449WO765 may intermittently adjust vehicle controls 761 in order to increase vehicle performance. For example, the one or more ADAS 765 may include a cruise control system, a lane departure warning system, a collision avoidance system, an adaptive braking system, and the like.

[0136] Steering control system 762 may be configured to control a direction of the vehicle. For example, during a non-autonomous mode of operation, steering control system 762 may be controlled by a steering wheel. For example, the user may turn the steering wheel in order to adjust a vehicle direction. During an autonomous mode of operation, steering control system 762 may be controlled by vehicle control system 730. In some examples, one or more ADAS 765 may adjust steering control system 762. For example, the vehicle control system 730 may determine that a change in vehicle direction is requested, and may change the vehicle direction via controlling the steering control system 762. For example, vehicle control system 730 may adjust axles of the vehicle in order to change the vehicle direction.

[0137] Braking control system 763 may be configured to control an amount of braking force applied to the vehicle. For example, during a non-autonomous mode of operation, braking control system 763 may be controlled by a brake pedal. For example, the user may depress the brake pedal in order to increase an amount of braking applied to the vehicle. Dunng an autonomous mode of operation, braking system 763 may be controlled autonomously. For example, the vehicle control system 730 may determine that additional braking is requested, and may apply additional braking. In some examples, the autonomous vehicle control system may depress the brake pedal in order to apply braking (e.g., to decrease vehicle speed and / or bring the vehicle to a stop). In some examples, the one or more ADAS 765 may adjust braking control system 763. The arousal estimation system 723 may also adjust braking control system 763, in some examples.

[0138] Acceleration control system 764 may be configured to control an amount of acceleration applied to the vehicle. For example, during a non-autonomous mode of operation, acceleration control system 764 may be controlled by an acceleration pedal. For example, the user may depress the acceleration pedal in order to increase an amount of torque applied to wheels of the vehicle, causing the vehicle to accelerate in speed. During an autonomous mode of operation, acceleration control system 764 may be controlled by vehicle control system 730. In some examples, the one or more ADAS 765 may adjust acceleration control system 764. For example, vehicle control system 730 may determine that additional vehicle speed is requested, and may increase vehicle speed via acceleration. In some examples, vehicle control system 730 may depress the acceleration pedal in order to accelerate the vehicle. As an example of an ADAS 765 adjusting acceleration control system 764, the ADAS 765 may be a cruiseAttorney Docket No. P240449WOcontrol system, and may include adjusting vehicle acceleration in order to maintain a desired speed during vehicle operation. The arousal estimation system 723 may also adjust acceleration control system 764, in some examples.

[0139] Control elements positioned on an outside of a vehicle (e.g., controls for a security system) may also be connected to in-vehicle computing system or infotainment system 609, such as via inter-vehicle system communication module 722. The control elements of the vehicle control system may be physically and permanently positioned on and / or in the vehicle for receiving user input. In addition to receiving control instructions from in-vehicle computing system or infotainment system 609, vehicle control system 730 may also receive input from one or more external devices 650 operated by the user, such as from mobile device 628. This allows aspects of vehicle systems 731 and vehicle control elements 761 to be controlled based on user input received from the external devices 650.

[0140] In-vehicle computing system or infotainment system 609 may further include one or more antennas 706. The in-vehicle computing system may obtain broadband wireless internet access via antennas 706, and may further receive broadcast signals such as radio, television, weather, traffic, and the like. The in-vehicle computing system or infotainment system 609 may receive positioning signals such as GPS signals via antennas 706. The in-vehicle computing system may also receive wireless commands via radio frequency (RF) such as via antennas 706 or via infrared or other means through appropriate receiving devices. In some embodiments, antenna 706 may be included as part of audio system 732 or telecommunication system 736. Additionally, antenna 706 may provide AM / FM radio signals to external devices 650 (such as to mobile device 628) via external device interface 712.

[0141] One or more elements of the in-vehicle computing system or infotainment system 609 may be controlled by a user via user interface 718 (e.g., UI 510). User interface 718 may include a graphical user interface presented on a touch screen, such as touch screen 608 and / or display screen 611 of FIG. 6, and / or user-actuated buttons, switches, knobs, dials, sliders, and so on. For example, user-actuated elements may include steering wheel controls, door and / or window controls, instrument panel controls, audio system settings, climate control system settings, and the like. A user may also interact with one or more applications of the in-vehicle computing system or infotainment system 609 and mobile device 628 via user interface 718, or the arousal estimation system 723. In addition to receiving a user’s vehicle setting preferences on user interface 718, vehicle settings selected by in-vehicle control system may be displayed to a user on user interface 718. Notifications and other messages (e.g.. received messages), as well as navigational assistance, may be displayed to the user on a display of theAttorney Docket No. P240449WOuser interface. User preferences / information and / or responses to presented messages may be performed via user input to the user interface.

[0142] The in-vehicle computing system or infotainment system 609 may include a DMS 721. The DMS 721 may receive data from various sensors and / or systems of the vehicle (e.g., sensor subsystem 710, cameras 725, microphone 702) and may monitor aspects of driver behavior to improve a performance of the vehicle and / or a driving experience of the driver. In some examples, one or more outputs of the DMS 721 may be inputs into the arousal estimation system 723. In various embodiments, the arousal estimation system 723 may be used to predict an arousal of the driver, and adjust one or more controls of the vehicle control system 730 based on the predicted arousal of the driver.

[0143] Thus, an arousal estimation system is proposed, which may continuously monitor physiological, camera, and other sensor data of the driver to determine whether an arousal level of the driver is within a desired range that is optimal for operating the vehicle. An arousal score of the driver may be estimated by an arousal estimation model trained on driving simulation data. The arousal estimation system may intervene in a first manner when the estimated arousal score decreases below a first threshold, which may indicate a low level of arousal of the driver that may manifest as drowsiness or fatigue. The arousal estimation system may intervene in a second manner when the estimated arousal score increases above a second threshold, which may indicate a high level of arousal that may manifest as cognitive load or stress of the driver. When the estimated arousal score decreases below the first threshold cognitive load, the arousal estimation system may notify the driver, and / or adjust one or more settings of the vehicle to increase the arousal of the driver. The one or more settings may be adjusted based on, or proportional to a first difference between the arousal score and the first threshold. When the estimated arousal score increases above the second threshold, the arousal estimation system may notify the driver, and / or adjust the one or more settings of the vehicle to reduce the arousal of the driver. The one or more settings may be adjusted based on, or proportional to a second difference between the arousal score and the second threshold.

[0144] In this way, a broader and more holistic solution is provided to detecting psychological and / or psychophysiological states of a driver than current approaches. The technical effect of training an ML model to estimate various psychophysiological states of a driver on a single arousal scale is that an accuracy of state detection may be greater than an accuracy of individual state detectors, even when combined or used in conjunction with each other. Conventionally, individual cognitive / mental states are typically qualitatively defined. As a result, approaches and methods for assessing the same psychophysiological states can beAttorney Docket No. P240449WOdifferent depending on different understandings of what each state represents. For example, both cognitive load and stress can be modeled through task difficulty- and intensity-, and often the same tasks are used for modelling both states, without a clear distinction between the two. Even defining and quantifying the state of drowsiness is not straightforward and is often done in qualitative descriptions of the state. Thus, implementing a quantitative approach to state detection and prediction that transcends individual states has clear limitations in terms of accuracy and reliability.

[0145] As an example, an existing state-detection system may claim to detect drowsiness as assessed qualitatively through self-reports, with individuals identifying themselves as either drowsy or not. This binary classification does not allow for the determination of varying levels of drowsiness. The inherent subjectivity and the all-or-nothing nature of such self-assessments preclude any differentiation between degrees of drowsiness. This highlights a fundamental flaw in both the terminology and methodology used to describe states of being.

[0146] Furthermore, human states may be confused when considered separately. A ty pical state detector output is normally associated with a particular aspect of the human state, such as drowsiness, cognitive load or stress. However, a psychophysiologcial state at each point in time includes or reflects numerous aspects of the person’s behaviour and functioning. Because states such as drowsiness and high cognitive load may impact driving performance in a similar way (e.g., both states increase reaction time and cause more inappropriate lane exits), driving behaviour may be helpful in disambiguating between different states. In addition, sensors that measure physiological data typically have some level of noise that affects the accuracy of state detection. For example, a typical accuracy may be between 70% and 90%, depending on the task. In a driving context, the detection of biosignals may be based on sensors that are remote from the driver. Signals from such remote sensors may have a higher noise level than signals from sensors in contact with a skin of the driver. The automotive environment may be particularly noisy, due to mechanical and environmental influences, which makes it difficult to achieve high accuracies in state detection. Thus, on one hand, biosignals can be misinterpreted due to noise and artifacts. On the other hand, certain states may be difficult to classify because of the way they are represented based on biosignals, when viewed separately. For example, in a drowsy state, people yawn frequently, which may be accompanied by breathing-related increases in heart rate which could be mistaken for episodes of acute stress if outputs of a stress detector alone were considered. In contrast, such yawns / breathing related changes in heart rate may be classified more accurately by the arousal estimation model described herein, because low- arousal levels related to drowsiness cannot be mistaken for stress.Attorney Docket No. P240449WO

[0147] Thus, interpreting states as independent phenomena, such as being in a drowsy state or not, stressed or not, and experiencing cognitive load or not. is scientifically inaccurate. Ultimately, these states are manifestations of varying levels of arousal. Attempting to interpret them outside the arousal scale as distinct and disconnected phenomena results in flaws within technical implementations of state detectors. For instance, if separate detectors exist for each psychophysiological state, the separate detectors could technically report different simultaneously detected respective states, even when the states are mutually exclusive when interpreted on the arousal scale. The arousal scale is more general regarding the psychophysiological states, because it characterizes an overall level of psychophysiological activation and alertness, which may be easier to interpret, especially in the automotive industry where the focus is on identifying risks and declines in performance. Arousal is a defining component of any qualitatively described state that can be used as a quantitative measure of the state.

[0148] Additionally, by estimating different psychophysiological states based on a single arousal scale, more precise ground truth data may be used, which may increase an overall performance of ML implementations of the arousal estimation model. For example, a first set of labeled training data used to train a drowsiness detector model need not include samples that are specifically labeled to disambiguate drowsiness from cognitive load, for example. A second set of labeled training data used to train a stress detector model need not include samples that are specifically labeled to disambiguate stress from cognitive load. The sensors used to detect stress may be different from the sensors used to detect drowsiness. Thus, an advantage of the arousal estimation model described herein is that it is trained on ground truth data collected in driving simulation (and / or on-road studies) that involve modelled and naturalistic conditions that encompass various states. The model’s input data and ground truth data is generated from a wider variety of sensors than would be used by an individual state detector, including heart inter-beat interval (IBI) data, eye gaze metrics, behavioral events, etc. Some sensor data undergoes labelling procedures, such as stress labelling with IBI data and behavioral labelling of drowsiness, performed by experts or done using automated processes. For example, closed eyes events that suggest that the person fell asleep and is not able to control the vehicle safely can be identified and labeled, and drowsiness can be identified as a certain period before the closed eyes event. The ground truth for arousal corresponding to stress can be based on labelled IBI sequences that determine acute stress events based on certain pattern of cardiac dynamics. The ground truth for cognitive load can be based on task difficulty, self-reports and other variables and their combinations. The ground truth for drowsiness can be identified throughAttorney Docket No. P240449WOdrowsy episodes or EEG data. For example, experts in human or animal behavior can label behaviors indicative of drowsiness, such as yawning and nodding. Additionally, specific brainwave patterns may also signal aspects of drowsiness. Because there may be no definitive ground truth for all human states and arousal levels (e.g., no scale or device exists to measure them directly), the disclosed arousal estimation system advantageously leverages partial correlations between arousal levels and certain biosignals and behaviors, which may be different for different categories of arousal corresponding to different psychophysiological states. By employing advanced statistical and machine learning methods, these correlations can be refined and used as approximations of the ground truth for arousal levels associated with the different psychophysiological states. As a result, the arousal estimation model described herein may generate more accurate estimations of a driver’s ability to perform driving tasks, and interventions may be made in a more timely manner. Further, unlike other conventional state detectors, the interventions can be tailored to the driver’s arousal level.

[0149] An additional technical advantage of the proposed arousal estimation model is that the ground truth data used to train the model may be generated automatically in a manner that does not rely on manual labeling, by mapping an assessment of a performance of a driving task to an arousal score using the Yerks-Dodson Law arousal curve.

[0150] The disclosure also provides support for an arousal estimation system, comprising: a processor, and a memory storing instructions that when executed, cause the processor to: estimate a level of arousal of a human, on a predefined arousal scale, using a model, based on information received from sensors, generate an arousal score of the human based on the estimated level of arousal, detect that the arousal score exceeds a threshold value, and in response, notify and / or perform an intervention with the human. In a first example of the system, the human is a driver of a vehicle, and further instructions are stored in the memory that when executed, cause the processor to: during operation of the vehicle by the driver: in response to detecting that the arousal score is below a first threshold value, make a first adjustment to an environmental parameter of a cabin of the vehicle in a first direction to increase the level of arousal of the driver, the first adjustment based on a difference between the arousal score of the driver and the first threshold value, in response to detecting that the arousal score is above a second threshold value, make a second adjustment to the environmental parameter of the vehicle in a second direction to reduce the level of arousal of the driver, the second adjustment based on a difference between the arousal score of the driver and the second threshold value. In a second example of the system, optionally including the first example, one of the first adjustment and the second adjustment is a proportional adjustment. In a thirdAttorney Docket No. P240449WOexample of the system, optionally including one or both of the first and second examples,: the environmental parameter is one of a temperature of the cabin, an illumination of the cabin, and a flow of air through the cabin from a heating, ventilation, and air conditioning (HVAC) system, the first adjustment is one of decreasing the temperature of the cabin, increasing the illumination of the cabin, and increasing the flow of air through the cabin, and the second adjustment is one of increasing the temperature of the cabin, decreasing the illumination of the cabin, and decreasing the flow of air through the cabin. In a fourth example of the system, optionally including one or more or each of the first through third examples, the information received from the sensors includes at least one of: images captured by an in-cabin camera of the vehicle, physiological data of the driver acquired via a biosensor in contact with a skin of the driver, physiological data of the driver acquired remotely via a biosensor in the cabin, and audio data captured by an in-cabin microphone of the vehicle. In a fifth example of the system, optionally including one or more or each of the first through fourth examples, the arousal scale is based on the Yerks-Dodson Law arousal curve, and includes a first range of values corresponding to drowsiness, a second range of values corresponding to fatigue, a third range of values corresponding to an optimal arousal range, a fourth range of values corresponding to cognitive load, and a fifth range of values corresponding to stress. In a sixth example of the system, optionally including one or more or each of the first through fifth examples, the model is an ML model trained on training data including: a first set of sensor data collected in real time from a plurality of users simulating driving in a driving simulator, from a first set of sensors arranged in a cabin of the driving simulator, the first set of sensors identical to the sensors of the cabin of the vehicle, and ground truth data generated from a second set of sensor data collected in real time concurrently with the first set of sensor data from a second set of sensors arranged in the cabin. In a seventh example of the system, optionally including one or more or each of the first through sixth examples, the second set of sensors include one of an electroencephalogram (EEG) sensor and an electrocardiogram (ECG) sensor. In a eighth example of the system, optionally including one or more or each of the first through seventh examples, the training data includes a set of training triplets, each training triplet including a first portion of the first set of sensor data corresponding to a first duration while a user is simulating driving, a second portion of the first set of sensor data corresponding to a second duration while the same user is simulating driving, and an annotation as ground truth data, the annotation indicating whether a first arousal score assigned to the first portion is greater than a second arousal score assigned to the second portion. In a ninth example of the system, optionally including one or more or each of the first through eighth examples, the model is anAttorney Docket No. P240449WOML model trained on training data including: a first set of sensor data collected in real time from a plurality of users simulating driving in a driving simulator, from a first set of sensors arranged in a cabin of the driving simulator, the first set of sensors identical to the sensors of the cabin of the vehicle, and ground truth data generated from assessments of a performance of the users on a driving task.

[0151] The disclosure also provides support for a method for training a machine learning (ML) model to estimate an arousal level of a driver of a vehicle based on sensor data of the vehicle, the method comprising: receiving a first set of sensor data of a plurality of users of a driving simulator in real time, receiving a second set of sensor data of the plurality of users in real time concurrently with the first set of sensor data, dividing the first set of sensor data and concurrently received second set of sensor data into portions of a predefined duration, assigning arousal scores to the portions of the first set of sensor data based on the concurrently received portions of the second set of sensor data, creating a set of training triplets, each training triplet including a first portion of the first set of sensor data corresponding to a first duration while a user is simulating driving, a second portion of the second set of sensor data corresponding to a second duration while the same user is simulating driving, and an annotation as ground truth data, the annotation indicating whether a first arousal score assigned to the first portion is greater than a second arousal score assigned to the second portion, and training the ML model on the training triplets. In a first example of the method, the first set of sensor data includes one or more of images from a camera, physiological data acquired from the user via a photoplethysmogram (PPG) sensor, and driving data recorded by the driving simulator. In a second example of the method, optionally including the first example, the second set of physiological sensor data includes one or more of images from a camera, electroencephalogram (EEG) data, and electrocardiogram (ECG) data of the users. In a third example of the method, optionally including one or both of the first and second examples, the arousal scores are assigned based on an arousal scale based on the Yerks-Dodson Law arousal curve, the arousal scale including a first range of values corresponding to drowsiness, a second range of values corresponding to fatigue, a third range of values corresponding to an optimal arousal range, a fourth range of values corresponding to cognitive load, and a fifth range of values corresponding to stress. In a fourth example of the method, optionally including one or more or each of the first through third examples, the first set of sensor data and the second set of sensor data are collected during different driving scenarios corresponding to the different ranges of values, in which the users are subjected to different conditions or asked to perform different tasks. In a fifth example of the method, optionally including one or more or each ofAttorney Docket No. P240449WOthe first through fourth examples, the ML model includes an output layer with two output nodes, and the method further comprises outputting a first estimated arousal score corresponding to the first portion at a first output node of the two output nodes, and outputting a second estimated arousal score corresponding to the second portion at a second output node of the two output nodes. In a sixth example of the method, optionally including one or more or each of the first through fifth examples, training the ML model on the training triplets further comprises: at each training cycle: generating the first estimated arousal score at the first output node, generating the second estimated arousal score at the second output node, calculating a comparator value indicating the greater of the first estimated arousal score and the second estimated arousal score, and adjusting weights and biases of the ML model based on backpropagating an error between the comparator value and the annotation through the ML model.

[0152] The disclosure also provides support for a method for training a machine learning (ML) model to estimate an arousal level of a driver of a vehicle based on sensor data of the vehicle, the method comprising: receiving portions of sensor data collected in real time from a plurality of users of a driving simulator during a performance of a driving task, the sensor data including at least images from a camera, data from a photoplethysmogram (PPG) sensor, and driving data recorded by the driving simulator, mapping the portions of sensor data to arousal scores on an arousal scale based on quantitative assessments of the performance calculated by the driving simulator, using a predetermined plot of the performance as a function of arousal, generating a training dataset of training pairs, each training pair including a portion of the sensor data as input data and a mapped arousal score as ground truth data, and training the ML model on the training dataset. In a first example of the method, the predetermined plot is the Yerks-Dodson Law arousal curve. In a second example of the method, optionally including the first example, the driving task is a braking task and the quantitative assessments of the performance are reaction times of the users.

[0153] The description of embodiments has been presented for purposes of illustration and description. Suitable modifications and variations to the embodiments may be performed in light of the above description or may be acquired from practicing the methods. For example, unless otherwise noted, one or more of the described methods may be performed by a suitable device and / or combination of devices, such as the embodiments described above with respect to FIGS. 1-8. The methods may be performed by executing stored instructions with one or more logic devices (e.g., processors) in combination with one or more hardware elements, such as storage devices, memory, hardware network interfaces / antennas, switches, clock circuits, andAttorney Docket No. P240449WOso on. The described methods and associated actions may also be performed in various orders in addition to the order described in this application, in parallel, and / or simultaneously. The described systems are exemplary in nature, and may include additional elements and / or omit elements. The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various systems and configurations, and other features, functions, and / or properties disclosed.

[0154] As used in this application, an element or step recited in the singular and proceeded with the word '‘a” or ‘'an” should be understood as not excluding plural of said elements or steps, unless such exclusion is stated. Furthermore, references to “one embodiment” or “one example” of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. The terms “first,” “second,” “third,” and so on are used merely as labels, and are not intended to impose numerical requirements or a particular positional order on their objects. The following claims particularly point out subject matter from the above disclosure that is regarded as novel and non-obvious.

Claims

Attorney Docket No. P240449WOCLAIMS:

1. An arousal estimation system, comprising:a processor, and a memory storing instructions that when executed, cause the processor to:estimate a level of arousal of a human, on a predefined arousal scale, using a model, based on information received from sensors;generate an arousal score of the human based on the estimated level of arousal; detect that the arousal score exceeds a threshold value, and in response, notify and / or perform an intervention with the human.

2. The arousal estimation system of claim 1, wherein the human is a driver of a vehicle, and further instructions are stored in the memory that when executed, cause the processor to:during operation of the vehicle by the driver:in response to detecting that the arousal score is below a first threshold value, make a first adjustment to an environmental parameter of a cabin of the vehicle in a first direction to increase the level of arousal of the driver, the first adjustment based on a difference between the arousal score of the driver and the first threshold value; in response to detecting that the arousal score is above a second threshold value, make a second adjustment to the environmental parameter of the vehicle in a second direction to reduce the level of arousal of the driver, the second adjustment based on a difference between the arousal score of the driver and the second threshold value.

3. The arousal estimation system of claim 2. wherein one of the first adjustment and the second adjustment is a proportional adjustment.

4. The arousal estimation system of claim 2, wherein:the environmental parameter is one of a temperature of the cabin, an illumination of the cabin, and a flow of air through the cabin from a heating, ventilation, and air conditioning (HVAC) system;the first adjustment is one of decreasing the temperature of the cabin, increasing the illumination of the cabin, and increasing the flow of air through the cabin; andthe second adjustment is one of increasing the temperature of the cabin, decreasing the illumination of the cabin, and decreasing the flow of air through the cabin.Attorney Docket No. P240449WO5. The arousal estimation system of claim 2. wherein the information received from the sensors includes at least one ofimages captured by an in-cabin camera of the vehicle;physiological data of the driver acquired via a biosensor in contact with a skin of the driver;physiological data of the driver acquired remotely via a biosensor in the cabin; and audio data captured by an in-cabin microphone of the vehicle.

6. The arousal estimation system of claim 1, wherein the arousal scale is based on the Yerks-Dodson Law arousal curve, and includes a first range of values corresponding to drowsiness, a second range of values corresponding to fatigue, a third range of values corresponding to an optimal arousal range, a fourth range of values corresponding to cognitive load, and a fifth range of values corresponding to stress.

7. The arousal estimation system of claim 2. wherein the model is an ML model trained on training data including:a first set of sensor data collected in real time from a plurality of users simulating driving in a driving simulator, from a first set of sensors arranged in a cabin of the driving simulator, the first set of sensors identical to the sensors of the cabin of the vehicle; and ground truth data generated from a second set of sensor data collected in real time concurrently with the first set of sensor data from a second set of sensors arranged in the cabin.

8. The arousal estimation system of claim 7, wherein the second set of sensors include one of an electroencephalogram (EEG) sensor and an electrocardiogram (ECG) sensor.

9. The arousal estimation system of claim 8, wherein the training data includes a set of training triplets, each training triplet including a first portion of the first set of sensor data corresponding to a first duration while a user is simulating driving, a second portion of the first set of sensor data corresponding to a second duration while the same user is simulating driving, and an annotation as ground truth data, the annotation indicating whether a first arousal score assigned to the first portion is greater than a second arousal score assigned to the second portion.Attorney Docket No. P240449WO10. The arousal estimation system of claim 2, wherein the model is an ML model trained on training data including:a first set of sensor data collected in real time from a plurality of users simulating driving in a driving simulator, from a first set of sensors arranged in a cabin of the driving simulator, the first set of sensors identical to the sensors of the cabin of the vehicle; and ground truth data generated from assessments of a performance of the users on a driving task.

11. A method for training a machine learning (ML) model to estimate an arousal level of a driver of a vehicle based on sensor data of the vehicle, the method comprising:receiving a first set of sensor data of a plurality of users of a driving simulator in real time;receiving a second set of sensor data of the plurality of users in real time concurrently with the first set of sensor data;dividing the first set of sensor data and concurrently received second set of sensor data into portions of a predefined duration;assigning arousal scores to the portions of the first set of sensor data based on the concurrently received portions of the second set of sensor data;creating a set of training triplets, each training triplet including a first portion of the first set of sensor data corresponding to a first duration while a user is simulating driving, a second portion of the second set of sensor data corresponding to a second duration while the same user is simulating driving, and an annotation as ground truth data, the annotation indicating whether a first arousal score assigned to the first portion is greater than a second arousal score assigned to the second portion; andtraining the ML model on the training triplets.

12. The method of claim 11, wherein the first set of sensor data includes one or more of images from a camera, physiological data acquired from the user via a photoplethysmogram (PPG) sensor, and driving data recorded by the driving simulator.

13. The method of claim 11, wherein the second set of physiological sensor data includes one or more of images from a camera, electroencephalogram (EEG) data, and electrocardiogram (ECG) data of the users.Attorney Docket No. P240449WO14. The method of claim 11, wherein the arousal scores are assigned based on an arousal scale based on the Yerks-Dodson Law arousal curve, the arousal scale including a first range of values corresponding to drowsiness, a second range of values corresponding to fatigue, a third range of values corresponding to an optimal arousal range, a fourth range of values corresponding to cognitive load, and a fifth range of values corresponding to stress.

15. The method of claim 14, wherein the first set of sensor data and the second set of sensor data are collected during different driving scenarios corresponding to the different ranges of values, in which the users are subjected to different conditions or asked to perform different tasks.

16. The method of claim 11, wherein the ML model includes an output layer with two output nodes, and the method further comprises outputting a first estimated arousal score corresponding to the first portion at a first output node of the two output nodes, and outputting a second estimated arousal score corresponding to the second portion at a second output node of the two output nodes.

17. The method of claim 1, wherein training the ML model on the training triplets further comprises:at each training cycle:generating the first estimated arousal score at the first output node; generating the second estimated arousal score at the second output node; calculating a comparator value indicating the greater of the first estimated arousal score and the second estimated arousal score; andadjusting weights and biases of the ML model based on backpropagating an error between the comparator value and the annotation through the ML model.

18. A method for training a machine learning (ML) model to estimate an arousal level of a driver of a vehicle based on sensor data of the vehicle, the method comprising:receiving portions of sensor data collected in real time from a plurality of users of a driving simulator during a performance of a driving task, the sensor data including at least images from a camera, data from a photoplethysmogram (PPG) sensor, and driving data recorded by the driving simulator;Attorney Docket No. P240449WOmapping the portions of sensor data to arousal scores on an arousal scale based on quantitative assessments of the performance calculated by the driving simulator, using a predetermined plot of the performance as a function of arousal;generating a training dataset of training pairs, each training pair including a portion of the sensor data as input data and a mapped arousal score as ground truth data; and training the ML model on the training dataset.

19. The method of claim 18, wherein the predetermined plot is the Yerks-Dodson Law arousal curve.

20. The method of claim 18, wherein the driving task is a braking task and the quantitative assessments of the performance are reaction times of the users.

Images