System for the analysis and validation of emergency interventions of a vehicle according to a behavioral state of a driver

Abstract

Electronic system (1) designed to analyse and validate of emergency interventions of a vehicle according to a behavioural state of a subject driving the vehicle and comprising: a sensory system (2) configured to output signals indicative of physiological and image data related to the subject; and an electronic processing unit (3) in communication with the sensory system and being configured to receive the signals and process them to output corresponding statistical data related to the subject. The electronic system (1) is designed to process the data outputted by electronic processing unit (3) to determine a behavioural state and / or transitions thereof to determine the sleepiness level, the motion sickness level and the emotional state of the subject and generate a feedback to provide to an end user and provide emergency intervention to the subject based on the sleepiness level, the motion sickness level and the emotional state of the subject.

Description

[0001] SYSTEM FOR THE ANALYSIS ND VALIDATION OF EMERGENCY INTERVENTIONS OF A VEHICLE ACCORDING TO A BEHAVIORAL STATE OF A DRIVER

[0002] Cross-Reference to Related Applications

[0003] This Patent Application claims priority from European Patent Application No. 23193432.4 filed on August 25, 2023 and from Italian Patent Application No. 102024000019135 filed on August 21, 2024, the entire disclosure of which is incorporated herein by reference.

[0004] Technical Field of the Invention

[0005] The present invention generally relates to real-time detection and / or prediction of one or more behavioural states and / or transitions between behavioural states, in particular to analyse and validate of emergency interventions of a vehicle according to a behavioural state of a subject driving the vehicle.

[0006] Thus, the present invention aims at providing a system for the analysis and validation of emergency interventions of a vehicle according to a behavioural state of a driver or subject, thereby providing means for performing said validation tests in a safe manner.

[0007] State of the Art

[0008] As is known, the continuous evolution of vehicle technologies offers new interesting solutions and, at the same time and in certain cases, may pose new potential issues. In particular, the electrification of the powertrain of a vehicle, in particular hybrid- or fully electric -propelled ones, and the autonomous driving technologies may induce motion sickness in a more pronounced manner with respect to traditional vehicles; on this regard, it is known that around 25-30% of the population regularly suffers from motion sickness, a figure which some reckon to be conservative. Symptoms of this poorly understood illness include nausea, sweating, pallor, hypothermia, headaches and vomiting. Mildly affected patients might also experience drowsiness, apathy or decreased cognitive abilities. In particular, it is estimated that 60 to 70% of travellers will suffer from it at some point in their lives.

[0009] Motion sickness is experienced most commonly in cars, giving rise to the well-known term “car sickness”, as the more specific version of “motion sickness”; in these cases, passengers are prone to feeling sick either because they are deprived of the capacity to anticipate trajectories, in contrast to drivers or because of the acceleration / deceleration profiles of the electrical vehicles used for driving.

[0010] It is furthermore noted that, by nature, an electric motor is more linear and quieter than a combustion engine; this advantage has the downside of preventing certain car users from assimilating the movement of the vehicle. For example, whereas the passengers would associate acceleration with the engine revving in classic cars, electric cars suddenly deprive them of this reference point. In addition, the combustion engine’s vibrations, which some perceive as soothing, are also drastically reduced or eliminated. The use of regenerative braking, which captures the kinetic energy from braking and converts it into the electrical power that charges the vehicle’s high voltage battery, can also upset passengers’ balance; the decelerations induced by this system are usually low frequency, which is typical of a sickness -inducing motion force.

[0011] Another technological advance inducing motion sickness is the growing presence of ever larger and numerous screens inside vehicles; in particular, these screens overburden users with visual information, which discourages them from looking outside, thereby making them lose the ability to take in the “correct” visual signals - i.e. the external view of the vehicle - which allow them to correctly perceive their position in space, inducing, in turn, motion sickness. Furthermore, when it comes to car design, the current trend consisting in inserting screens in as many nooks as possible makes it harder for the passengers to fight motion sickness; in fact, the rise of screens in cars is likely to increase in the coming years, including vehicles that could even feature screens on glass surfaces or offer on-board virtual reality experiences. This invasive environment can, in turn, impact upon passengers’ wellbeing. Indeed, the mere knowledge one is likely to suffer nausea from screens can stress vulnerable passengers, with research linking up to 40% of motion sickness symptoms to passenger psychology.

[0012] Furthermore, the race among car manufacturers to create the first fully-automated vehicle is also likely to worsen the problem. While today’s vehicles are only partially automated, in future, they will be able to pilot themselves. As mentioned above, this is problematic when the driver knows the act of driving is the best way to anticipate trajectories and curb symptoms. Moreover, the disappearance of the driving cockpit will make it possible to redesign vehicle interiors to become more welcoming, like a rolling living room. These new configurations will give passengers more freedom, allowing them for example to turn their seat rearwards-facing to chat with other occupants. In addition, removing the driving cockpit to completely reorganize the cabin could be detrimental, as many cannot stand not driving or seating rearwards-facing.

[0013] Another promise of the autonomous vehicle is to allow its passengers to devote “idle” travel time to productive tasks or entertainment. In fact, the increasing appeal of taxi and Uber travel, where users tend to gaze at their digital devices, goes hand in hand with this trend; once again, such distractions deter passengers from engaging with the landscape.

[0014] Finally, the incidence of motion sickness ultimately remains moderate in non-automated cars because of drivers’ ability to adapt their driving style when their passengers report discomfort. This human dimension is set to disappear in autonomous vehicles, whose driving style will be less flexible and less natural than that of a human driver.

[0015] Furthermore, the validation of in-cabin monitoring technologies, like drowsiness monitoring, currently rely on subjective assessment. In fact, the Karolinska Sleepiness Scale (KSS) uses a multi-level scale and the status of the driver is assessed by an observer at a regular frequency. More generally the current behavioural measurements through image sensors are based on face detection, yawning, eyes detection.

[0016] Ob ject and Summary of the Invention

[0017] The Applicant has noted that the abovementioned solutions fall short in several relevant scenarios.

[0018] With respect to image-based measures, in fact, the system’s performance is severely affected in cases where it is difficult to track facial data due to obstacles. In addition, they provide details on the subject's state of sleepiness at a very advanced stage, when his cognitive state is no longer able to carry out its activity. Above all, they cannot cover the large population of people who fall asleep with their eyes open, especially people with OSAS (Obstructive Sleep Apnea Syndrome). Consequently, any validation cannot be performed in a real vehicle and in realistic conditions, since it should reach the point when the driver is no longer able to control the vehicle because of the excessive fatigue level.

[0019] The Applicant notes that same holds also for new systems monitoring and, possibly, predicting motion sickness.

[0020] The Applicant furthermore notes that, in a recent keynote held at InCabin 2023 ^Outlook for Occupant Status Monitoring" , Adriano Palao - EuropNCAP ADAS VP, Brussels), EuroNCAP raised the concern about the real effectiveness of the very simplistic implementations to determine drowsiness by OEMs. Most importantly, EuroNCAP clearly admitted that a realistic validation method to assess the real performance of an in-cabin monitoring system once integrated in a production vehicle is not available.

[0021] The object of the present invention is to provide a system for the analysis and validation of emergency interventions of a vehicle according to a behavioural state of a driver that solves at least in part the problems of the known solutions.

[0022] According to the present invention, a system for the analysis and validation of emergency interventions of a vehicle according to a behavioural state of a driver is provided, as claimed in the appended claims.

[0023] Brief Description of the Drawings Figure 1 schematically shows an electronic system according to the present invention.

[0024] Figure 2 schematically shows a sleep prediction algorithm implemented by a sleep prediction control logic of an electronic system according to the present invention.

[0025] Figure 3 schematically shows a motion sickness algorithm implemented by a motion sickness control logic of an electronic system according to the present invention.

[0026] Figure 4 schematically shows an emotional state algorithm implemented by an emotional state control logic of an electronic system according to the present invention.

[0027] Figure 5 shows a block diagram of an attention detection algorithm implemented by a control logic of an electronic system according to the present invention.

[0028] Figure 6 shows Realistic validation on Dynamic Vehicle Simulator.

[0029] Detailed Description of Preferred Embodiments of the Invention

[0030] The present invention will now be described in detail with reference to the accompanying drawings in order to allow a skilled person to implement it and use it. Various modifications to the described embodiments will be readily apparent to those of skill in the art and the general principles described may be applied to other embodiments and applications without however departing from the protective scope of the present invention as defined in the appended claims. Therefore, the present invention should not be regarded as limited to the embodiments described and illustrated herein but should be allowed the broadest protection scope consistent with the features described and claimed herein.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning commonly understood by one of ordinary skill in the art to which the invention belongs. In case of conflict, the present specification, including the definitions provided, will control. Furthermore, the examples are provided for illustrative purposes only and as such should not be considered limiting.

[0032] In particular, the block diagrams included in the attached figures and described below are not to be understood as a representation of the structural features, i.e. constructional limitations, but must be understood as a representation of functional features, i.e. intrinsic properties of the devices defined by the effects obtained, that is to say functional restrictions, which can be implemented in different ways, so as to protect the functionalities thereof (operational capability).

[0033] In order to facilitate the understanding of the embodiments described herein, reference will be made to some specific embodiments and a specific language will be used to describe the same. The terminology used herein is used for the purpose of describing particular embodiments only and is not intended to limit the scope of the present invention.

[0034] As it will be apparent from the disclosure set in the following paragraphs, the present electronic system brings together multiple sensors acquiring different and complementary information, specifically both biometric and images, that intrinsically provide a thorough view to an end user of a subject’s cognitive level. In particular, the sleep onset prediction capability is based on the seamless interaction among the different sensing and processing units of the electronic system itself; as a result, a broader coverage of comer cases may be obtained, providing a robust solution with respect to the ones currently available.

[0035] As better described in the following paragraphs, the present invention proposes an innovative method to analyse and predict motion sickness based on a simplified model of the autonomic response by monitoring various physiological parameters. Such a simplified model of the autonomous nervous system can be effectively used for advanced path tracking control of the vehicle aimed at minimising the motion sickness of the subject.

[0036] The present system is based on the smart combination of vision sensors and contactless sensors and can also include the adoption of wearables; in particular, contactless sensors and wearables are capable to extract the physiological parameters needed to feed the simplified model of the autonomous nervous system, while the vision sensor provides details about the behavioural state of the subject. More importantly, the present invention provides a system architecture to safely validate the system on the vehicle. Such a solution can be successfully applied in all the critical circumstances when the driver is no longer able to control the vehicle due to motion sickness, drowsiness and impairment (e.g. heart attack).

[0037] In consideration of the present invention, the Applicant noted that the extensive autonomic nervous system symptomatology, and the reported benefit of medications that manipulate central autonomic receptors imply that research of the autonomic nervous system might contribute to the diagnosis and prediction of motion sickness. Power spectral analysis of heart rate variability (HRV) provides a quantitative assessment of the autonomic control of the cardiovascular / cardiorespiratory system. When exposed to Coriolis cross-coupled stimulation, seasickness-susceptible subjects showed a significant reduction in the mid- and high-frequency power of HRV, indicating a decrease in parasympathetic activity during the time course of motion sickness. While HRV in the mid-frequencies recovered during the post-rotation period, the high-frequency decrement was maintained, suggesting that the decrease in parasympathetic activity persisted after rotation. Similar changes in HRV have recently been reported in motion sickness-susceptible subjects when exposed to vection (z.e. the sensation of self-motion in the absence of physical motion), thereby producing stimulus.

[0038] As better described in the following paragraphs, the present invention provides a system based on a simplified model of the autonomic nervous system (ANS) that controls the behaviour of the cardiorespiratory system (CRS); in particular, the CRS behaviour is monitored in real time through algorithms for sleep prediction, motion sickness and emotional state. In further detail, the abovementioned algorithm is designed to process a reduced number of physiological parameters, which are measured either by means of contact-based sensing devices (e.g., custom or commercial off-the-shelf smart watches) or contactless sensing devices (e.g., RF sensors such as radar or camera-based sensors), to derive various statistical parameters. Vision sensors e.g. camera) allow to add details about facial, eyes and mouth movements as well as the attention level of the subject. Based on the variability of such statistical parameters over time, the present invention allows to classify a subject’s drowsiness state, and it identifies the sleep onset. In a similar manner, the present invention allows to provide early indication about the motion sickness as well as the emotional state of the subject.

[0039] Thus, as better described in the following paragraphs, the system according to the present invention allows to:

[0040] - acquire physiological signals from multiple sensors, such as a wearable sensor and / or a contactless sensor, as well as image signals from the vision sensors;

[0041] - provide a physical interface to allow transfer of physiological and image data from the sensors to an electronic processing unit which runs a control logic; and

[0042] - generate, through the control logic, an output to be presented to the end user as local and / or remote feedback.

[0043] Furthermore, the validation of an in-cabin monitoring system will follow multi-step approach, in order to provide emergency intervention measures for the subject or driver of the vehicle.

[0044] Figure 1 schematically shows an electronic system 1 designed to analyse and validate of emergency interventions of a vehicle according to a behavioural state of a subject driving the vehicle. In particular, the electronic system 1 comprises:

[0045] - a sensory system 2 configured to output signals indicative of physiological and image data related to the subject; and

[0046] - an electronic processing unit 3 in communication with the sensory system and being configured to receive the signals and process them to output corresponding statistical data related to the subject.

[0047] In particular, the electronic system 1 is designed to process the data outputted by electronic processing unit 3 to determine a behavioural state and / or transitions thereof to determine the sleepiness level, the motion sickness level and the emotional state of the subject and generate a feedback to provide to an end user and provide emergency intervention to the subject based on the sleepiness level, the motion sickness level and the emotional state of the subject, in particular at an interface 11 to the vehicle system (not shown) of the vehicle. Exemplarily, the interface 11 comprises a display system (not shown) for providing the determined information to the subject; according to an aspect of the present invention, the interface 11 is configured to communicate with the vehicle system of the vehicle to implement the emergency interventions, as better disclosed in the following.

[0048] Furthermore, the electronic system 1 further comprises a control logic 4 in communication with electronic processing unit 3 ,here integrated therein, and configured to receive the data outputted by electronic processing unit 3) and process them to determine a behavioural state and / or transitions thereof to determine the sleepiness level, the motion sickness level and the emotional state of the subject and generate a feedback to provide to an end user and provide emergency intervention to the subject based on the sleepiness level, the motion sickness level and the emotional state of the subject.

[0049] Thus, as also better described in the following paragraphs, the present electronic system 1 provides a sense-compute-act methodology upon a prediction warning. In particular, the sensory system 2 provides the sense portion of the methodology, as it allows to acquire the physiological signals to be computed by the electronic processing unit 3; specifically, the computing portion of the methodology is implemented by the electronic processing unit 3, which is furthermore configured to carry out a prediction step through the control logic 4 for determining the sleepiness level, the motion sickness level and the emotional state of the subject. Finally, the act portion of the methodology is implemented by the same electronic processing unit 3 along with the interface 11, as, after the computation, the result of said computation is provided to the user and triggers one or more emergency interventions.

[0050] As better shown in Figure 1, the sensory system 2 comprises at least a wearable sensor 5 (e.g., a smartwatch), a contactless sensor 6 (e.g. RF sensors such as a radar or camera-based sensors, here e.g. a radar or imaging PPG) and a vision sensor 7 (e.g., camera). In particular, the vision sensor 7 is configured to output respective signals indicative image data representative of the subject; furthermore, the wearable and contactless sensors 5, 6 are configured to output respective signals indicative of physiological data of the subject.

[0051] According to an aspect of the present invention, the physiological data comprise at least cardiac input data, in particular heart rate variability (HRV), and respiration input data, in particular respiration rate (RR).

[0052] It is noted that, as reported in Figures 2-4, the physiological and image data, represented by the signals outputted by the sensory system 2, are sampled at e.g. 1 Hz; according to further aspects of the present invention, not disclosed in detail hereinafter, the sampling frequency may be different. The control logic 4 thus processes different physiological inputs, depending on the capabilities of the adopted sensing solution (either contact- or contact-less). In particular, the Applicant notes that the heart rate variability is a relevant marker reflecting the ANS activities. In fact, ANS plays a vital role in modulating heart rates and cardiac output in response to physiological and psychological stress; specifically, ANS comprises of two self-regulatory systems, namely the Sympathetic Nervous System (SNS) and the Parasympathetic Nervous System (PNS), whose activities are antagonistic. In further detail, while an increase in SNS activities quickens heart rates, an increase in PNS activities slows down heart rates. Therefore, the SNS is responsible for the fight-or-flight response in stressful situations and the PNS is for resting and digesting in relaxed conditions. Due to the continuous antagonistic inputs from the SNS and PNS, the cardiac rhythm cyclically fluctuates slightly around the mean heart rate. Consequently, by measuring this fluctuation, the SNS and PNS activities, and their balance can be assessed since the rates and pressure intensity signals of the baroreceptors, situated in the aorta and the internal carotid arteries, will respond to the demand of the cardiac needs as a reaction to stressors experienced by the body. These responses are usually in the form of changing blood pressure levels, heart rates, and emotions, etcetera. As bio- signals, such responses can be measured by changes in heart beats which can be precisely represented by the heart rate variability by using spectrum analysis, also referred to as frequency domain analysis and / or time domain analysis.

[0053] In further detail, a frequency domain analysis considers two frequency bands, namely the low frequency, LF, band (z.e., frequencies comprised between 0.04 and 0.15 Hz) and the high frequency, HF, band (z.e., frequencies comprised between 0.15 and 0.40 Hz) where cardiorespiratory events are included. Following, a LF / HF ratio is also used as a useful predictor of sleep, motion sickness and emotional states.

[0054] Additional features can be derived in the time domain analysis such as:

[0055] 1. average NN, namely the average time between normal heartbeats. Low values denote an elevated heart rate that could indicate excitement, physical activity and coffee assumption and higher NN values typically denote resting;

[0056] 2. SDNN, namely the standard deviation of the time between heartbeats and can be used to estimate physiological stress;

[0057] 3. RMSSD, namely the root mean square of successive differences of heartbeats and it has been used to predict the perceived mental stress;

[0058] 4. SDSD, namely the standard deviation of successive differences; and

[0059] 5. NN50, namely the number of adjacent NN intervals that differ from each other by more than 50 ms (NN50) and requires a 2 min epoch. The proportion term pNN50 is defined as NN50 divided by the total number of NNs. A high percentage indicates complexity in heart rate variability, correlated with good psychological and physiological state. Consequently, in view of the above, the Applicant notes that the heart rate value may be considered a good biomarker and different versatile algorithms can be used for predictive and descriptive modelling as well as for discriminative variable selection (e.g. partial least squares- discriminant analysis, PLS-DA, support vector machine, SVM - RBF, etcetera), based on the combination of LF, HF, LF / HF ratio, SDNN, RMSSD and / or pNN50.

[0060] According to an aspect of the present invention and referring to Figures 2-4, the control logic 4 comprises:

[0061] - a sleep prediction control logic 8 configured to receive and process the signals outputted by the sensory system 2 to determine the sleepiness level of the subject, in particular according to a four-levels scale derived from the KSS, in particular a four step KSS, without it being limiting to the present invention;

[0062] - a motion sickness control logic 9 configured to receive and process the signals outputted by the sensory system 2 to determine the motion sickness level, referred to as M levels, of the subject; and

[0063] - an emotional state control logic 10 configured to receive and process the signals outputted by the sensory system 2 to determine the emotional state level, referred to as e levels, of the subject.

[0064] In particular, the Sleep Prediction (SP) algorithm implemented by the sleep prediction control logic 8 is configured to receive as an input the physiological data from the sensory system 2 and process it to evaluate the drowsiness state of the person according to a reduced KSS scale based on four levels. The Applicant notes that the compression of original ten levels KSS scale into four levels is believed to better reflect the behavioural state of subject (e.g. KS1 - AWAKE, KS4 - SLEEPING) while focusing with more attention on the grey area describing the DROWSY region (e.g. KS2, KS3) which is the most relevant for in-cabin monitoring applications and with regards to Drowsiness Monitoring System (DMS). However, without it being limiting to the present invention, different scoring levels can be used.

[0065] In a similar manner, the Motion Sickness (MS) and the emotional state (ES) algorithms implemented by the motion sickness control logic 9 and the emotional state control logic 10 respectively both receive as an input the physiological data to output a classification of M and s levels; according to further aspects of the present invention, not disclosed in further detail hereinafter, it is possible to define different levels for each control logic 9, 10, specifically according to the specific necessities of the subject and / or the application field.

[0066] According to an aspect of the present invention and referring to Figure 5, the control logic 4 is configured to implement an attention detection algorithm, in particular to combine the physiological and image data outputted by the sensory system 2 to determine at least one of the following attention detection quantities:

[0067] - a grade of attention GA;

[0068] - an attention array AAp from physiological reactions of the subject based on the physiological data; and

[0069] - an attention array AAfe from facial expressions of the subject based on the image data.

[0070] In further detail, the control logic 4 is further configured to determine an attention behaviour AB indicative of an averaged activity level determined from the attention detection quantities.

[0071] Therefore, the control logic 4 is configured to process the signals outputted by the sensory system 2 to obtain a look-up table, which is used to analyse the emotional state of the subject and must be calibrated through experimental activity considering additional features like age, gender, weight, and height. In particular, the emotional state and, more generally, the activation level while driving can be derived through an attention detection algorithm based on physiological reaction. In particular, said algorithm relies on redundant sensors, thereby combining visual / facial information acquired through a vision sensor 7 and biometric information, which may be derived from the signals from the wearable sensor 5 and the contactless sensor 6.

[0072] In particular, as shown in Figure 5, the control logic 4 is configured to:

[0073] - receive the physiological and image data and process the image data to determine the active / passive emotions and the physiological data to determine the emotional stages of the subject (blocks 20-21);

[0074] - determine an activation level for both the image and the physiological data in order to determine the respective grade of attention GA from the physiological and the image data (blocks 22-23);

[0075] - apply a thresholding algorithm to both the physiological and image data to determine the attention arrays AAp and AAfe from the physiological and image data respectively (blocks 24-25);

[0076] - apply merging algorithms to both attention arrays AAp and AAfe to determine one or more statistical quantities, here the average value, of said attention arrays AAp and AAfe, referred to as AAp and AAfe respectively (blocks 26-27);

[0077] - apply windowing algorithms to determine a window size based on the one or more statistical quantities AAp and AAfe and according to a calibration based on attention gains, the latter being stored in the control logic 4 (blocks 28-29); and

[0078] - apply calibratable attention counter filters to determine a subject behaviour (blocks 30- 33) to smooth the results obtained from the processing carried out by the control logic 4 and obtain an averaged activity level, defined as the attention behaviour.

[0079] According to a further aspect of the present invention, not disclosed in further detail hereinafter, the abovementioned one or more statistical quantities may include further statistical quantities other than the abovementioned average value, for example mean, variance and the like, without it being limiting to the present invention.

[0080] The Applicant furthermore notes that validation tests should be designed to achieve a statistical significance; thus, the data summarizing the experimentation results are reported here according to a 2x2 confusion matrix, where each row of the matrix represents the instances in an actual class while each column represents the instances in a predicted class., reported hereinafter as Table 1

[0081] According to a further aspect of the present invention, the control logic 4 is configured to determine at least one among the following to classify sleep, emotional state and / or motion sickness events based on the signals outputted by the sensory system 2:

[0082] - a true positive TP indicative of a correct identification of a sleep, emotional state and / or motion sickness event as positive or being present;

[0083] - a true negative TN indicative of an incorrect identification of a sleep, emotional state and / or motion sickness event as negative or being not present;

[0084] - a false negative FN indicative of an incorrect identification of a sleep, emotional state and / or motion sickness event as positive or being present; and

[0085] - a false positive FP indicative of a correct identification of a sleep, emotional state and / or motion sickness event as negative or being not present.

[0086] Consequently, to summarize the experimentation results, three synthetic quantities may be described as follows:

[0087] TP

[0088] - sensitivity

[0089] Jor true R positive rate is defined a TPR = - and it represents the ratio of TP+FNRpositive predicted conditions (TP) over the actual number of positive conditions (TP+FN); and - sp

[0090] 1ecificity J or true neg Bative rate is defined as T NR — - and it represents the ratio FP+TN1of negative predicted conditions (TN) over the actual number of negative conditions (FP+RN); and

[0091] TP+TN

[0092] - accuracy

[0093] Jwhich is defined as ACC = - and it represents the ratio between TP+TN+FP+FN1the correctly predicted conditions and the actual conditions.

[0094] According to another aspect of the present invention, the control logic 4 is configured to:

[0095] - acquire data from the sensory system 2 in a most critical condition and in safe conditions and process them to determine the occurrence of a sleep, emotional state and / or motion sickness event; and

[0096] - if a sleep, emotional state and / or motion sickness event is determined to be present, generate a notification for the subject to warn of the occurrence of a sleep, emotional state and / or motion sickness event.

[0097] In particular, the validation of an in-cabin monitoring system here follows a multi-step approach, described in the following with reference to Figure 6.

[0098] Firstly, a validation test is performed on a Dynamic Driving Simulator (DVS) where subjects can even reach the most critical situation (e.g. falling asleep, strong nausea and motion sickness) in the safest conditions. A Maintenance of Wakefulness Test (MWT) is believed to be a useful clinical test for the evaluation of excessive sleepiness by the Task Force of the American Academy of Sleep Medicine (AASM); in this regard, said MWT requires the patient to fight against sleepiness in a soporific condition and it is considered as a validated, objective measure of the ability to stay awake.

[0099] The standard execution of the test includes at least four forty-minute steps during which the subjects to be examined are subjected to a polysomnographic recording (at least electroencephalogram, electromyogram and electrooculogram recordings) which is able to recognize the main behavioural phases: waking, non-REM sleep 1,2,3 and REM sleep. The polysomnography is thus referred to as the “ground truth” against which the in-cabin monitoring system must demonstrate its performance.

[0100] Within this test, the subject is equipped with a wearable sensing device able to acquire the reduced set of physiological parameters needed to run, for instance, a first sleep prediction algorithm implemented by the sleep prediction control logic 8. In addition to the wearable, a contactless device acquires physiological parameters needed to run, for instance, a second sleep prediction algorithm, also implemented by the sleep prediction control logic 8. The capability of such algorithms about predicting with accuracy the sleep onset of a subject is validated against the abovementioned “ground truth” during the test on the DVS. To summarize, the time instant(s) in which when Sleep Prediction algorithm rises the “flag” about a “falling asleep” event, in the three to eight minutes range, is compared with the scoring performed by a sleep expert medical doctor based on the polysomnography. In this manner, the warning raised by the wearable / contactless sensor can be trusted also for additional measures when the polysomnography cannot be used, as in the case of driving a real car due to its invasiveness.

[0101] Following the abovementioned steps, the validation test is then carried out on a real product vehicle under strict safety conditions. Taking as an example a drowsiness monitoring system, this means that at least an event classified as “true positive” must be identified which is related to the loss of cognitive capability of the driver. This means that the test, in order for it to be meaningful, should reach a critical point when the subject is no longer able to drive. The validation test is performed in a proving ground where safety areas are available along the track.

[0102] Furthermore, in this validation test, the driver is equipped with a wearable / contactless sensing device able to acquire the reduced set of physiological parameters needed to run, for instance, the first and second sleep prediction algorithms. The adoption of redundant sensors is to provide the adequate level of safety in the case that one sensor is faulty during the test. Moreover, hardware and software diversity is compliant with functional safety requirements.

[0103] The combination of the information coming from the two sensors is managed by a truthtable to consider the approaching and exceeding, for instance, of the threshold of the sleep prediction algorithm. In case of different evaluations, the wearable has the priority due to its better behaviour against motion artefacts than contactless sensors. Such flags are then used to automatically trigger the preparation and take-over of an emergency intervention on the vehicle driven by the subject.

[0104] Furthermore, the control logic 4 is further configured to provide an emergency intervention upon determining the occurrence of a sleep, emotional state and / or motion sickness event; in more detail, the emergency intervention comprises at least one between:

[0105] - in case of an automated vehicle and if available, raise the automation level of the vehicle and take the full control of the vehicle to perform a safety manoeuvre by decreasing speed and parking the car in the nearest rest area; and / or

[0106] - control the vehicle entirely via a teleoperator with remote driving, including steering, braking, and acceleration.

[0107] The Applicant furthermore notes that, thanks to the multi-level scale adopted, for instance in the sleep prediction algorithm, is possible to modulate the automation of the emergency intervention. The same approach is used in the case of motion sickness.

[0108] This means that the system allows to determine that the subject is approaching a critical event (e.g. falling asleep) and following to prepare what is necessary for the emergency manoeuvre. As soon as the last threshold is exceeded, the safety manoeuvre must be performed immediately.

[0109] The present invention has several advantages, as also clear from the disclosure above.

[0110] In particular, the present system provides an innovative method to analyse and predict motion sickness based on a simplified model of the autonomic response by monitoring various physiological parameters. Such a simplified model of the autonomous nervous system can be effectively used for advanced path tracking control of the vehicle aimed at minimising the motion sickness of the subject.

[0111] More importantly, the present system provides an architecture to safely validate the system on the vehicle. Such a solution can be successfully applied in all the critical circumstances when the driver is no longer able to control the vehicle due to motion sickness, drowsiness and impairment (e.g. heart attack).

[0112] The key concept is to provide a sense-compute-act methodology through a prediction of the consciousness level of the driver based vital signs, i.e. implementing a sense-compute-act methodology upon a prediction warning; in fact, thanks to the activation of a flag, two modes can be activated depending on the type of vehicle (e.g. through ADAS systems) and the possibility of operating remotely (i.e. tele-operation).

[0113] The proposed system thus provides a meaningful way to test new homologated vehicles equipped with a Drowsiness Monitoring System in a fully objective way against the current subjective methods. Such a method provides an innovative mechanism to run homologation tests in a secure manner by preparing and taking-over of an emergency intervention on the vehicle driven by the subject either through ADAS (when available) or remote tele-operation.

Claims

CLAIMS1. Electronic system (1) designed to analyse and validate of emergency interventions of a vehicle according to a behavioural state of a subject driving the vehicle; the electronic system (1) comprising:- a sensory system (2) configured to output signals indicative of physiological and image data related to the subject; and- an electronic processing unit (3) in communication with the sensory system and being configured to receive the signals and process them to output corresponding statistical data related to the subject, characterised in that the electronic system (1) is designed to process the data outputted by electronic processing unit (3) to determine a behavioural state and / or transitions thereof to determine the sleepiness level, the motion sickness level and the emotional state of the subject and generate a feedback to provide to an end user and provide emergency intervention to the subject based on the sleepiness level, the motion sickness level and the emotional state of the subject.

2. The electronic system (1) according to claim 1, wherein the physiological data comprise at least cardiac input data, in particular heart rate variability, and respiration input data, in particular respiration rate.

3. The electronic system (1) according to any one of the preceding claims and further comprising a control logic (4) in communication with electronic processing unit (3) and configured to receive the data outputted by electronic processing unit (3) and process them to determine a behavioural state and / or transitions thereof to determine the sleepiness level, the motion sickness level and the emotional state of the subject and generate a feedback to provide to an end user and provide emergency intervention to the subject based on the sleepiness level, the motion sickness level and the emotional state of the subject.

4. The electronic system (1) according to any one of the preceding claims, wherein the sensory system (2) comprises at least a wearable sensor (5), a contactless sensor (6) and a vision sensor (7), and wherein the vision sensor (7) is configured to output respective signals indicative image data representative of the subject and the wearable and contactless sensors (5, 6) are configured to output respective signals indicative of physiological data of the subject.

5. The electronic system (1) according to any one of claims 3 or 4, wherein the control logic (4) comprise:- a sleep prediction control logic (8) configured to receive and process the signals outputted by the sensory system (2) to determine the sleepiness level of the subject;- a motion sickness control logic (9) configured to receive and process the signals outputted by the sensory system (2) to determine the motion sickness level of the subject; and- an emotional state control logic (10) configured to receive and process the signals outputted by the sensory system (2) to determine the emotional state level of the subject.

6. The electronic system (1) according to claim 4, wherein the control logic (4) is configured to combine the physiological and image data outputted by the sensory system (2) to determine at least one of the following attention detection quantities:- a grade of attention (GA);- an attention array (AAp) from physiological reactions of the subject based on the physiological data; and- an attention array (AAfe) from facial expressions of the subject based on the image data, wherein the control logic (4) is further configured to determine an attention behaviour indicative of an averaged activity level determined from the attention detection quantities.

7. The electronic system (1) according to any one of the preceding claims, wherein the control logic (4) is configured to determine at least one among the following to classify sleep, emotional state and / or motion sickness events based on the signals outputted by the sensory system (2):- a true positive (TP) indicative of a correct identification of a sleep, emotional state and / or motion sickness event as positive or being present;- a true negative (TN) indicative of an incorrect identification of a sleep, emotional state and / or motion sickness event as negative or being not present;- a false negative (FN) indicative of an incorrect identification of a sleep, emotional state and / or motion sickness event as positive or being present; and- a false positive (FP) indicative of a correct identification of a sleep, emotional state and / or motion sickness event as negative or being not present.

8. The electronic system (1) according to any one of claims 5-7, wherein the control logic (4) is configured to:- acquire data from the sensory system (2) in a most critical condition and in safeconditions and process them to determine the occurrence of a sleep, emotional state and / or motion sickness event; and- if a sleep, emotional state and / or motion sickness event is determined to be present, generate a notification for the subject to warn of the occurrence of a sleep, emotional state and / or motion sickness event.

9. The electronic system (1) according to claim 8, wherein the control logic (4) is further configured to provide an emergency intervention upon determining the occurrence of a sleep, emotional state and / or motion sickness event, wherein the emergency intervention comprises at least one between:- in case of an automated vehicle and if available, raise the automation level of the vehicle and take the full control of the vehicle to perform a safety manoeuvre by decreasing speed and parking the car in the nearest rest area; and / or- control the vehicle entirely via a teleoperator with remote driving, including steering, braking, and acceleration.

10. Computer program product loadable in and executable on an electronic system (1) according to any one of the preceding claims, the computer program product comprising instructions which, when executed by the electronic system (1), cause the electronic system (1) to operate according to any one of the preceding claims.

Images