Monitoring the status of a driver in the cabin of a vehicle

Abstract

System (1) for monitoring the status of a driver in the cabin of a vehicle (2), the system (1) comprising: a sensory system (3) configured to acquire one or more quantities related to the status of the driver in the cabin of the vehicle (2); and electronic computing resources (4) in communication with the sensory system (3) and a vehicle network (5) of the vehicle (2). The electronic computing resources (4) are configured to: receive the one or more quantities related to the status of the driver in the cabin of the vehicle (2) from the sensory system (3) and one or more quantities related to the dynamics of the vehicle (2) from the vehicle network (5); process the received one or more quantities related to the status of the driver in the cabin of the vehicle (2) and the one or more quantities related to the dynamics of the vehicle (2) to determine the occurrence of an altered condition in the status of the driver; and provide assistance to the driver upon determining the occurrence of an altered condition in the status of the driver.

Description

[0001] MONITORING THE STATUS OF A DRIVER IN THE CABIN OF A VEHICLE

[0002] Cross-Reference To Related Applications

[0003] This patent application claims priority from European patent application no. 24158838.3 filed on February 21, 2024, and from Italian patent application no. 102025000003213 filed on February 19, 2025, the entire disclosure of which is incorporated herein by reference.

[0004] Technical Field of the Invention

[0005] The present invention generally relates to monitoring the status of a driver in the cabin of a vehicle, in particular to a system for monitoring the status of a driver in the cabin of a vehicle and a related computer program product or software.

[0006] Background of the Invention

[0007] As is known, personal mobility is a distinctive trait of modernity; in fact, in many countries, people spend a considerable amount of time in cars: the average daily time is about one hour in the United States and 1.1 hours in Germany, respectively. Furthermore, a private vehicle is one of the best-equipped environments in the daily life, namely providing a great opportunity to convert the vehicle into a health monitoring facility and to use the time people spent in their cars for health monitoring. Furthermore, the increasingly widespread use of wearable technology allows to connect the person's rest phase, in particular the quality of sleep, with the new driving mission.

[0008] Thus, as also stated above, in-vehicle health monitoring is a relevant both in terms of research and industrial development, in particular including environmental, physiological, and behavioural monitoring, in detail:

[0009] - environmental parameters include temperature, air quality, humidity, weather and light conditions and speed. Environmental parameters are generally acquired by default for in-car well-being and driver’s assistance systems;

[0010] - physiological parameters typically include vital signs, such as heart rate (HR), heart rate variability (HRV), respiration rate (RR), respiration rate variability (RRV), the measurement of volumetric changes in the blood (PPG), oxygen saturation (SpCh), body surface temperature and skin impedance; and

[0011] - behavioral parameters quantify physical activities during the drive to reflect the driver’s attention level, tiredness and well-being.

[0012] With respect to physiological parameters, a wide range of technologies is known to be used for the scope, e.g.:

[0013] - ballistocardiography (BCG), radar, and thermography for HR, RR, and cardiopulmonary analysis;

[0014] - electrocardiography (ECG), capacitive ECG (cECG), radar, BCG and seismocardiography (SCG), video imaging, photoplethysmography (PPG) and PPG imaging (PPGI), magnetic induction (MI), and thermography capture body surface potentials, displacements and temperatures, the superficial perfusion, and the intrathoracic impedance;

[0015] - camera sensors allow not only to directly observe the driver’s activities, as well as deliver vital signs; and

[0016] - wearable devices for measurement of volumetric changes in the blood (PPG), oxygen saturation in the blood (SpO2), heart rate (HR) and respriration rate RR, as well as, in some modern devices, electrocardiography (ECG).

[0017] Ob ject and Summary of the Invention

[0018] The Applicant notes that driving is a complex task, requiring full concentration and a vital balance between alertness and a calm attitude. In past few decades, a sharp increase in road accidents and loss of life was experienced mainly due to the increase in driver drowsiness and fatigue level. The definition, analysis, and implementation of in-vehicle monitoring systems that collect data which are informative of the status of the joint drivervehicle system represent a topic of strong interest from both academic players and industrial manufacturers. Moreover, unobtrusive in-vehicle health monitoring has the potential to use the driving time to perform regular medical check-ups.

[0019] An object of the present invention is thus providing systems and methods that allow to overcome at least in part the disadvantages of the known prior art.

[0020] According to the present invention, a system for monitoring the status of a driver in the cabin of a vehicle and a related computer program product or software are provided, as claimed in the appended set of claims.

[0021] Brief Description of the Drawings

[0022] Figure 1 schematically shows a system for monitoring the status of a driver in the cabin of a vehicle according to the present invention.

[0023] Description of Preferred Embodiments of the Invention

[0024] The present invention will now be described in detail with reference to the accompanying drawings in order to allow a skilled person to implement it and use it. Various modifications to the described embodiments will be readily apparent to those of skill in the art and the general principles described may be applied to other embodiments and applications without however departing from the protective scope of the present invention as defined in the appended claims. Therefore, the present invention should not be regarded as limited to the embodiments described and illustrated herein but should be allowed the broadest protection scope consistent with the features described and claimed herein.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning commonly understood by one of ordinary skill in the art to which the invention belongs. In case of conflict, the present specification, including the definitions provided, will control. Furthermore, the examples are provided for illustrative purposes only and as such should not be considered limiting.

[0026] In particular, the block diagrams included in the attached figures and described below are not to be understood as a representation of the structural features, i.e. constructional limitations, but must be understood as a representation of functional features, i.e. intrinsic properties of the devices defined by the effects obtained, that is to say functional restrictions, which can be implemented in different ways, so as to protect the functionalities thereof (operational capability).

[0027] In order to facilitate the understanding of the embodiments described herein, reference will be made to some specific embodiments and a specific language will be used to describe the same. The terminology used herein is used for the purpose of describing particular embodiments only and is not intended to limit the scope of the present invention.

[0028] Figure 1 shows a system 1 for monitoring the status of a driver in the cabin of a vehicle 2, in particular a car or the like; in particular, the system 1 comprises:

[0029] - a sensory system 3 configured to acquire one or more quantities related to the status of the driver in the cabin of the vehicle 2; and

[0030] - electronic computing resources 4 in communication with the sensory system 3 and a vehicle network 5 (e.g. a CAN bus) of the vehicle 2.

[0031] In particular, according to an aspect of the present invention, the electronic computing resources 4 are configured to:

[0032] - receive the one or more quantities related to the status of the driver in the cabin of the vehicle 2 from the sensory system 3 and one or more quantities related to the dynamics of the vehicle 2 from the vehicle network 5;

[0033] - process the received one or more quantities related to the status of the driver in the cabin of the vehicle 2 and the one or more quantities related to the dynamics of the vehicle 2 to determine the occurrence of an altered condition in the status of the driver; and

[0034] - provide assistance to the driver upon determining the occurrence of an altered condition in the status of the driver.

[0035] In order to determine the occurrence of an altered condition in the status of the driver, the electronic computing resources 4 are configured to determine the occurrence of one among a status where the driver is distracted, a status where the driver is drowsy and a status where the driver is determined to be unable to operate. Furthermore, in order to provide assistance to the driver upon determining the occurrence of an altered condition in the status of the driver, the electronic computing resources 4 are configured to generate a notification if the occurrence of an altered condition in the status of the driver is determined to be present; in particular, the notification may be a sound notification, e.g. an alarm to be reproduced on the speakers of the vehicle 2, or a visual notification, e.g. a notification to be shown on a Human-Machine Interface (HMI) mounted on the vehicle 2.

[0036] In further detail, the one or more quantities related to the status of the driver in the cabin of the vehicle comprise environmental, physiological and behavioral parameters related to the status of the driver in the cabin of the vehicle 2. Furthermore, the one or more quantities related to the dynamics of the vehicle 2 comprise at least one among a vehicle speed, a position of the accelerator pedal, a position of the brake pedal, a position of the clutch pedal, a position of the steering wheel, data relative to the current gear, data relative to the activation of the lights, data relative to the activation of the turning indicators, data relative to the activation of the infotainment system, data relative to the activation of the air conditioner, data relative to the activation of the window regulators and data relative to the activations of the seat position controller. In addition to the abovementioned quantities, according to an aspect of the present invention, the electronic computing resources 4 are further configured to receive, in particular through the vehicle network 5, further quantities relating to the dynamics of the vehicle 2, in particular data relative to a predetermined trajectory of the vehicle 2 and relative to any deviation from the same trajectory; exemplarily, the electronic computing resources 4 are configured to receive the abovementioned data from one or more driving assistance systems (not shown, e.g. Lane Departure Warning, LDW, systems or Lane Keeping Assist, LKA, systems) in communication with the electronic computing resources 4 through the vehicle network 5. In particular, as also clear from the disclosure below, the abovementioned data from the one or more driving assistance systems allows, together with the abovementioned parameters, to determined whether the driver is in an altered state or not.

[0037] According to an aspect of the present invention, the sensory system 3 comprises environment sensors configured to acquire data related to the environment in the cabin of the vehicle 2; in particular, according to an aspect of the present invention, the environment sensors comprise air quality sensors 6 configured to acquire data relative to the breath of the driver in the cabin of the vehicle 2 and process them to determine the presence or absence of one or more substances, e.g. alcohol or the like, in the breath of the driver.

[0038] In further detail, the air quality sensor 6 is of the in-cabin type, meaning that it is positioned in the cabin of the vehicle 2, in particular in the vicinity of the driver (e.g. on the cabin ceiling, in particular right above the driver’s seat) so as to collect air samples, and in particular the breath of the driver to identify the presence of specific substances, such as alcohol. Furthermore, as also specified above, the air quality sensor 6 is equipped with hardware / software features that ensure automatic collection and processing of the acquired air samples and output a corresponding output signal indicative of the data relative to the presence or absence of one or more substances to the electronic computing resources 4.

[0039] The sensory system 3 further comprises an imaging sensory system configured to acquire imaging data relative to the physical conditions of the driver in the cabin of the vehicle 2; in particular, according to an aspect of the present invention, the imaging sensory system comprises:

[0040] - cameras 7, here one shown by way of example, configured to acquire frames of the cabin of the vehicle 2 and process them to detect and classify relevant scenarios relating to the occurrence of an altered condition in the status of the driver;

[0041] - radars 8, here one shown by way of example, configured to acquire data relative to physical parameters, in particular at least one among heart rate and respiration rate, of the driver, a distance between the driver and each radar 8 and to the movements of the driver in the cabin of the vehicle 2; and

[0042] - one or more wearable sensors, one shown in Figure 1, 11 configured to acquire data relative to the physiological conditions of the driver, in particular physical parameters, in particular at least one among heart rate (HR), respiration rate (RR), a value relating to a measurement of volumetric changes in the blood (PPG), a value of an oxygen saturation SpCh. and the movement status of the driver .

[0043] In particular, the camera 7 is of the in-cabin type, meaning that it is positioned in the cabin of the vehicle 2 so that the field of view of the camera 7 is such that at least a portion of the body of the driver is observed, namely torso, arms and head (e.g. on the dashboard of the vehicle 2 facing the driver. In further detail, the camera 7 is configured to acquire a video feed of the driver of the vehicle 2, to process the frames forming the video feed and identify relevant scenarios, such as distraction and drowsiness. In further detail the camera 7 is equipped with a lens / sensor combination suitable for collecting enough information to recognize, e.g., the position of the hands of the driver and / or gestures, objects held in hands, arms positions and gestures, head roll, head pitch, head yaw, mouth, nose, eyes (including eye movements and eyelid movements). Furthermore, the camera 7 is not only a frame capture device, but also comprises, in particular embeds, intelligence in the form of suitable hardware / software features to autonomously collect, process, and classify the scenarios and transmit the relative data to the electronic computing resources 4.

[0044] Referring to the radar 8, the latter comprises an inertial measurement unit (not shown) configured to detect the movements of the driver in the cabin of the vehicle 2. Furthermore, according to an aspect of the present invention, the radar 8 is of the in-cabin type, meaning that it is positioned so that its field of view is such that at least a portion of the body of the driver is observed, namely torso, including the diaphragm (e.g. on the cabin windshield, facing the driver). In further detail, the radar 8 is provided with hardware / software features in particular configured to generate a radar micro doppler map of the driver’s torso and to extract the relative physical parameters, namely the heart rate and the respiration rate, as well as the distance of the radar 8 from the driver and provide said parameters to the electronic computing resources 4. Furthermore, the radar 8 is equipped with an inertial measurement unit (in particular comprising a three-axial accelerometer and a gyroscope) configured to detect movements of the radar 8 and to output relative data to send to the electronic computing resources 4.

[0045] Furthermore, the one or more wearable sensors 11 each comprise a physiological sensor, here a PPG sensor, configured to acquire data relative to the physical status of the driver, in particular an electrocardiogram, the heart rate, the respiration rate, the oxygen saturation and the like, as well as the movements of the same driver in the cabin of the vehicle 2.

[0046] The sensory system 3 further comprises physiological sensors configured to acquire data relative to the physiological conditions of the driver in the cabin of the vehicle 2; in particular, according to an aspect of the present invention, the physiological sensors comprise steering wheel electrocardiogram (ECG) sensors 9 configured to acquire data relative to the electrocardiogram of the driver when the latter holds, in particular with both hands, the steering wheel of the vehicle 2.

[0047] The sensory system 3 furthermore comprises motion sensors configured to acquire data relative to the movements of the driver in the cabin of the vehicle 2; in particular, according to an aspect of the present invention, the motion sensors comprise seatbelt inertial measurements sensors (IMU) 10, here comprising a three-axial accelerometer and a gyroscope, configured to acquire data relative to the movements of the driver in the cabin of the vehicle 2.

[0048] Considering the disclosure above, in order to determine the occurrence of an altered state in the status of the driver in the cabin of the vehicle, the electronic computing resources 4 are configured to:

[0049] - determine the presence or absence of substances in the breath of the driver in the cabin of the vehicle 2 based on the data provided by the environment sensors;

[0050] - determine the occurrence of movements of any monitored part of the body of the driver based on the data provided by the imaging sensory system, in particular the cameras 7, the data provided by the motion sensors and / or the data provided by the one or more wearable sensors 11; and

[0051] - determine the physiological conditions of the driver based on the data provided by the imaging sensory system, in particular the radars 8, the data provided by the physiological sensors and / or the data provided by the one or more wearable sensors 11.

[0052] Thus, to summarize, the electronic computing resources 4 are configured to:

[0053] - receive the data sent by the sensory system 3, in particular from the air quality sensor 6, the camera 7, the radar 8, the steering wheel ECG sensors 9, the seatbelt inertial measurements sensors 10 and the one or more wearable sensors 11;

[0054] - receive the data sent by the vehicle network 5, in particular the one or more quantities related to the dynamics of the vehicle 2; and

[0055] - process the incoming data stream to provide the following feedbacks: a) in case of driver distraction, e.g. gaze not focused on the road in case of forward drive or use of a smartphone or other devices during driving, the electronic computing resources 4 are configured to generate an alarm upon identification of said condition; b) in case the physiological state of driver indicates an incoming transition from an awake state to a sleep state, the electronic processing resources 4 are configured to generate an alarm upon identification of said condition. For this purpose, the drowsiness detected by the camera 7 is fused with the analysis of the variation over time of the statistical parameters of respiration rate as measured by the radar 8 and the heart rate variability as measured by the steering wheel ECG sensors 9 and / or the wearable sensors 11. The raw respiration rate provided by the radar 8 and heart rate variability as provided by the steering wheel ECG sensors 9 and / or the wearable sensors 11 are then filtered to remove motion artifact by considering variation of the distance from radar to subject, motion as detected by the seatbelt inertial measurements sensors 10 and / or the wearable device 11 and commands / operations performed by the driver as detected through the vehicle network 5 and / or the wearable sensors 11; and c) in case substances like alcohol is detected in the driver’s breath and said substance is present at a quantity that is above a given threshold, the electronic computing resources 4 are configured to generate an alarm indicating the driver is incapacitated to operate. Moreover, the driver respiration rate and heart beat are monitored by means of radar 8, steering wheel ECG sensors 9 and / or the wearable sensors 11: in case of sudden drop of one or both these parameters, the electronic computing resources 4 are configured to generate an alarm indicating that the driver may suffer from a physical condition. Variations of the heart sinus rhythm as measured by the steering wheel ECG sensors 9 and / or the wearable sensors 11 are processed by the electronic computing resources 4 to identify critical conditions such as fibrillation and heart stroke. In case critical conditions are identified, the electronic computing resources 4 are configured to generate a corresponding alarm. It is noted that alarms are generated both locally in the form of audio and video messages displayed either through the vehicle HMI (not shown) or through a dedicated HMI, also located in the vehicle 2, and remotely by sending all the generated alarm through to a remote cloud server (not shown), e.g. through wireless connection such as connection based on 3G connectivity, that can be operated by fleet manager or emergency response center.

[0056] From the disclosure above, the present invention has several advantages.

[0057] In particular, as also discussed in detail above, the present invention provides for a system for monitoring the status of a driver in the cabin of a vehicle, in particular by providing a sensor hub for comprehensive analysis of the driver’ s physiology, thus aiming at providing for: - an accurate analysis of the health status of the driver and possibly preventing road accidents;

[0058] - a way to monitor, over long period of time, the health status of the driver and provide early information about possible incipient problems; and - a way to assess the health status of the driver and the latter’s capability to perform the driving mission safely (e.g. FIT TO DRIVE) prior to the start the mission itself. For example, according to an aspect of the present invention, the body battery integrated in the one or more wearable sensors 11 is configured to provide data relative to the sleep quality of the driver, thereby providing further information relating to the general status of the same and the occurrence of any sign of an altered state.

[0059] Finally, it is clear that modifications and variations may be made to the object of the present patent application described and illustrated herein without departing from the protective scope of the present invention as defined in the appended claims.

Claims

CLAIMS1. System (1) for monitoring the status of a driver in the cabin of a vehicle (2), the system (1) comprising:- a sensory system (3) configured to acquire one or more quantities related to the status of the driver in the cabin of the vehicle (2); and- electronic computing resources (4) in communication with the sensory system (3) and a vehicle network (5) of the vehicle (2), the electronic computing resources (4) being configured to:- receive the one or more quantities related to the status of the driver in the cabin of the vehicle (2) from the sensory system (3) and one or more quantities related to the dynamics of the vehicle (2) from the vehicle network (5);- process the received one or more quantities related to the status of the driver in the cabin of the vehicle (2) and the one or more quantities related to the dynamics of the vehicle (2) to determine the occurrence of an altered condition in the status of the driver; and- provide assistance to the driver upon determining the occurrence of an altered condition in the status of the driver.

2. The system (1) according to claim 1, wherein, in order to determine the occurrence of an altered condition in the status of the driver, the electronic computing resources (4) are configured to determine the occurrence of one among a status where the driver is distracted, a status where the driver is drowsy and a status where the driver is determined to be unable to operate.

3. The system (1) according to claim 1 or 2, wherein, in order to provide assistance to the driver upon determining the occurrence of an altered condition in the status of the driver, the electronic computing resources (4) are configured to generate a notification if the occurrence of an altered condition in the status of the driver is determined to be present.

4. The system (1) according to any one of the previous claims, wherein the one or more quantities related to the status of the driver in the cabin of the vehicle (2) comprise environmental, physiological and behavioral parameters related to the status of the driver in the cabin of the vehicle (2).

5. The system (1) according to any one of the previous claims, wherein the sensory system (3) comprises:- environment sensors (6) configured to acquire data related to the environment in the cabin of the vehicle (2);- an imaging sensory system (7, 8) configured to acquire imaging data relative to the physical conditions of the driver in the cabin of the vehicle (2);- physiological sensors (9) configured to acquire data relative to the physiological conditions of the driver in the cabin of the vehicle (2);- motion sensors (10) configured to acquire data relative to the movements of the driver in the cabin of the vehicle (2); and- one or more wearable sensors (11) configured to acquire data relative to the physiological conditions of the driver and the movement status of the same driver.

6. The system (1) according to claim 5, wherein:- the environment sensors (6) comprise air quality sensors (6) configured to acquire data relative to the breath of the driver in the cabin of the vehicle (2) and process them to determine the presence or absence of one or more substances in the breath of the driver;- the imaging sensory system (7, 8) comprises cameras (7) configured to acquire frames of the cabin of the vehicle (2) and process them to detect and classify relevant scenarios relating to the occurrence of an altered condition in the status of the driver and radars (8) configured to acquire data relative to physical parameters of the driver and a distance between the driver and each radar (8) and to the movements of the driver in the cabin of the vehicle (2);- the physiological sensors (9) comprise steering wheel electrocardiogram sensors (9) configured to acquire data relative to the electrocardiogram of the driver when the latter holds the steering wheel of the vehicle (2);- the motion sensors (10) comprise seatbelt inertial measurements sensors (10) configured to acquire data relative to the movements of the driver in the cabin of the vehicle (2); and- the one or more wearable sensors (11) comprise each a physiological sensor configured to acquire data relative to the physical status of the driver and the movements of the driver in the cabin of the vehicle (2), and wherein, in order to determine the occurrence of an altered state in the status of the driver in the cabin of the vehicle (2), the electronic computing resources (4) are configured to:- determine the presence or absence of substances in the breath of the driver in the cabin of the vehicle based on the data provided by the environment sensors (6);- determine the occurrence of movements of any monitored part of the body of the driver based on the data provided by the imaging sensory system (7, 8), in particular the cameras (7), the data provided by the motion sensors (10) and / or the data provided by the one or more wearable sensors (11); and- determine the physiological conditions of the driver based on the data provided by the imaging sensory system (7, 8), in particular the radars (8), the data provided by the physiological sensors (9) and / or the data provided by the one or more wearable sensors (11).

7. The system (10) according to claim 6, wherein each radar (8) comprises an inertial measurement unit configured to detect the movements of the driver in the cabin of the vehicle (2).

8. The system (1) according to any one of the previous claims, wherein the one or more quantities related to the dynamics of the vehicle (2) comprise at least one among a vehicle speed, a position of the accelerator pedal, a position of the brake pedal, a position of the clutch pedal, a position of the steering wheel, data relative to the current gear, data relative to the activation of the lights, data relative to the activation of the turning indicators, data relative to the activation of the infotainment system, data relative to the activation of the air conditioner, data relative to the activation of the window regulatorsand data relative to the activations of the seat position controller, as well as data relative to a predetermined trajectory of the vehicle (2) and relative to any deviation from the same trajectory.

9. A computer program product comprising instructions which, when the program is executed by electronic computing resources of a system (1) for monitoring the status of a driver in the cabin of a vehicle (2) according to any one of claims 1-8, cause the system to operate according to any one of claims 1-8.

Images