Analysis and prediction of microsleep events in fatigue monitoring applications

Abstract

A computer program product (6) comprising instructions which, when executed by electronic processing resources (10) of a system (1) for analyzing and predicting one or more behavioral states and / or transitions thereof among Awake (W), Microsleep (M), Sleep (S) and Dozing (D) of a subject, cause the electronic processing resources (10) to: receive a biometric signal of a subject; process the received biometric signal to classify it into one of different classes associated with the one or more behavioural states and / or transitions among Awake (W), Microsleep (M), Sleep (S) and Dozing (D) phases of a subject; and detect and / or predict a behavioural state and / or a transition among Awake (W), Microsleep (M), Sleep (S) and Dozing (D) phases of the subject based on the classified biometric signal. The instructions are further configured to cause the electronic processing resources (10) to: compute at least one statistical quantity (σHRV, µsHRV, madACT) for and based on the received biometric signal; verify that the at least one statistical quantity (σHRV, µsHRV, madACT) satisfies one or more proprietary criteria; and classify the biometric signal into one of different classes associated with the one or more behavioural states and / or transitions among among Awake (W), Microsleep (M), Sleep (S) and Dozing (D) phases based on the result of the verification.

Description

[0001]"ANALYSIS AND PREDICTION OF MICROSLEEP EVENTS IN FATIGUE MONITORING APPLICATIONS" Cross-Reference To Related Applications This patent application claims priority from European patent application no. 24158830.0 filed on February 21, 2024 and from Italian patent application no. 102025000003096 filed on February 18, 2025, the entire disclosure of which is incorporated herein by reference. Technical Field of the Invention The present invention generally relates to fatigue monitoring, in particular the analysis and prediction of microsleep events in fatigue monitoring applications. In further detail, the present invention relates to a system for analyzing and predicting one or more behavioural states, particularly microsleep events, in particular deriving from fatigue, and / or transitions thereof of a subject, in particular a driver of a vehicle, and a related computer program product or software. Background of the Invention As is known, fatigued driving is one of the main contributors to road traffic accidents. Poor sleep quality and lack of sleep negatively affect driving performance and extreme states of fatigue can cause microsleep; in particular, microsleeps are short bursts of sleep and can last for a period of a few seconds to minutes, all without the driver even knowing they have fallen asleep. In further detail, the analysis of microsleep, in particular in fatigue monitoring, is very complex, since it is normally a combination of very short episodes in the much longer transition phase between wakefulness and sleep status. To analyse the fatigue of the driver different measures have been implemented: - image based measures: some drowsiness signs are detectable and recordable by artificial vision systems, in particular cameras or visual sensors; in particular, the drowsiness signals are indicative of the driver’s facial expressions and movements, especially the head and eye movements. Generally, such image-based systems are non- intrusive, non-invasive and cost-effective, as said systems require only an artificial vision system to collect the needed data. However, the system’s performance is severely affected in cases where it is difficult to track facial and eye data due to obstacles. In addition, the abovementioned systems provide details on the subject's state of sleepiness at a very advanced stage, when the subject’s cognitive state is no longer able to carry out its activity. Above all, said systems cannot cover the large population of people who fall asleep with their eyes open, especially people with OSAS (Obstructive Sleep Apnea Syndrome); - biological-based measures: many biological signals have been used to detect the driver’s drowsiness, such as brain activity, heart rate, breathing rate, pulse rate and body temperature signals. These biological signals, also known as physiological measures, are proven to be more accurate and reliable for detecting drowsiness; in particular, the accuracy of said systems configured to acquire said biological-based measures is due to their ability to capture early biological changes that may appear in case of drowsiness, thus alerting the driver before any physical drowsiness signs appear. Several activities are aimed at developing cost effective and the least intrusive, possibly contactless, sensors able to provide accurate measurement of the required biometric parameters; - vehicle-based measures: these measures depend on tracing and analysing driving patterns. Every driver forms a unique driving pattern and, as such, the driving patterns of a drowsy driver can be distinguished from those of an alert driver. Since this is an indirect way of detecting drowsiness, such measure may result in being neither accurate nor fast enough to regain the consciousness level of the driver; and - hybrid-based measures: a hybrid drowsiness detection system employs a combination of image-, biological and vehicle-based measures to extract drowsiness features, with the aim of producing a more robust, accurate, and reliable drowsiness detection system. Object and Summary of the Invention The Applicant notes that, currently, driver monitoring systems analyse gaze, eyes blinking, through camera, and vehicle data (e.g., steering wheel movements, lane holding, acceleration) to detect states of fatigue. However, said systems are neither accurate nor fast enough to detect short microsleeps and, most of all, cannot provide an adequate analysis when the driver loses the consciousness level, as in the case of microsleep events, with eyes open. An object of the present invention is thus providing systems and methods that allow to overcome at least in part the disadvantages of the known prior art. According to the present invention, a system for analyzing and predicting one or more behavioural states, particularly microsleep events, and / or transitions thereof of a subject and a related computer program product or software are provided, as claimed in the appended set of claims. Brief Description of the Drawings Figure 1 schematically shows a block diagram of a system for analyzing and predicting one or more behavioral states of a subject according to the present invention. Figure 2 shows a variation over time of the mean value of a heart rate variability determined by a system for analyzing and predicting one or more behavioral states and / or transitions thereof of a subject according to the present invention. Figure 3 shows a variation over time of the standard deviation of a heart rate variability determined by a system for analyzing and predicting one or more behavioral states and / or transitions thereof of a subject according to the present invention. Description of Preferred Embodiments of the Invention The present invention will now be described in detail with reference to the accompanying drawings in order to allow a skilled person to implement it and use it. Various modifications to the described embodiments will be readily apparent to those of skill in the art and the general principles described may be applied to other embodiments and applications without however departing from the protective scope of the present invention as defined in the appended claims. Therefore, the present invention should not be regarded as limited to the embodiments described and illustrated herein but should be allowed the broadest protection scope consistent with the features described and claimed herein. Unless otherwise defined, all technical and scientific terms used herein have the same meaning commonly understood by one of ordinary skill in the art to which the invention belongs. In case of conflict, the present specification, including the definitions provided, will control. Furthermore, the examples are provided for illustrative purposes only and as such should not be considered limiting. In particular, the block diagrams included in the attached figures and described below are not to be understood as a representation of the structural features, i.e. constructional limitations, but must be understood as a representation of functional features, i.e. intrinsic properties of the devices defined by the effects obtained, that is to say functional restrictions, which can be implemented in different ways, so as to protect the functionalities thereof (operational capability). In order to facilitate the understanding of the embodiments described herein, reference will be made to some specific embodiments, and a specific language will be used to describe the same. The terminology used herein is used for the purpose of describing particular embodiments only and is not intended to limit the scope of the present invention. In particular, as also disclosed in further detail in the following, the present invention relates to a system implementing an automatic method for the analysis and prediction of one or more behavioural states and / or transitions thereof, in particular of micro sleep events, through the processing of physiological measurements of a subject, in particular a driver of a vehicle. Figure 1 schematically shows a system 1 for analyzing and predicting one or more behavioral states and / or transitions thereof among Awake (W), Microsleep (M), Sleep (S) and Dozing (D) of a subject (e.g. a driver of a vehicle) comprising: - a sensing unit 2 configured to communicate with a sensory system, here comprising either a wearable sensor 3 (e.g. a smartwatch) and / or a contactless sensor 4 (e.g. a RADAR, a camera), to receive biometric signals of a subject; - electronic computing resources 10, here comprising a processing unit 5, configured to communicate with the sensing unit 2 to receive the biometric signals and to store, load and execute, when in use, a computer program product or software or algorithm 6 therein to output data relative to the behavioural state and / or transitions between behavioural states of the subject on the basis of the biometric signals; and - a feedback unit, here comprising either a local feedback unit 7 and / or a remote feedback unit 8, configured to communicate with the electronic computing resources 10 and to receive the output generated by the latter to provide feedback to external resources, e.g. further electronic devices external to the system 1. In particular, the software 6 comprises instructions which, when executed by the electronic processing resources 10, cause the electronic processing resources 10 to: - receive a biometric signal of a subject, in particular from the sensory system through the sensing unit 2; - process the received biometric signal to classify it into one of different classes associated with the one or more behavioural states and / or transitions among Awake (W), Microsleep (M), Sleep (S) and Dozing (D) phases of a subject; and - detect and / or predict a behavioural state and / or a transition among Awake (W), Microsleep (M), Sleep (S) and Dozing (D) phases of the subject based on the classified biometric signal. In further detail, the software is configured to cause the electronic processing resources 10 to: - compute at least one statistical quantity for and based on the received biometric signal; - verify that the at least one statistical quantity satisfies one or more proprietary criterion; and - classify the biometric signal into one of different classes associated with the one or more behavioural states and / or transitions among among Awake (W), Microsleep (M), Sleep (S) and Dozing (D) phases based on the result of the verification. According to an aspect of the present invention, the biometric signal comprises alternatively a subset of a physiological data or a variable derived from said physiological data. In particular, the physiological data and / or variable derived from said physiological data comprise values indicative of at least one cardiac output, in particular heart rate HR and heart rate variability HRV, and the subject activity ACT. In further detail, the heart rate HR is indicative of the measure of the regular movement of the heart measured in beats per minutes and the heart rate variability HRV is indicative of the amount of time between two consecutive heartbeats, measured for example in milliseconds. According to an aspect of the present invention, the heart rate HR and the heart rate variability HRV may be obtained through a contact-based measurements, e.g. photoplethysmography or electrocardiogram, or through contact-less measurements, e.g. imaging photoplethysmography or radar. Furthermore, the subject activity ACT quantifies the motion of the subject in a given amount of time (e.g., 30 seconds), in a range from 0% (i.e., absence of motion) to 100% (i.e., uninterrupted motion). According to an aspect of the present invention, the subject activity ACT is obtained by processing contact-based measurements, e.g. from accelerometers like the ones in the wearable sensor 3, or through contactless measurements, e.g. from imaging systems or radars like the ones in the contactless sensor 4. In order to compute at least one statistical quantity for and based on the received biometric signal, the software 6 is configured to cause the electronic processing resources 10 to compute at least a standard deviation of the heart rate variability ^^^^^^^^, a mean value of the standard deviation of the heart variability rate ^^^^^^^^^^, and the mean absolute deviation of the activity ^^^^^^^^^^^^In further detail, the electronic processing resources 10 are configured to determine a mean value of the values of the heart rate ^^^^^^^^(where i is an index comprised between 1 and NHR, the latter being the number of samples acquired for the heart rate HR), a mean value of the values of the heart rate variability ^^^^^^^^^^(where j is an index comprised between 1 and NHRV, the latter being the number of samples acquired for the heart rate variability HRV), a standard deviation of the heart rate variability ^^^^^^^^^^, where k is an index comprised between 1 and NHRVas above, an average value of the activity ^^^^^^^^where w is an index comprised between 1 and NACTthe number of computed values of the subject activity ACT. In further detail, assuming that the values of the heart rate HR and the heart rate variability have been sampled for a given amount of time, it is possible to define Q partially overlapping subsets of the values of the heart rate HR and the heart rate variability HRV, each composed of (NHR, NHRV) samples. Each subset i, j may be described with the following Equations (1)-(5): ^^^^^^^^^^^^^^^^^^ = √1 ^^∑ (^^^^^^^^ − ^^^^^^^^)2^^^^^^(3)^^=1 Furthermore, the obtained subsets can then be grouped in partially overlapping subsets each composed of M consecutive values of the standard deviation of the heart rate variability ^^^^^^^^^^, for which the following statistical parameter is computed (Equation (6))): Where n is an index comprise between 1 and M and ^^^^^^^^^^is the mean value of the standard deviation of the heart rate variability HRV. Thus, in this way, two discrete functions, namely the standard deviation of the heart rate variability ^^^^^^^^and the mean value of the heart variability rate ^^^^^^^^^^. In order to verify that the at least one statistical quantity satisfies one or more proprietary criterion, the software 6 is configured to cause the electronic processing resources 10 to: - verify if the value of the mean absolute deviation of the activity ^^^^^^^^^^^^is below a given threshold TH1 (e.g., 10%, i.e. is determined to be very low) and if the mean value of the standard deviation of the heart rate variability ^^^^^^^^^^assumes the minimum value to determine if a transition between the Awake (W) and Microsleep (M) phases occurs; - verify if the value of the mean absolute deviation of the activity ^^^^^^^^^^^^is below a given threshold, here TH1(e.g., 10%) and if the standard deviation of the heart rate variability ^^^^^^^^assumes substantially the same value in a first-time interval X to determine if a transition between the Awake (W) and Sleep (S) phases occurs; and - verify if the value of the mean absolute deviation of the activity ^^^^^^^^^^^^is below a given threshold TH2 (e.g., 20%, i.e. is determined to be low) and if the standard deviation of the heart rate variability ^^^^^^^^assumes substantially the same value in a second time interval Y to determine if a transition between the Awake (W) and Dozing (D) phases occurs, the second time interval Y being separate and greater than the first- time interval X. In order to classify the biometric signal into one of different classes associated with the one or more behavioural states and / or transitions among among Awake (W), Microsleep (M), Sleep (S) and Dozing (D) phases based on the result of the verification, the software 6 is configured to cause the electronic processing resources 10 to: - determine the occurrence of the transition between the Awake (W) and Microsleep (M) phases if the mean value of the values of the standard deviation of the heart rate variability ^^^^^^^^^^assumes the minimum value and the value of the mean absolute deviation of the activity ^^^^^^^^^^^^is below the given threshold TH1; - determine the occurrence of the transition between the Awake (W) and Sleep (S) phases if the standard deviation of the heart rate variability ^^^^^^^^assumes substantially the same value in the first time interval X and the value of the mean absolute deviation of the activity ^^^^^^^^^^^^is below the given threshold TH1; and - determine the occurrence of the transition between the Awake (W) and Dozing (D) phases if the standard deviation of the heart rate variability ^^^^^^^^assumes substantially the same value in the second time interval Y and the value of the mean absolute deviation of the activity ^^^^^^^^^^^^is below the given threshold TH2. According to an aspect of the present invention, the first-time interval X comprises approximately one up to four seconds, preferably two seconds, and the second time interval Y comprises approximately from 60 to 200 seconds, in particular as a function of the standard deviation of the heart rate variability ^^^^^^^^. It is furthermore noted that the duration of the time intervals X, Y is adjusted dynamically, based on the values of the mean values ^^^^^^^^and ^^^^^^^^^^to compensate for the different physiologies among different subjects, and to accommodate to the Circadian rhythm of a subject. Thus, by analysing the mathematical properties of the standard deviation of the heart rate variability ^^^^^^^^and the mean value of the standard deviation of the heart rate variability ^^^^^^^^^^, it is possible to determine the transition from Awake (W) phase to one of the following phases: - Microsleep (M), i.e. sleep events lasting for less than 15 seconds; - Sleep (S), i.e. sleep events lasting for more than 15 seconds; and - Dozing (D) which can be defined a physiological state that it neither microsleep, nor sleep while not being awake. In further detail, as also anticipated above, the transition from Awake (W) to Microsleep (M) phases is identified when very low activity is observed and through the local minima of the mean value of the standard deviation of the heart rate variability ^^^^^^^^^^, the latter indicating a sudden reduction and stabilization of standard deviation of the heart rate variability HRV, thereby indicating a significant and sudden increased activity of the parasympathetic autonomous nervous systems and a reduction of activity of the sympathetic nervous system. This is shown in Figure 2, where circle A indicates the minimum value of the mean value of the standard deviation of the heart rate variability ^^^^^^^^^^. Furthermore, referring to Figure 3, the transition from Awake (W) to Sleep (S) phases is identified when very low activity is observed and through plateau of the standard deviation of the heart rate variability ^^^^^^^^, where the function exhibits minimal differences for a number of consecutive seconds of the first-time interval X. In particular, it is noted that the term “minimal difference” is to be understood as the difference between two consecutive values is lower than 55.5-0.5^^^^^^^^^, where ^^^^^^^^is the average heart rate value in the subset W. The transition from Awake (W) to Dozing (D) phases is identified when reduced activity is observed and through plateau of the standard deviation of the heart rate variability ^^^^^^^^, where the function exhibits minimal differences for a number of consecutive seconds of the first time interval X over an interval of a number of consecutive seconds of the second time interval Y. From the disclosure above, the present invention has several advantages. In particular, the present invention provides for a system and a software that are able to predict short microsleeps by using a combination of biometrics signals acquired either through wearable or contactless sensors. Finally, it is clear that modifications and variations may be made to the object of the present patent application described and illustrated herein without departing from the protective scope of the present invention as defined in the appended claims.

Claims

CLAIMS 1. A computer program product (6) comprising instructions which, when executed by electronic processing resources (10) of a system (1) for analyzing and predicting one or more behavioral states and / or transitions thereof among Awake (W), Microsleep (M), Sleep (S) and Dozing (D) of a subject, cause the electronic processing resources (10) to: - receive a biometric signal of a subject; - process the received biometric signal to classify it into one of different classes associated with the one or more behavioural states and / or transitions among Awake (W), Microsleep (M), Sleep (S) and Dozing (D) phases of a subject; and - detect and / or predict a behavioural state and / or a transition among Awake (W), Microsleep (M), Sleep (S) and Dozing (D) phases of the subject based on the classified biometric signal, wherein the instructions are further configured to cause the electronic processing resources (10) to: - compute at least one statistical quantity (^^^^^^^^, ^^^^^^^^^^, madACT) for and based on the received biometric signal; - verify that the at least one statistical quantity (^^^^^^^^, ^^^^^^^^^^, madACT) satisfies one or more proprietary criteria; and - classify the biometric signal into one of different classes associated with the one or more behavioural states and / or transitions among among Awake (W), Microsleep (M), Sleep (S) and Dozing (D) phases based on the result of the verification.

2. The computer program product (6) according to claim 1, wherein the biometric signal comprises alternatively a subset of a physiological data or a variable derived from said physiological data.

3. The computer program product (6) according to claim 2, wherein the physiological data and / or variable derived from said physiological data comprise values indicative of at least one cardiac output, in particular heart rate (HR) and heart rate variability (HRV).

4. The computer program product (6) according to claim 3, wherein, in order to compute at least one statistical quantity (^^^^^^^^, ^^^^^^^^^^, madACT) for and based on the received biometric signal, the instructions of the computer program product (6) are further configured to cause the electronic processing resources (10) to compute at least a standard deviation of the heart rate variability (^^^^^^^^), a mean value of the standard deviation of the heart variability rate (^^^^^^^^^^), and a mean absolute deviation of the activity (madACT).

5. The computer program product (6) according to claim 4, wherein, in order to verify that the at least one statistical quantity (^^^^^^^^, ^^^^^^^^^^, madACT) satisfies one or more proprietary criteria, the instructions of the computer program product (6) are further configured to cause the electronic processing resources (10) to: - verify if the value of the mean absolute deviation of the activity (^^^^^^^^^^^^) is below a first threshold (TH1) and if the mean value of the standard deviation of the heart rate variability (^^^^^^^^^^) assumes the minimum value to determine if a transition between the Awake (W) and Microsleep (M) phases occurs; - verify if the value of the mean absolute deviation of the activity (^^^^^^^^^^^^) is below the first threshold (TH1) and if the standard deviation of the heart rate variability (^^^^^^^^) assumes substantially the same value in a first-time interval (X) to determine if a transition between the Awake (W) and Sleep (S) phases occurs; and - verify if the value of the mean absolute deviation of the activity (^^^^^^^^^^^^) is below a second threshold (TH2) and if the standard deviation of the heart rate variability ^^^^^^^^assumes substantially the same value in a second time interval (Y) to determine if a transition between the Awake (W) and Dozing (D) phases occurs, the second time interval (Y) being separate and greater than the first-time interval (X).

6. The computer program product (6) according to claim 5, wherein, in order to classify the biometric signal into one of different classes associated with the one or more behavioural states and / or transitions among among Awake (W), Microsleep (M), Sleep (S) and Dozing (D) phases based on the result of the verification, the instructions of the computer program product (6) are further configured to cause the electronic processing resources (10) to:- determine the occurrence of the transition between the Awake (W) and Microsleep (M) phases if the mean value of the values of the standard deviation of the heart rate variability (^^^^^^^^^^) assumes the minimum value and the value of the mean absolute deviation of the activity (^^^^^^^^^^^^) is below the first threshold (TH1); - determine the occurrence of the transition between the Awake (W) and Sleep (S) phases if the standard deviation of the heart rate variabilityassumes substantially the same value in the first-time interval (X) and the value of the mean absolute deviation of the activity (^^^^^^^^^^^^) is below the first threshold (TH1); and - determine the occurrence of the transition between the Awake (W) and Dozing (D) phases if the standard deviation of the heart rate variabilityassumes substantially the same value in the second time interval (Y) and the value of the mean absolute deviation of the activity (^^^^^^^^^^^^) is below the second threshold (TH2).

7. The computer program product (6) according to claim 6, wherein the first time interval (X) comprises approximately one up to four seconds, preferably two seconds, and the second time interval (Y) comprises approximately from 60 to 200 seconds, in particular as a function of the standard deviation of the heart rate variability (^^^^^^^^).

8. System (1) for analyzing and predicting one or more behavioral states and / or transitions thereof among Awake (W), Microsleep (M), Sleep (S) and Dozing (D) of a subject comprising electronic processing resources (10) configured to store and execute a computer program product according to any one of the preceding claims.

Images