System and method for monitoring operational cognitive state of an individual

Abstract

A system is presented for monitoring an operational cognitive state of an individual during certain activity of the individual. The system comprises: a sensing system, and a control system. The sensing system comprises at least one motion sensor for measuring motion originated on at least one body part of the individual and generating corresponding measured data indicative of said motion, wherein changes in the corresponding measured data are sensitive to at least first and second levels of first and second cognitive conditions of an individual associated with, respectively, at least first and second characteristic factors defining together a combined effect on the operational cognitive condition of an individual. The combined effect correlates with the changes in said first and second levels of the first and second cognitive conditions. The measured data is therefore indicative of changes in the operational cognitive condition. The control system comprises a data processor and analyzer which analyzes the measured data, and, upon identifying that said motion comprises one or more features corresponding to a predetermined thresholding condition of said combined effect, generates output data indicative thereof. This enables detection of a critical operational cognitive state of the individual during said certain activity, irrespective of whether or not at least one of said first and second levels of the first and second cognitive conditions has reached at least one of first and second respective thresholds.

Description

[0001] SYSTEM AND METHOD FOR MONITORING OPERATIONAL COGNITIVE

[0002] STATE OF AN INDIVIDUAL

[0003] TECHNOLOGICAL FIELD AND BACKGROUND

[0004] The present disclosure is in the field of monitoring techniques and relates to a method and system for monitoring operational cognitive state of an individual during certain activity.

[0005] A person's cognitive condition, or cognitive state, is also known as a person's "state of mind". This state of mind may be normal (e.g., interested, sleepy, asleep, alert, bored, curious, doubtful, etc.), or it may be indicative of some type of pathology (e.g., amnesia, confusion, panic etc.). Often, such states of mind will manifest themselves measurably before a person (subjectively) realizes that he / she is entering such a state of mind.

[0006] The cognitive condition / state represents the momentary state of cognitive functioning, which can be influenced by various factors / conditions such as fatigue, intoxication (drunkenness), emotional state, stress, medication, environmental factors, or other transient elements. Cognitive states can fluctuate throughout the day and can affect a person's cognitive abilities and performance of tasks.

[0007] For example, if an individual is well-rested, focused, and not experiencing significant stress, said individual may be in a more optimal cognitive operational state, leading to improved cognitive performance. Conversely, if an individual is tired, distracted, or emotionally overwhelmed, the cognitive operational state of said individual may be suboptimal, resulting in decreased cognitive abilities temporarily.

[0008] Techniques have been developed aimed at monitoring individual's state / condition based on brain commands' detection, using EEG signals and / or movements. For example, US Patent Nos. 10,413,246 and 11,141,113, assigned to the assignee of the present application, describe techniques dealing with controlling machine operation via monitoring cognitive brain commands of a user, in order to detect the motor command related data corresponding to a condition of user's detection of error in his / her cognitive brain command. WO22123568, assigned to the assignee of the present application, describes a technique for monitoring an individual's activity via analysis of motion patterns collected over time from individual's body in order to identify a cognitive operational state of the individual during said activity.

[0009] In various contexts, it is crucial to monitor individuals for specific physiological or cognitive conditions to prevent accidents, ensure optimal performance, or protect their health.

[0010] GENERAL DESCRIPTION

[0011] There is a need in the art for a novel approach in determining an individual's cognitive operational state during a certain activity of an individual, which can be identified from parallel analysis of different physiological or cognitive conditions of the individual being monitored during said certain activity.

[0012] It should be noted that in the description below the terms "physiological", "emotional" and "cognitive" are used interchangeably with respect to a cognitive condition of an individual. The term " operational cognitive state" as used herein refers to the individual's cognitive state during a certain activity of the individual and is a result of concurrent effects of two or more cognitive conditions of the individual, associated with respective two or more characteristic factors. The term "characteristic factor", at time referred to as "factor", relates to a source / reason for the alteration of the respective cognitive condition from the normal one.

[0013] When one wants to identify what is the cause / source of alteration of the emotional or cognitive condition of an individual from a normal cognitive condition, one is forced to use different measurement techniques to evaluate each specific factor as an option to be a source of said alteration of the cognitive condition. For example, cameras in vehicles are suggested to be used to monitor driver's head position and / or eyelid closure and / or pupil movements, as an indicator / evaluator of such a characteristic factor as fatigue indicative of driver's inattention (change in the cognitive condition or operational cognitive state of the driver). Breath analyzers can be used to measure such a factor as drunkenness (the level of alcohol in the driver's blood) affecting a change in the driver's cognitive condition or operational cognitive state (e.g., a state of alertness). Traditional methods rely on independent models for each cognitive condition, issuing warnings or alerts when a single (specific) characteristic factor (e.g., drunkenness, fatigue, etc.) reaches a predetermined level defined as a threshold.

[0014] However, the inventor has found that in cases where multiple factors interact (i.e. are concurrently present) and each affects the cognitive condition of an individual (e.g., alertness while driving a car), their combined effect might exceed the "combined" warning level (threshold) related to certain activity of the individual (e.g., driving a car), even if each characteristic factor independently does not exceed its respective warning level (threshold). Moreover, the case may be such that increase of an effect of one factor reduces that of the other factor on the cognitive condition to be below the warning threshold (i.e., antagonistic effect), and only the combined effect appears to be the effective measure of the operational cognitive state of the individual.

[0015] The present disclosure relates to the field of utilizing artificial intelligence (e.g., machine learning) techniques for the parallel analysis of different factors / cognitive conditions synergistically affecting a cognitive operational state of a person during certain activity of said person. Specifically, the present disclosure addresses a situation where an effect of each of such different factors and each corresponding cognitive condition, when considered independently from the other factor(s) and condition(s), may not exceed a warning threshold level defined for the respective cognitive condition. However, the combined effect of two or more different factors and corresponding cognitive conditions on a person's operational cognitive state surpasses the warning threshold, necessitating a warning or an alert. The inventors propose a solution to detect and respond to such combined effects in real-time, thereby ensuring the safety and well-being of individuals, particularly but not limited to scenarios like monitoring car drivers for fatigue and drunkenness simultaneously.

[0016] Thus, according to one broad aspect of the present disclosure, it provides a system for monitoring an operational cognitive state of an individual during certain activity of the individual, the system comprising: a sensing system comprising at least one motion sensor configured and operable to measure motion originated on at least one body part of the individual and generate corresponding measured data indicative of said motion, wherein changes in the corresponding measured data being sensitive to at least first and second levels of first and second cognitive conditions of an individual associated with, respectively, at least first and second characteristic factors defining together a combined effect on the operational cognitive condition of an individual, said combined effect correlating with the changes in said first and second levels of the first and second cognitive conditions, said measured data being therefore indicative of changes in the operational cognitive condition; a control system comprising a data processor and analyzer configured and operable to analyze the measured data and upon identifying that said motion comprises one or more features corresponding to a predetermined thresholding condition of said combined effect, generate output data indicative thereof, thereby enabling detection of a critical operational cognitive state of the individual during said certain activity, irrespective of whether or not at least one of said first and second levels of the first and second cognitive conditions has reached at least one of first and second respective thresholds.

[0017] The monitoring system preferably further includes a notification utility configured and operable to be responsive to said output data to generate a corresponding notification (e.g., warning / alert).

[0018] In some embodiments, the control system also generates the output data (warning / alert) upon identifying that the combined effect reaches a condition corresponding to pre-defined threshold level of each of the first and second cognitive conditions separately and independently.

[0019] The monitoring system preferably includes a triggering utility. In some embodiments, the triggering utility is configured and operable to be responsive to sensing data indicative of the changes in at least one of the first and second levels of the first and second cognitive conditions, to identify that said at least one of the first and second levels of the first and second cognitive conditions has reached a respective first or second value corresponding to a certain fraction of the at least one of the first and second thresholds, and activate said data processor and analyzer to analyze said measured data. In some other embodiments, the triggering utility is configured and operable to be responsive to sensing data indicative of the changes in at least one of the first and second levels of the first and second cognitive conditions, to identify that said at least one of the first and second levels of the first and second cognitive conditions has reached a value corresponding to a certain fraction of the at least one of the first and second thresholds, and activate said sensing system to measure said motion and generate said measured data.

[0020] The sensing system may be configured and operable to provide said sensing data to the triggering utility. Alternatively or additionally, the triggering utility is configured for data communication with at least one sensor to receive said sensing data therefrom and analyze said sensing data to identify whether the sensing data is indicative of that said at least one of the first and second levels of the first and second cognitive conditions has reached a value corresponding to a certain fraction of the at least one of the first and second thresholds.

[0021] The at least one sensor may be configured and operable to collect data indicative of motion originated on at least one body part of the individual and generate motion data, said triggering utility being configured and operable to analyze the motion data by applying thereto at least one of first and second model-based processing associated with, respectively, at least one the first and second cognitive conditions, to extract said sensing data. Alternatively, or additionally, the at least one sensor is configured and operable to directly collect said sensing data from the individual.

[0022] In some embodiments, the certain activity comprises driving a vehicle, said at least first and second characteristic factors comprising fatigue and drunkenness factors, independently affecting the first and second cognitive conditions.

[0023] In some embodiments, the certain activity comprises driving a vehicle, said at least first and second characteristic factors comprising fatigue and drunkenness factors, independently affecting the first and second cognitive conditions, said first and second values of the certain fractions of the first and second thresholds are, respectively.

[0024] In some embodiments, the at least one sensor of the sensing system comprises at least one of the following: camera measuring the motion of individual's eyes and / or head; pressure sensor measuring the motion associated with individual's pressure on a gas pedal; a sensor measuring steering wheel angular motion; a pressure sensor measuring changes in individual's pressure on a seat chair; radar system, etc.

[0025] As described above, in some embodiments, the monitoring system includes a triggering utility configured and operable to be responsive to sensing data indicative of the changes in at least one of the first and second levels of the first and second cognitive conditions, to identify that said at least one of the first and second levels of the first and second cognitive conditions has reached a respective first or second value corresponding to a certain fraction of the at least one of the first and second thresholds, and activate at least one of said data processor and analyzer to analyze said measured data or said sensing system to provide the measured data, said sensing data being directly collected from the individual by at least one of a breathalyzer and a face camera.

[0026] In another broad aspect of the present disclosure, it provides a method for monitoring an operational cognitive state of an individual in relation to acting as a driver of a vehicle, the method comprising: measuring motion originated on at least one body part of the individual and generating corresponding measured data indicative of said motion, wherein changes in the corresponding measured data being sensitive to at least first and second levels of first and second cognitive conditions of an individual associated with, respectively, fatigue and drunkenness characteristic factors defining together a combined effect on the operational cognitive condition of an individual, said combined effect correlating with the changes in said first and second levels of the first and second cognitive conditions, said measured data being therefore indicative of changes in the operational cognitive condition; analyzing the measured data and upon identifying that said motion comprises one or more features corresponding to a predetermined thresholding condition of said combined effect, generating output data indicative thereof, thereby enabling detection of a critical operational cognitive state of the individual acting as the driver of vehicle, irrespective of whether or not at least one of said first and second levels of the first and second cognitive conditions has reached at least one of first and second respective thresholds in relation to, respectively, fatigue and drunkenness factors.

[0027] The present disclosure, in its yet further broad aspect, provides a vehicle comprising: a sensing system comprising at least one motion sensor configured and operable to measure motion originated on at least one body part of the individual and generate corresponding measured data indicative of said motion, wherein changes in the corresponding measured data being sensitive to at least first and second levels of first and second cognitive conditions of an individual associated with, respectively, fatigue and drunkenness characteristic factors defining together a combined effect on the operational cognitive condition of a vehicle's driver, said combined effect correlating with the changes in said first and second levels of the first and second cognitive conditions, said measured data being therefore indicative of changes in the operational cognitive condition; a controller comprising a data processor and analyzer configured and operable to analyze the measured data and upon identifying that said motion comprises one or more features corresponding to a predetermined thresholding condition of said combined effect, generate output data indicative thereof, thereby enabling detection of a critical operational cognitive state of the driver, irrespective of whether or not at least one of said first and second levels of the first and second cognitive conditions has reached at least one of first and second respective thresholds.

[0028] In yet another broad aspect, the present disclosure provides a control system for use in monitoring an operational cognitive state of an individual during certain activity of the individual, the control system being configured and operable as a computerized system comprising: a data processor and analyzer configured and operable to analyze input measured data indicative of motion originated on at least one body part of the individual, to analyze the measured data and upon identifying that said motion comprises one or more features corresponding to a predetermined thresholding condition, generate output data indicative thereof; a triggering utility configured and operable to be responsive to sensing data indicative of changes in at least one of first and second levels of first and second cognitive conditions of an individual associated with, respectively, at least first and second characteristic factors defining together a combined effect on the operational cognitive condition of an individual, said combined effect correlating with the changes in said first and second levels of the first and second cognitive conditions; and upon identifying that said at least one of the first and second levels of the first and second cognitive conditions has reached a respective first or second value corresponding to a certain fraction of the at least one of the first and second thresholds, generate a triggering signal to activate said data processor and analyzer to analyze said measured data; said control system thereby enabling detection of a critical operational cognitive state of the individual during said certain activity, irrespective of whether or not at least one of said first and second levels of the first and second cognitive conditions has reached at least one of first and second respective thresholds.

[0029] BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to better understand the subject matter that is disclosed herein and to exemplify how it may be carried out in practice, embodiments will now be described, by way of non-limiting examples only, with reference to the accompanying drawings, in which:

[0031] Fig. 1 shows schematically the method for monitoring an operational cognitive state of an individual during certain activity according to the present disclosure;

[0032] Figs. 2A to 2C exemplify variation of cognitive conditions caused by such characteristic factors as drunkenness F1and fatigue F2(Figs. 2A and 2B), and a combined effect of these cognitive conditions on the cognitive operational state (Fig. 2C) of an individual; wherein Fig. 2A shows the level of intoxication (constituting a cognitive condition) as a function of the blood alcohol content (BAC) measured by breathalyzer; Fig. 2B shows the level of fatigue-related cognitive condition as a function of specific facial features obtained by a camera; and Fig. 2C shows the combined effect of alcohol- and fatigue-related cognitive conditions determined by two-dimensional parametric investigation;

[0033] Figs. 3A and 3B exemplify schematically the technique of the present disclosure for monitoring the operational cognitive state of an individual, wherein Fig. 3A shows two calibration curves (e.g., obtained by machine learning techniques), one for the drunkenness factor (F1) and one for the fatigue factor F2, describing a degree / level L3of a third characteristic factor F3(a so-called "effective" or "combined" factor) created by parametric manipulation of each of the characteristic factors F1and F2; and Fig. 3B shows an effect of the effective factor on the cognitive condition associated with a synergistic effect of the two factors F1and F2(e.g., a case where although a threshold TH31 is crossed, the actual intoxication level is not exceeded (i.e., L1< Lx(THi)); Fig. 4 is a block diagram exemplifying a system for monitoring operational cognitive state of an individual;

[0034] Figs. 5A and 5B show a model creation system (Fig. 5A) and a method of model creation (Fig. 5B) for use in monitoring the operational cognitive state of an individual, according to the present disclosure;

[0035] Fig. 6 shows a flow diagram exemplifying a method of the present disclosure to create a model (e.g., by using machine learning techniques) to identify the combined effect of the multiple (at least two) cognitive conditions, associated with respective characteristic factors, on the operational cognitive state of an individual; and

[0036] Figs. 7A-7C exemplify, by way of flow diagrams, three possible implementations of the technique of the present disclosure for real time monitoring the operational cognitive state of a vehicle's driver.

[0037] Figs. 8A to 8D exemplify features extracted from motion patterns sensed from vehicle's steering wheel which are sensitive to both cognitive conditions: intoxication of the driver and driver’s fatigue; wherein Figs. 8A and 8B show two sets of dynamical measurements of steering wheel angle where Fig. 8A shows that the intercept of a linear fit of data is different for Operational cognitive state as compared to Inoperational cognitive state and Fig. 8B shows that the difference between intercepts of a linear and polynomial fit is another distinguishing feature between Operational and Inoperational cognitive states; Fig. 8C shows the feature of median frequency obtained from analysis of power spectra of measured steering wheel acceleration signals in Operational and Inoperational cognitive states, respectively; and Fig. 8D shows the feature of intercepts of model fitted to estimate autocorrelation coefficients obtained from the respective analysis of steering wheel angle dynamical (i.e., time dependent) data measured for Operational and Inoperational cognitive states.

[0038] DETAILED DESCRIPTION OF EMBODIMENTS

[0039] As mentioned above, the present disclosure provides a solution for real-time monitoring (detecting and possibly responding to) an operational cognitive state of an individual while performing certain activity, thereby ensuring the safety and well-being of individuals. The technique of the present disclosure can advantageously be used in controlling the individual during certain activity.

[0040] More specifically, the technique of the present disclosure is useful in scenarios like monitoring car drivers for fatigue and drunkenness simultaneously and is therefore exemplified below with respect to this specific application. However, it should be noted, and it is also clear from the present disclosure, that the principles of the present disclosure are not limited to this specific application. In the description below the cognitive conditions associated with (affected / induced by) such factors as fatigue and drunkenness are at times referred to as, respectively, fatigue-related cognitive condition and drunkenness- or alcohol-related cognitive condition.

[0041] Reference is made to Fig. 1 showing schematically the principles underlying the technique of the present disclosure for monitoring an operational cognitive state of an individual during certain activity. The cognitive operational state of an individual during and / or for the purposes of certain activity of the individual is a result of concurrent effects of two or more cognitive conditions (e.g., Cognitive condition 1, Cognitive condition 2, . .., Cognitive condition M) of the individual, associated with respective two or more characteristic factors (e.g., F1, F2, FM). As already mentioned above, the term "characteristic factor", at time referred to as "factor" in this disclosure, relates to a source / reason for the alteration of the respective cognitive condition from the normal one.

[0042] More specifically, the technique of the present disclosure can be used for monitoring operational cognitive state of a vehicle's driver (during driving and / or intending to start driving), and is therefore described below with reference to this specific application. However, it should be understood that the principles of the present disclosure are not limited to this specific application.

[0043] Considering driving of vehicle by an individual, fatigue and drunkenness are two significant factors that can compromise a driver's cognitive and physiological abilities (i.e., his / her operational cognitive state).

[0044] Traditionally, independent technique(s) is / are used for evaluating / monitoring each cognitive condition associated with a specific factor, in order to properly issue warnings or alerts when a direct measure of the level of the respective cognitive condition (e.g., poor judgment / alertness condition 1 because of drunkenness, or poor judgment / alertness condition 2 because of fatigue) corresponds to a predetermined threshold (THi or TH2). However, the inventor has found that in cases where multiple factors interact (i.e. are concurrently present) and each affects the cognitive condition of an individual (alertness while e.g., driving a car), some combined factor appears causing a combined effect which, when reaching a combined threshold THCom (in relation to certain activity of the individual, defines a change in the operational cognitive state of the individual. Moreover, the inventor has found that even if each characteristic factor independently does not reach / exceed its respective warning threshold, such "effective" or "combined" factor denoted as F3in Fig. 1 and representing a third characteristic factor F3, might arrive to a threshold affecting the operational cognitive state of the individual.

[0045] The present disclosure addresses a situation where an effect of each of such different factors (e.g., F1and F2) and each corresponding cognitive condition, when considered independently from the other factor(s) and condition(s), may not exceed a warning threshold level defined for the respective cognitive condition (i.e., THi, TH2). However, the combined effect of two or more different factors (e.g., represented by a third "combined" factor F3) and corresponding cognitive conditions on a person's operational cognitive state surpasses the warning threshold (denoted F3(THCom) in Fig. 1), necessitating a warning or an alert.

[0046] Reference is made to Figs. 2A to 2C exemplifying a straightforward (direct) measurement to evaluate the combined effect of two different cognitive conditions associated with respective two characteristic factors, drunkenness F1and fatigue F2, on the cognitive operational state of an individual, e.g., alertness during car / vehicle driving. The levels of cognitive conditions are measured / evaluated using suitable sensors.

[0047] For example, Fig. 2A shows a variation of a degree / level, L1, of a cognitive condition, associated with such characteristic factor as intoxication / drunkenness, measured as blood alcohol content (BAC), which may be detected e.g., by a breathalyzer in the breath of a driver. When BAC reaches a predetermined critical value / threshold THi (e.g., of about 0.08%), the respective cognitive condition arrives at a thresholding level, Lx(THi), of intoxication, a warning signal can be issued.

[0048] In a similar manner, Fig. 2B shows a variation of the degree / level, L2, of another cognitive condition associated with its respective characteristic factor, fatigue, directly measured / determined from image data, i.e., respective features obtained by a camera. Driver facing camera(s) may track eyes and / or other facial features and / or head movement, and based on pre-defined corresponding critical / threshold appearance TH2 of such features, a corresponding critical level, L2(TH2), of the respective cognitive condition is identified, requiring a warning about a deterioration in driver's cognitive and physiological abilities. For example, it is known to evaluate the fatigue-related cognitive condition via correlation between the directly acquired image data and known Karolinska Sleepiness Scale (KSS).

[0049] However, it is generally known that alcohol itself can induce fatigue, resulting in poor judgment while driving vehicle and thus leading to accident cases. Therefore, it is desired and even important to identify properly and effectively a combined effect of both cognitive conditions (associated with both characteristic factors) on the operational cognitive state of a driver. Such combined effect is actually a result of a synergistic interaction between drunkenness and fatigue factors.

[0050] Fig. 2C exemplifies the meaning of the combined effect of the cognitive conditions caused by two characteristic factors, F1and F2, on a person's operational cognitive state, as compared to the independent / separate effect of each of these factors on the cognitive condition. The figure shows a mutual dependence between the evolutions of the levels Li and L2 of the alcohol-related and fatigue-related cognitive conditions, and the respective parametric spaces of threshold. As shown, a zone Zi corresponds to "normal" levels of both the alcohol-related and the fatigue-related cognitive conditions, and zone Z2 presents an abnormal / alertness operational cognitive state caused by a combined effect / factor of both "normal" cognitive conditions, i.e., while each of the alcohol-related and the fatigue-related cognitive conditions is at its under-thresholding level. This abnormal / alertness operational cognitive state would not be detected by monitoring each of the alcohol-related and the fatigue-related cognitive conditions separately, i.e., the abnormal / alertness operational cognitive state would not be detected by any one of the independently measured levels of alcohol-related and the fatigue-related cognitive conditions. In other words, the level of each of the cognitive conditions, L1and L2, may not exceed the respective warning threshold level (L1< L'(THi) and L2< L2(TH2)), however, the combined effect of both cognitive conditions on a person's operational cognitive state may result in severe impairment and surpass the warning threshold e.g., of alertness. It is therefore desired to identify impairments in the operational cognitive state of an individual in relation to a certain activity of the individual by considering the combined effect of various cognitive conditions, where each cognitive condition, separately, may not reach the warning threshold.

[0051] The inventors understood that it is difficult and even not practical to directly measure / evaluate such impairment via a thresholding condition ("combined threshold") by the parametric evaluation of two-dimensional combined effect of alcohol- and fatigue- related cognitive conditions. Establishing a full parametric correlation between the levels of the alcohol-related and fatigue-related cognitive conditions requires a prohibitively large number of complicated measurements even if technically possible. There are two problems in creating an accurate, empirical model for the joint effects of fatigue and intoxication. First, in order to create such a model, different combinations of the two factors are to be parametrically manipulated and the model is to be trained to identify them. Such parametric manipulations require expensive and lengthy large-scale experiments. Second, after such a parametric manipulation has been performed and a model is created that detects combinations of fatigue and drunkenness, there still is a need to decide what are the thresholds that require a warning. The trained model by itself cannot provide information on a specific threshold that constitutes a hazard. In order to do this, a long and complicated series of additional experiments are to be conducted in which the effect of the various combinations of alcohol and fatigue on the actual functioning is tested.

[0052] Referring back to Figs. 2A and 2B, they illustrate straightforward direct measurements of the cognitive conditions associated with drunkenness (alcohol) and fatigue factors. The present disclosure provides an indirect monitoring technique to measure / evaluate the combined effect of various cognitive conditions on the operational cognitive state of the individual where each of these cognitive conditions, separately, may not reach the warning threshold for the respective one of these cognitive conditions.

[0053] The present disclosure introduces the concept of utilizing a third variable, which is a parameter / measure F3of a so-called "effective factor" or "combined factor" affecting the operational cognitive state as a result of a combined effect of two or more different cognitive conditions associated with two or more respective different factors. This parameter F3by itself correlates with the changes / variation in the level(s) of the cognitive condition(s) associated with at least one of said different factors F1and / or F2, e.g., at least one of the alcohol- and fatigue-related factors F1and F2, and while being affected by both of these cognitive conditions, is also correlated with both factors independently to a certain extent.

[0054] By observing how variation of this third parameter F3responds to the variation in the levels of the two cognitive conditions (drunkenness- and fatigue-related cognitive conditions), an alert can be issued upon identifying that a level (value) of this third parameter F3is indicative of a predetermined thresholding condition of the combined effect, even if the direct (or model-based) measurement for any one of these cognitive conditions separately does not warrant an alert.

[0055] Reference is made to Fig. 3A exemplifying, schematically, the principles of the technique of the present disclosure for monitoring the operational cognitive state of an individual.

[0056] In a non-limiting example of an individual' activity while driving a vehicle, the vehicle may be equipped with a monitoring system (including a sensing system and a control system, as will be described more specifically further below) configured and operable to evaluate the driver's operational cognitive state based on detecting (by the sensing system) and analyzing (by the control system) measured data indicative of motion patterns originated in at least one body part of the individual and collected over time by the sensing system. Such motion patterns can be sensed directly from a body part of a driver or from a device operated by direct interaction with the driver's body part. This may be motion patterns sensed from vehicle's steering wheel, vehicle itself, pedal(s), one or more sensors attached to one or more parts inside the vehicle, etc.

[0057] The analyzing of the motion pattern data is aimed at identifying therein specific movement features or sub-movement(s). The term "sub-movement" used herein refers to a smaller, more specific component or offshoot of a larger movement, characterized by its unique feature(s). Sub-movements can be identified, for example, as changes in the steering wheel angle of the car, car speed, car acceleration, pedal operation, vibrations in a piezoelectric sensor installed under the driver's seat or by using an in-cabin radar that reads the driver's motion. Thus, in some embodiments, of the present disclosure, the combined effect of two or more cognitive conditions, associated with different factors, on the operational cognitive state of an individual may be identifiable / measurable as sub-movement(s) in the motion pattern originated by the individual's body part. For the purpose of exemplifying the technique of the present disclosure, one of sub-movements-based variables, e.g., sub-movements during pedal operation, is chosen to present / constitute the above-described " third variable" or "third index" or third parameter " denoted F3.

[0058] As described above, a change in the third index F3correlates with variation of a level of the cognitive condition associated with at least one of the alcohol- and fatigue- related factors F1and / or F2. For example, the third index F3may be correlated with the readings of the breathalyzer, and therefore, this third index F3by itself may reflect the level of intoxication reasonably well (in fact can be used as an alternative measure for intoxication). However, if the third index F3is also sensitive, to a certain extent, to the effect of fatigue on the sub -movements, then this third index F3, while still being correlated with the breathalyzer, may warn of a level of intoxication slightly higher than the level of intoxication Lx(THi) that the breathalyzer is configured to warn of.

[0059] Specifically, referring to Fig. 3A, two calibration curves are exemplified (obtained by machine learning techniques), one calibration curve C(F') describing a variation of the level / value L3of the third index F3with the change in the level L1of the cognitive condition associated with the drunkenness factor F1, and the other calibration curve C(F2) describing a variation of the level / value L3of the third index F3with the change in the level L2of the cognitive condition associated with the fatigue factor F2.

[0060] The inventors have already shown that there is / are certain one or more movement features (e.g., sub-movements during pedal operation), which are common for both characteristic factors, the drunkenness factor F1and the fatigue factor F2. i.e., common movement feature(s) in the motion pattern appearance of which or change of which is affected by both factors, F1and F2.

[0061] Each one of the calibration curves C(FX) and C(F2), describing the variation of the degree / level L3of the third index F3, may be created by parametric manipulation of each of the respective factors (F1and F2) separately. Thus, values of the level L3of the third index F3defined by corresponding thresholds, TH31 and TH32, can be identified and are denoted as L3(TH3I) and L3(TH32) in Fig. 3A.

[0062] The inventors have found that the combined effect of the specific cognitive conditions, e.g., fatigue- and drunkenness-related cognitive conditions, can be utilized to enhance the safety of the car driver. Let us consider the scenario described in Fig. 3B.

[0063] For example, if the value of the level L3of the third index F3crosses the level corresponding to the threshold TH31, i.e., L3> L3(TH3i), a warning of intoxication is issued indicating that an alcohol level exceeds the threshold established for the warning by direct measurement with breathalyzer (e.g., THi). However, since the third index F3(i.e., the chosen feature(s), e.g. sub-movement) is affected by both factors F1and F2, the crossing of the threshold L3(TH3i) might occur due to combined (e.g., synergistic) effect of intoxication and fatigue causing a cognitive decline of the same severity as the cognitive decline (expressed by L'(THi)) caused by the alcohol level that exceeds the threshold established for the warning by direct measurement with breathalyzer (e.g., THi). In other words, in this scenario, although the threshold condition L3(TH3i) was reached / crossed, the actual intoxication level was not exceeded (i.e., L1< Lx(THi)). In such a situation, a warning will be given for crossing the threshold of intoxication even if the breathalyzer does not indicate that the threshold of intoxication has been crossed, because a combined effect of intoxication and fatigue suggested a dangerous impairment of driver's cognitive state.

[0064] Reference is made to Fig. 4 showing by way of a block diagram the configuration and operation of a monitoring system 150 of the present disclosure, for monitoring operational cognitive state of an individual during certain activity of the individual. The system 150 includes a sensing system 110, a control system 170 and, in some embodiments, a triggering utility 190.

[0065] The sensing system 110 includes a number N (N>1) of sensors including at least one motion sensor, e.g., Sensor 1, configured and operable to detect motion originated on at least one body part of the individual and generate corresponding measured / sensed data MD indicative of said motion. The control system 170 is configured as a computerized system including, inter alia, an input utility 180, an output utility 182, a memory 184, and a processor & analyzer 183, and in some embodiments also includes a notification utility 185.

[0066] Variation of the measured data MD provided by the at least one motion sensor, Sensor 1, is sensitive to (depends on) changes in at least first and second levels, L1and L2, of first and second cognitive conditions of an individual associated with, respectively, at least first and second characteristic factors, F1and F2, defining together a combined effect on the operational cognitive state of an individual. Such combined effect defines a third variable, F3, which is a parameter / measure of the combined effect / factor affecting the operational cognitive state. This combined effect, represented by the parameter F3, correlates with the changes / variation in the first and second levels, L1and L2, of the first and second cognitive conditions. The measured data MD is therefore indicative of changes in the operational cognitive state.

[0067] The data processor & analyzer 183 is configured and operable to analyze the measured data MD and, upon identifying that the measured motion data is indicative of one or more motion-related features corresponding to a predetermined thresholding condition, F3(THCom), of the combined effect F3, generate output data indicative thereof. This enables detection of a critical operational cognitive state of the individual during the certain activity, irrespective of whether or not at least one of the first and second levels (L1and / or L2) of the first and second cognitive conditions (forming together the combined effect), has reached the respective at least one of first and second thresholds, THi and / or TH2.

[0068] The notification utility 185 is configured and operable to be responsive to said output data to generate a corresponding notification when a critical operational cognitive state of the individual is detected.

[0069] In some embodiments, the monitoring system also includes the triggering utility 190 which is configured and operable to be responsive to sensing data indicative of the changes in at least one of the first and second levels of the first and second cognitive conditions, and upon identifying triggering condition(s) for at least one of the first and second cognitive conditions, activate the data processor and analyzer 183 to analyze the measured data MD or activate the sensing system 110 (its motion sensor(s)) to start measurements. More specifically, the triggering condition is that at least one of the first and second levels of the first and second cognitive conditions has reached a respective first or second value corresponding to a certain fraction of the at least one of the first and second thresholds (i.e., fiTHi and / or f2TH2, fi < 1 and f2 < 1). In such case, the triggering utility generates an activation signal AS addressed to the data processor and analyzer 183 to activate it to start analyzing the measured / sensed data MD, or generates an activation signal AS addressed to the sensing system 110 to activate it to start measure the motion with the at least one motion sensor (e.g., Sensor 1) and generate the measured data MD.

[0070] The control system 170 is configured and operable to communicate with the sensing system 110 using any known suitable communication techniques and protocols. The control system 170 is also configured to access data stored in database 140. The database may be stored in memory 184 of the monitoring system or in a separate storage device to which the monitoring system has access via any known suitable communication technique. To this end, the control system 170 includes a data communication port / utility which is not specifically shown.

[0071] Reference is made to Figs. 5A and 5B exemplifying, by way of block diagrams, a model creation system 100 (Fig. 5A) and method 200 (Fig. 5B) for monitoring the operational cognitive state of an individual according to the present disclosure. The model creation system 100 utilizes machine learning techniques to create model(s) to be used later by the above-described monitoring system 150 for real-time monitoring the operational cognitive state of an individual during certain activity.

[0072] The model(s) is / are created once during a leaming / training stage, for each of two or more cognitive conditions of an individual in relation to a given activity (i.e., considering corresponding two or more factors associated with / affecting said two or more cognitive conditions). Also, the leaming / training stage considers given types of sensing data, i.e., measuring conditions including body part being sensed for movements and types of sensor(s) being used. All these model-based data (model(s) per given data about the activity-related factors and measuring conditions) is properly stored in database 140. Such database 140 typically includes a library with its data interpretation engine.

[0073] The monitoring system 150 can then repeatedly apply such model(s) to newly (real time) measured data pieces collected under the respective measuring conditions and relating to the corresponding activity performed by the individual, to monitor the operational cognitive state of the individual in order to properly detect / identify whether a combined effect of monitored cognitive conditions surpasses the pre-set warning threshold for at least one of these cognitive conditions.

[0074] The model creation system 100 includes a sensing system 110, a specific factor related threshold provider 118, and a combined effect model creator system 130.

[0075] The sensing system 110 includes a number N ( 2) of sensors (data sources). These data sources include at least one data source (e.g., Sensor 1 and / or Sensor 2) providing measured / sensed data MDi1and MD22indicative of, respectively, first and second levels L1and L2of the first and second cognitive conditions associated with first and second factors, F1and F2, respectively; and may also include at least one additional data source / sensor configured and operable as a so-called "combined sensor" for detecting and providing corresponding measured / sensed data MD31’2indicative of a level L3of cognitive condition of an individual affected by both cognitive conditions associated with the first and second factors F1and F2.

[0076] It should be noted that, generally, the same sensor can be used to provide sensing data enabling to extract therefrom different features indicative of separate effects of different characteristic factors, respectively F1and F2on the cognitive conditions of an individual. For example, both cognitive conditions can be separately analyzed / evaluated from the same sub-movement data utilizing specific / characteristic features distinguishing the two cognitive conditions. This technique is described in WO 2023 / 242842 assigned to the assignee of the present application. This technique can be used to monitor, for each specific factor separately, the level of the effect of said specific factor on the associated cognitive cognition, based on its unique (sub)movement features.

[0077] As exemplified in Fig. 5A, the sensing system 110 includes at least one first data source (Sensor 1) providing measured / sensed data MD11and at least one second data source (Sensor 2) providing measured / sensed data MD22, which data MD11and are MD22are specific for detection of first and second levels L1and L2of the first and second cognitive conditions associated with the first and second factors, F1and F2, respectively. These may for example be breathalyzer for drunkenness level detection and camera facing the driver for fatigue level detection. Such at least one first sensor (or at least one first and at least one second sensors) provides at least one first and at least one second measured / sensed data pieces MDi1and MD22in association with the first and second factors, F1and F2, respectively. The sensing system 110 further includes at least one combined sensor, e.g., Sensor 3, configured and operable for providing measured / sensed data MD31’2indicative of a level L3of cognitive condition of an individual affected by both cognitive conditions associated with the first and second factors F1and F2. Such combined sensor may for example include a motion sensor capable of monitoring movements originated on individual's body (e.g., driver's body), either via direct sensing of the body movement or via sensing movements of a device operable via interaction with the individual's body (e.g., steering wheel in a vehicle).

[0078] The inventors have found that, although different motion patterns originated at body part(s) of the individual can be used to identify and distinguish between different factors describing / relating to different cognitive conditions of an individual (e.g., intoxication, fatigue) during individual's activity, certain movement features may be indicative of the combined effect of both of such different factors on the operational cognitive state of an individual. These may be some sub-movements' characteristics (features) indicative of a critical level of the operational cognitive state of an individual as a result of concurrently occurring fatigue and intoxication, while the cognitive condition associated with at least one of the fatigue and intoxication factors may not be critical (i.e., may not be above the respective threshold). This will be exemplified further below.

[0079] It should be noted that the meaning of the term "sensor" should not be limited to sensors directly measuring motion / movement. Indeed, there is a variety of sensors which are not motion sensors per se and are capable of measuring one or more parameters from which motion-related data, being data indicative of motion, can be derived. Such sensors include for example a piezoelectric sensor, a pressure sensor, a force sensor. Hence, for the purposes of the present disclosure, a sensor may be of any known suitable type providing sensing data indicative of (i.e., which can be transformed into) motion data (also referred to as motion-related data). It should also be noted, and is mentioned above, that measurements of the motion-related data (sub-movements originated at the body part of the individual) may be implemented by a sensor physically attached to the body part (i.e., direct measurement of motion originated at the body part) or sensor measuring motion data originated at the body part via motion of a device / element interacting with the body part of the individual.

[0080] The sensing system 110 provides the respective measured / sensed data MDi1(Sensor 1), MD22(Sensor 2), MD31’2(Sensor 3), . . MDN1,2(Sensor N). The measured / sensed data is stored in a storage system 125 to be accessed by the model creation system 100 (using any known suitable data communication technique) for further analysis, or the measured / sensed data may be directly transferred to the model creation system 100 for further analysis and model building by the respective model creators of system 100.

[0081] The specific factor related threshold provider 118 includes a specific factor model creator 120A, a direct measurement system 120B, and memory utility 120C. The specific factor model creator 120A includes at least two machine learning based analyzers, 122 and 124, each configured and operable to perform machine learning procedures to create models or model data Mi1and M22for evaluation of the cognitive conditions associated with / affected by the characteristic factors F1and F2, respectively. To this end, the analyzer 122 is configured and operable to analyze the measured / sensed data MD11(collected by Sensor 1) using respective train set of labeled data pieces to create model data Mi1enabling accurately classify theoretical / predicted cognitive conditions associated with / affected by the characteristic factor F1from the respective measured / sensed data MD11. Similarly, the analyzer 124 is configured and operable to analyze the measured / sensed data MD22(collected by Sensor 2) using respective train set of labeled data pieces to create model data M22enabling accurately classify theoretical / predicted cognitive condition associated with / affected by the characteristic factor F2from the respective measured / sensed data MD22.

[0082] Thus, separate data analysis is performed in relation to cognitive conditions associated with the characteristic factors F1and F2, respectively, and separate respective models Mi1and M22are created.

[0083] The specific factor model creator 120A is further configured to utilize the model data Mi1and M22to determine threshold levels, THi and TH2, respectively, for the cognitive conditions associated with the characteristic factors F1and F2. These thresholds are determined based on scientific research, empirical data, and safety regulations, ensuring that an individual's impairment is appropriately assessed.

[0084] The combined factors model creator 130 includes an analyzer of "combined factors" correlation 132 configured and operable to analyze the measured / sensing data MD31’2(collected by at least one combined sensor, Sensor 3) and determine the combined-effect warning threshold, which may be warning threshold TH31 associated with the first characteristic factor F1(e.g., intoxication) and / or warning threshold TH32 associated with the second characteristic factor F2(e.g., for fatigue).

[0085] More specifically, the analyzer 132 includes a processor 134 operable to process the sensing data MD31’2and evaluate correlation of the combined effect F3with changes in the cognitive conditions associated with factors F1and F2and generate corresponding correlation data; and a model creator 136 which operates to analyze the correlation data and define respective model data M31and / or M32and generate data indicative of the combined-effect warning threshold(s) TH31 and / or TH32.

[0086] More specifically, the analyzer 132 may operate as follows: The processor 134 receives the measured / sensed data MD31’2(collected by at least one of combined sensors, e.g., Sensor 3) and identifies at least one variable / parameter or a combination of variables / parameters that is / are correlated with the evaluation of the level(s) L1and / or L2of at least one of the cognitive conditions associated with at least one of the factors F1and F2(e.g., fatigue or alcohol levels). To this end, individuals from the train group are parametrically manipulated within a full range of levels of drunkenness / fatigue related cognitive conditions. Each of the at least one variable / parameter is chosen based on its sensitivity to a change in the combined effect of both cognitive conditions associated with both factors F1and F2(e.g., of fatigue and drunkenness). Typically, such third variable / parameter(s) include(s) one or more specific sub-movements in the motion pattern originated in the body part.

[0087] For example, in a car, such a third variable can be the deviation from the center of a lane or a reaction time to various warnings of the vehicle (if there is a camera or a radar, it is possible to detect the reaction time of the driver's eye or head); or this can be a standard deviation of a distance from the vehicle in front; or a standard deviation of pressures on the gas or the intensity of pressures on the brake. The model creator 136 utilizes the correlation between the third variable F3(or the combination of measured / sensed data{ MD31’2(Sensor 3), MDN1’2(Sensor N) } data) and the level of at least one of the cognitive conditions (e.g., fatigue or alcohol) to create machine learning model(s), M31(and / or M32) configured to classify, through training and statistical analysis, various levels L (F') and L3(F2) of a change of the third parameter F3with the variation of the respective cognitive conditions associated with the factors F1and F2. The model(s) M31(or M32) allows the definition of respective warning threshold TH31 (e.g., for intoxication) and / or warning threshold TH32 (e.g., for fatigue) for the combined effect presenting an operational cognitive state of an individual during the certain activity.

[0088] It should be noted that since the measured data MD31’2(Sensor 3), MDN1’2(Sensor N) is sensitive to the combined effect associated with the characteristic factors of fatigue and drunkenness, it is practically impossible to identify (in real time) which of these factors is responsible for that the level of the combined effect exceeds either one of thresholds TH31 or TH32.

[0089] The present disclosure is aimed at detecting an impairment of an individual that would otherwise go unnoticed and, as will be described further below, the third variablebased indicator may be combined with one or more specific / separate indicators for identification of the specific cause of the impairment, if needed. This enhances safety measures, particularly in critical scenarios such as monitoring car drivers, where subtle impairments can lead to severe consequences.

[0090] The output of the model creation system 100 is saved in a library which is typically associated with an interpretation engine 140 for future use by the monitoring system 150 during "online" (real time) operational mode for monitoring an operational cognitive state of "real" individual.

[0091] As described above, the sensing system 110 includes at least one sensor unit which is configured to provide measured / sensing data sensitive to levels of cognitive conditions associated with characteristic factors F1and F2(e.g., fatigue and alcohol), respectively. As also described above, these measured data may be indicative of motion patterns (e.g., including sub-movements) detected over time from at least one body part of an individual, being measured directly from said body part or from a device being operated in association with said body part.

[0092] Thus, in some embodiments, there may be a situation where a local computer / controller in the vehicle is installed with corresponding model data and is configured and operable to analyze the driver's operational cognitive state through motion pattern analysis using respective models, e.g., first model data (e.g., Mi1) related only to the cognitive condition associated with the alcohol-related factor (i.e., sensitive to changes in the level of alcohol only) and second model data (e.g., M22) related only to the cognitive condition associated with fatigue (i.e., sensitive to changes in the level of fatigue only). Third model data (e.g., M31) can be used to analyze driver's specific operational cognitive state (e.g., alertness) through motion pattern analysis (i.e., identification of one or more specific features, e.g., sub-movements). This model data M31describes changes in a level, L3, of the third variable F3, and correlates with the changes of level of the alcohol-related cognitive condition, but unlike the alcohol-related model data Mi1, it is also sensitive to changes in the level of fatigue-related cognitive condition; and / or model data M32is used which describes changes in the level, L3, and correlates with the changes of level of the fatigue-related cognitive condition, but unlike the fatigue-related model data M22, it is also sensitive to changes in the level of alcohol-related cognitive condition. When the model-based analysis, using model data M31and / or model data M32, identifies a condition that the level L3reaches or crosses the warning threshold for alcohol (e.g., TH31) or warning threshold for fatigue (TH32), a warning signal is generated even though the alcohol-related model data (Mi1) based analysis or the fatigue-related model data M22based analysis has not yet detected a crossing of the respective warning threshold THi or TH2.

[0093] Generally, there may be multiple sensor units (N>1), which may include sensor units of the same type associated with different body parts of the individual and / or may include sensors of different types associated with the same body part of the individual, as the case may be. Thus, generally, the sensing data may include multiple data portions / pieces relating to a plurality of N sensor units, (MD1,23 . . . MD1,2N), where the combination of said sensing data was shown to be correlated with at least one of the measured conditions / factors F1and F2during the stage of model creation. Some examples of variables / indicators sensitive to both intoxication and fatigue, not necessarily requiring analysis of sub-movements include changing lane, change of steering wheel angle, erratic driving (e.g., abnormal usage of brakes / gas pedal).

[0094] It is described in the above-indicated patent application WO 2023 / 242842 assigned to the assignee of the present application that there are some motion pattern features common to both fatigue-related and alcohol-related cognitive conditions. Therefore, for the purposes of the technique of the present disclosure, there is no need to perform a full parametric manipulation of a full range of levels of intoxication / fatigue related cognitive conditions and measuring their cross-sensitivities. It should be noted that, depending on the individual's activity being monitored and / or an operational cognitive state to be identified, the monitoring system 150 may also be configured and operable to record data indicative of the monitoring results and / or properly transfer such data to and / or allow access to such data for an authorized person / entity. In addition, information can be used to implement an intervention that is specifically relevant to the characteristic factor that caused the cognitive decline. For example, in case of fatigue shake the chair; in case of drunkenness slow down the speed of travel.

[0095] The processor and analyzer 183 of the control system 170 receives the measured data from the sensing system 110 or storage system 125 and operates to utilize a predetermined model data stored in the library with its associated interpretation engine 140. As described above, the model data includes at least two sets of characteristic features (e.g., motion pattern related features), which describe separately first and second specific cognitive conditions, respectively; and includes a third set of characteristic features (typically motion pattern related features) which are sensitive to changes in the levels of both the first and the second cognitive conditions.

[0096] The data processor and analyzer 183 operates as described above to process the measured data by applying thereto the above models to identify the operational cognitive state of the individual in relation the individual's activity, enabling timely generation of an alert signal. The operation of the data processor and analyzer 183 is exemplified more specifically further below with reference to Fig. 6. The technique of the present disclosure provides real-time monitoring and alerting capabilities ensuring immediate responses when the combined effect of two cognitive conditions surpasses the warning threshold.

[0097] Reference is made to Fig. 5B exemplifying by way of a flow diagram 200 a method of the present disclosure to create a third model (e.g., by using machine learning techniques) to be used to identify combined effects of cognitive conditions on the operational cognitive state of an individual to enhance safety measures. It should be noted that the description of this example actually includes many possible options for the implementation of the data processing algorithm. Therefore, the exemplified order of the method steps described herein should not be limiting (the described order of events is not mandatory).

[0098] The method includes several learning mode stages, involving parametric manipulation of conditions / factors affecting human cognitive state. Specifically, in step 202 measured data MDi1and MD22are collected from at least two sources (e.g., Sensor 1 and Sensor 2) specific for a first and a second cognitive conditions, F1and F2, respectively. For example, breathalyzer for drunkenness and camera facing the driver for fatigue. Alternatively, fatigue and intoxication data can be acquired from motion sensors placed on the driver's body, electronic devices in contact with the person (e.g., car seat), or remote monitoring devices such as cameras or radars.

[0099] In a separate step (step 204) measured data MD31’2is collected from at least one third source(s) known to be affected by both cognitive conditions, F1and F2, for example, motion sensors placed on driver's body measuring sub-movements of the driver during various activities, etc.

[0100] Data pre-processing takes place in step 206, including noise removal, feature extraction, elimination of noise or outliers, and normalization, to ensure compatibility and quality for subsequent analysis. Depending on the data sources and modalities, appropriate preprocessing techniques such as image processing, signal filtering, or feature extraction algorithms may be employed.

[0101] In step 208, a first model Mi1for the first cognitive condition / factor F1and a second model M22for the second cognitive condition / factor F2, are created by training the models using labeled data MD11and MD22and independent parametric manipulation of the two conditions / factors. Once the models are trained, warning thresholds THi and TH2are determined (step 210) for each respective cognitive condition F1and F2independently (e.g., based on scientific research, empirical data, safety regulations etc.).

[0102] A third model M31(and / or M32) utilizing the measured data of the at least third sensor, MD31’2, is created (step 212) and its correlation with the measured data (by performing training with parametric manipulation of levels of first and / or second conditions) of at least one of the two sensors (Sensor 1 and / or Sensor 2) is analyzed. This correlation allows the determination of at least one warning threshold TH31 based on M31model (possibly also TH32 based on M32) (step 214).

[0103] Thus, steps 202 to 214 described above are considered as a preparatory stage and are performed only once with the chosen factors / conditions, e.g., drunkenness, fatigue, etc., for a given activity-type performed by an individual.

[0104] Fig. 6 exemplifies a flow diagram 300 of the implementation of the created models when running the model -based processing of the measured data in "real time". Measured data related to the conditions of interest, such as fatigue and drunkenness, are collected from specified sources including at least two sensors for two cognitive conditions and at least one (third) sensor affected by both cognitive conditions (step 302). The specific (and separate) models Mi1and M22are applied to the preprocessed measured data to analyze the levels of each cognitive condition independently to determine whether one of first or second conditions is above its threshold (step 304). If neither condition exceeds its threshold, the algorithm proceeds to the next step (step 308) to perform combined effect analysis of the third variable. Otherwise, an alert is generated (step 306), and further analysis is unnecessary.

[0105] The triggering utility 190 analyzes the measured data (step 308) to identify whether one of first or second levels of the first factor related and second factor related cognitive conditions reached a certain respective fraction ? or / 2 (fi andf2 may or may not be equal) of its respective threshold. For example, regarding the drunkenness factor, blood alcohol content (BAC) of 0.03 or above indicates that individual's cognitive condition is abnormal, and that further testing is required. Similarly, regarding the fatigue factor, Level 5 or above of the Karolinska Sleepiness Scale (KSS), or a corresponding fatigue level measured by other fatigue measure (e.g., by a camera facing the driver) indicates that fatigue level is abnormal, and that further testing is required.

[0106] If none of the levels were found abnormal (above the predetermined certain fraction of the respective threshold), no action is taken, and monitoring is continued (step 310). However, if at least one of the first or second measures of the first and second cognitive conditions reaches a certain fraction of its respective threshold, a combined effect analysis of the third variable is activated (step 312). This activation may be implemented by generation and sending an activation signal AS (by the triggering utility 190) to the sensing system 110 to acquire measured data by the third sensor affected by both cognitive conditions.

[0107] The combined effect of the first and second conditions, as indicated by the threshold(s) defined by the third model, M31(and / or M32) is analyzed (step 314). If the features derived from the third set of measured data show that a predetermined threshold is surpassed for at least one of the conditions (e.g., fatigue or alcohol), an alert is generated through appropriate means, e.g. via notification utility (step 316), notifying the individual (driver) or authorized person(s) and / or systems responsible for taking necessary actions. Otherwise, monitoring is continued (step 315).

[0108] In some embodiments the third index may be used as an index for crossing a warning threshold only if it is evident that it is correlated with at least one of the main indices (fatigue or alcohol) in real time. For example, in a situation where a warning about crossing a warning threshold for alcohol or fatigue is given based on a 10-minute measurement, then the third index will only be taken into account if during the 10 minutes it changes in a coordinated manner with the alcohol or fatigue index. For example, the 10-minute time window can be broken down into 10 one-minute time windows, giving a separate drunkenness score, a separate fatigue score, and a score for the third index separately for each minute of measurement. Then, a correlation is calculated between the different measurements. If a correlation is found between at least one of the main indicators and the third indicator, then it should be taken into account.

[0109] Fig. 7A shows an embodiment where the first model, Mi1, is a sub-movement- based machine learning model, utilizing unique features to detect intoxication / drunkenness only (having a threshold THi), the second model, M22, is a camera-based machine learning model utilizing unique features to detect fatigue only (having a threshold TH2), and the third model, M31, is a sub-movement-based machine learning model detecting combined effects of intoxication and fatigue. The third model includes a threshold for intoxication only e.g., TH31. If one of models Mi1or M22was found to be above a certain fraction of its respective threshold, the combined effect analysis of sub-movements is performed to detect intoxication, and the threshold TH31 is used for issuing a warning threshold, even when neither one of the specific thresholds (THi and TH2) was surpassed. For example, the regulation might require installation of a camera-based fatigue detection model (while not allowing use of motion-based model).

[0110] Fig. 7B shows an embodiment where the first model, Mi1, is a sub-movement- based machine learning model utilizing unique features to detect fatigue only (having a threshold TH2), the second model, M22, is a machine learning model based on breathalyzer to detect intoxication only (having a threshold THi), and the third model, M31, is a sub-movement-based machine learning model detecting combined effects of intoxication and fatigue. The third model may include a threshold for intoxication only e.g., TH31. If one of Mi1or M22models was found to be above a certain fraction of its respective threshold, the combined effect analysis of sub-movements is performed to detect intoxication, and the threshold TH31 is used for issuing a warning threshold, even when neither one of the specific thresholds (THi and TH2) was surpassed. Similarly, the case may be such that the regulation requires installation of a breathalyzer-based BAC detection model (not allowing use of motion-based model).

[0111] Fig. 7C shows an embodiment where the first and the second models are two independent sub-movement-based models Mi1and M22, each detecting uniquely, intoxication and fatigue, respectively, having respective thresholds, THi and TH2. The third model, M31or M32, is configured and operable to detect a third variable, e.g., pressure on the gas pedal, etc. (not necessarily sub-movement-based), having at least one threshold TH31 and / or TH32. It was shown that such 3rdvariable (not necessarily based on sub-movements) may correlate with each one of the independent models.

[0112] Similar to the cases described in relation to Figs. 7B and 7C, if one of Mi1or M22models was found to be above a certain fraction of its respective threshold, the combined effect analysis of the third variable (e.g., gas pedal, etc.) is performed to detect intoxication, and the threshold TH31 is used for issuing a warning threshold, even when neither one of the specific thresholds (THi and TH2) was surpassed.

[0113] For example, experiments conducted by the inventors have shown that indicators that reflect unexpected driving behavior such as frequent clicks at certain rhythms on the pedals are found in a certain correlation with drunkenness-related cognitive condition and with fatigue-related cognitive condition. The correlation with each of these factors is not high enough to accurately measure them individually. Both the sensitivity and specificity of this measure are not good enough. However, the fact that this third index is sensitive to both fatigue- and drunkenness-related cognitive conditions indicates that it is sensitive to their joint effect. This is why the correlation of the level of third index with the level of each of fatigue- and drunkenness-related cognitive conditions individually is relatively low. Hence, in a situation where the more accurate indices of fatigue and drunkenness each show the crossing of a certain threshold that does not yet require a warning, then when an index of the rate of pressing the pedal shows the crossing of the alert threshold pre-set for fatigue or drunkenness, it can be considered as a warning that reflects the joint effect of alcohol and fatigue that requires an alert.

[0114] Standard deviation of steering wheel angle (SDSWA) was shown by the inventors to correlate with both fatigue and intoxication. Thus, although this factor cannot be used as a unique identifier of the root cause of driver's impairment, it can be used as a single factor / indicator for warning, because SDSWA is affected by both fatigue and intoxication and therefore may be sensitive to their synergistic effects on the cognitive state of the driver.

[0115] Reference is made to Figs. 8A to 8D exemplifying features extracted from motion patterns sensed from vehicle's steering wheel which are sensitive to both intoxication of the driver and driver’s fatigue. These features serve to build a model, which is then used to process the measured data for distinguishing between an “operational state” and an “inoperational state” of a driver.

[0116] The curves marked as “operational state” in the figures describe a situation in which the driver is not tired at all and drank alcohol in an amount that causes the model to detect a level of 0.03 BAC, or a situation in which the driver did not drink alcohol at all and is tired at level KSS 4. The curves marked as "inoperational state" describe a situation in which the driver drank alcohol to a level of 0.03 BAC and is tired at level KSS 4. The inventors have shown that in such a situation the model, built based on the features described below, alerts both on crossing the warning threshold for alcohol (0.06) and on crossing the warning threshold for fatigue (KSS 8).

[0117] Figs. 8A and 8B show two sets of measurements of steering wheel angle when the driver is in a normal cognitive state (Operational cognitive state) and when the cognitive state of the driver is not normal (Inoperational cognitive state). The inoperational state of the driver can be achieved due to the combined effect of, e.g., intoxication and fatigue. For example, the curve of the Operational cognitive state may represent data obtained while the driver had 0.03% BAC and was not tired at all, or alternatively, did not drink alcohol at all and was tired at Level 4 of KSS, whereas the curve of Inoperational cognitive state may represent data obtained while the driver had 0.03% BAC and fatigue Level 4 of KSS . The inventors found that the dynamics (i.e., time dependence) of steering wheel angle changes as a result of changes in the cognitive state and responds to both intoxication and fatigue. Therefore, the inventors assume that this feature is sensitive to the combined effect of fatigue and intoxication.

[0118] It can be seen in Fig. 8A that a linear fit to the measured data of the steering wheel angle can be used to distinguish between Operational state of the driver and a general Inoperational cognitive state, without specifying / quantifying whether any one of the factors, e.g., fatigue or intoxication is above its individual threshold level. Fig. 8A shows that the intercept of the linear fit to the measured data of the steering wheel angle corresponding to an Operational state is significantly different from the corresponding intercept belonging to the data of the Inoperational state. The difference between the linear fit intercepts may be considered as a first feature distinguishing between Operational and Inoperational states based on measurements of steering wheel angle as a function of time.

[0119] Fig. 8B shows the difference between intercepts of linear and polynomial fits of steering wheel angle as a function of time, where such difference exemplifies a feature that can distinguish between Operational and Inoperational states based on measurements of steering wheel angle. The inventors found that in the Operational cognitive state there exists a large difference between the intercepts of a linear and polynomial fits (indicated by arrows), whereas in the Inoperational cognitive state the intercepts of a linear and polynomial fits (indicated by arrows) are very similar.

[0120] It should be noted that in both Fig. 8 A and Fig. 8B the specific value of the measured steering wheel angle is not used as the distinguishing feature between Operational and Inoperational states. It is either the difference between intercepts of linear fits and / or the difference between intercepts of a linear and polynomial fits which is used to define the thresholds of Inoperational state requiring a warning.

[0121] Reference is made to Fig. 8C showing power spectra of measured steering wheel acceleration signals in Operational and Inoperational cognitive states, respectively. The respective median frequencies of the two curves are indicated by vertical dashed lines. The median frequency of the Inoperational cognitive state is lower than the median frequency of the Operative cognitive state and can thus be used as a third feature distinguishing between the two cognitive states being sensitive to the combined effects of at least two factors, e.g., intoxication and fatigue.

[0122] Reference is made to Fig. 8D showing autocorrelation coefficients obtained from the respective analysis of steering wheel angle dynamical (i.e., time dependent) data measured for Operational and Inoperational cognitive states and the respective model fits that estimate the autocorrelation coefficients. Each respective model estimates the respective autocorrelation coefficients of the measured steering wheel angle data as a function of time. The inventors found that the intercept of the fitted model may serve as a fourth feature distinguishing between Operational and Inoperational cognitive states being sensitive to the combined effects of at least two factors, e.g., intoxication and fatigue.

[0123] It should be noted that this distinguishing feature is not sensitive to the quality of the model fit to the autocorrelation coefficients, however the intercept of the fitted model was found by the inventors to distinguish between the measured data of the two cognitive states and thus may serve as a thresholding parameter to identify an Inoperational cognitive state.

[0124] Thus, the inventors found a set of features in the steering wheel dynamics of a driver that respond to both alcohol and fatigue. A combination of all the features described in Figs. 8A to 8D (along with additional features not described in the graphs) provide the basis for creating a model indicating of the driver's competency level, i.e., whether the pre-defined competency threshold was crossed or not. Specifically, the inventors were able to calibrate the model to warn of incompetence (inoperational state) when the alcohol level crosses 0.05 BAC and the fatigue level crosses level KSS 7.

Claims

CLAIMS:

1. A system for monitoring an operational cognitive state of an individual during certain activity of the individual, the system comprising: a sensing system comprising at least one motion sensor configured and operable to measure motion originated on at least one body part of the individual and generate corresponding measured data indicative of said motion, wherein changes in the corresponding measured data being sensitive to at least first and second levels of first and second cognitive conditions of an individual associated with, respectively, at least first and second characteristic factors defining together a combined effect on the operational cognitive condition of an individual, said combined effect correlating with the changes in said first and second levels of the first and second cognitive conditions, said measured data being therefore indicative of changes in the operational cognitive condition; a control system comprising a data processor and analyzer configured and operable to analyze the measured data and upon identifying that said motion comprises one or more features corresponding to a predetermined thresholding condition of said combined effect, generate output data indicative thereof, thereby enabling detection of a critical operational cognitive state of the individual during said certain activity, irrespective of whether or not at least one of said first and second levels of the first and second cognitive conditions has reached at least one of first and second respective thresholds.

2. The system according to claim 1, further comprising a notification utility configured and operable to be responsive to said output data to generate a corresponding notification.

3. The system according to claim 1 or 2, further comprising a triggering utility configured and operable to be responsive to sensing data indicative of the changes in at least one of the first and second levels of the first and second cognitive conditions, to identify that said at least one of the first and second levels of the first and second cognitive conditions has reached a respective first or second value corresponding to a certain fraction of the at least one of the first and second thresholds, and activate said data processor and analyzer to analyze said measured data.

4. The system according to claim 1 or 2, further comprising a triggering utility configured and operable to be responsive to sensing data indicative of the changes in at least one of the first and second levels of the first and second cognitive conditions, to identify that said at least one of the first and second levels of the first and second cognitive conditions has reached a value corresponding to a certain fraction of the at least one of the first and second thresholds, and activate said sensing system to measure said motion and generate said measured data.

5. The system according to claim 3 or 4, wherein said sensing system is configured and operable to provide said sensing data to the triggering utility.

6. The system according to any one of claims 3 to 5, wherein said triggering utility is configured for data communication with at least one sensor to receive said sensing data therefrom and analyze said sensing data to identify whether the sensing data is indicative of that said at least one of the first and second levels of the first and second cognitive conditions has reached a value corresponding to a certain fraction of the at least one of the first and second thresholds.

7. The system according to claim 6, wherein said at least one sensor is configured and operable to collect data indicative of motion originated on at least one body part of the individual and generate motion data, said triggering utility being configured and operable to analyze the motion data by applying thereto at least one of first and second model-based processing associated with, respectively, at least one the first and second cognitive conditions, to extract said sensing data.

8. The system according to claim 6 or 7, wherein said at least one sensor is configured and operable to directly collect said sensing data from the individual.

9. The system according to any one of the preceding claims, wherein said certain activity comprises driving a vehicle, said at least first and second characteristic factors comprising fatigue and drunkenness factors, independently affecting the first and second cognitive conditions.

10. The system according to any one of claims 3 to 8, wherein said certain activity comprises driving a vehicle, said at least first and second characteristic factors comprising fatigue and drunkenness factors, independently affecting the first and second cognitive conditions, said first and second values of the certain fractions of the first and secondthresholds are, respectively, Level 5 of the Karolinska Sleepiness Scale and 0.03% Blood Alcohol Content.

11. The system according to claim 9 or 10, wherein said at least one sensor of the sensing system comprises at least one of the following: camera measuring the motion of individual's eyes and / or head; pressure sensor measuring the motion associated with individual's pressure on a gas pedal; a sensor measuring steering wheel angular motion; a pressure sensor measuring changes in individual's pressure on a seat chair.

12. The system according to any one of claims 9 to 10, wherein said at least one sensor of the sensing system comprises a sensor configured and operable to measure steering wheel angular motion, and said one or more features corresponding to a predetermined thresholding condition of said combined effect comprises at least one of the following: an intercept of a linear fit of measured steering wheel angle dynamical data, a difference between intercepts of linear and polynomial fits of measured steering wheel angle dynamical data, a median frequency of power spectra of measured steering wheel acceleration signals, an intercept of model fitted to estimate autocorrelation coefficients obtained from analysis of steering wheel angle dynamical data.

13. The system according to any one of claims 9 to 12, comprising a triggering utility configured and operable to be responsive to sensing data indicative of the changes in at least one of the first and second levels of the first and second cognitive conditions, to identify that said at least one of the first and second levels of the first and second cognitive conditions has reached a respective first or second value corresponding to a certain fraction of the at least one of the first and second thresholds, and activate at least one of said data processor and analyzer to analyze said measured data or said sensing system to provide the measured data, said sensing data being directly collected from the individual by at least one of a breathalyzer and a face camera.

14. A method for monitoring an operational cognitive state of an individual in relation to acting as a driver of a vehicle, the method comprising: measuring motion originated on at least one body part of the individual and generating corresponding measured data indicative of said motion, wherein changes in the corresponding measured data being sensitive to at least first and second levels of first and second cognitive conditions of an individual associated with, respectively, fatigueand drunkenness characteristic factors defining together a combined effect on the operational cognitive condition of an individual, said combined effect correlating with the changes in said first and second levels of the first and second cognitive conditions, said measured data being therefore indicative of changes in the operational cognitive condition; analyzing the measured data and upon identifying that said motion comprises one or more features corresponding to a predetermined thresholding condition of said combined effect, generating output data indicative thereof, thereby enabling detection of a critical operational cognitive state of the individual acting as the driver of vehicle, irrespective of whether or not at least one of said first and second levels of the first and second cognitive conditions has reached at least one of first and second respective thresholds in relation to, respectively, fatigue and drunkenness factors.

15. The method according to claim 14, further comprising in response to said output data, generating a corresponding notification to at least one of the individual and an authorized entity.

16. The method according to claim 14 or 15, further comprising providing sensing data indicative of the changes in at least one of the first and second levels of the first and second cognitive conditions, and analyzing the sensing data and upon identifying that said at least one of the first and second levels of the first and second cognitive conditions has reached a respective first or second value corresponding to a certain fraction of the at least one of the first and second thresholds, activating said analyzing of the measured data.

17. The method according to claim 14 or 15, further comprising providing sensing data indicative of the changes in at least one of the first and second levels of the first and second cognitive conditions, and analyzing the sensing data to identify that said at least one of the first and second levels of the first and second cognitive conditions has reached a value corresponding to a certain fraction of the at least one of the first and second thresholds, and activate said measuring of the motion and said generating of said measured data.

18. The method according to claim 16 or 17, wherein said providing of the sensing data comprises detecting motion originated on at least one body part of the individual and generating motion data, and applying to said motion data at least one of first and secondmodel-based processing associated with, respectively, at least one the first and second cognitive conditions, to extract said sensing data.

19. The method according to any one of claims 16 to 18, wherein said providing of the sensing data comprises directly collecting said sensing data from the individual.

20. The method according to claim 19, wherein said sensing data is directly collected from the individual by at least one of a breathalyzer and a face camera.

21. The method according to any one of claims 14 to 20, wherein the fatigue and drunkenness factors independently affect the first and second cognitive conditions, said first and second values of the certain fractions of the first and second thresholds are, respectively, Level 5 of the Karolinska Sleepiness Scale and 0.03% Blood Alcohol Content.

22. The method according to any one of claims 14 to 21, wherein said measuring of the motion is performed by at least one sensor comprising at least one of the following: camera measuring the motion of individual's eyes and / or head; pressure sensor measuring the motion associated with individual's pressure on a gas pedal; a sensor measuring steering wheel angular motion; a pressure sensor measuring changes in individual's pressure on a seat chair.

23. The method according to any one of claims 14 to 22, wherein said at least one sensor of the sensing system comprises a sensor configured and operable to measure steering wheel angular motion, and said one or more features corresponding to a predetermined thresholding condition of said combined effect comprises at least one of the following: an intercept of a linear fit of measured steering wheel angle dynamical data, a difference between intercepts of linear and polynomial fits of measured steering wheel angle dynamical data, a median frequency of power spectra of measured steering wheel acceleration signals, an intercept of model fitted to estimate autocorrelation coefficients obtained from analysis of steering wheel angle dynamical data.

24. A vehicle comprising: a sensing system comprising at least one motion sensor configured and operable to measure motion originated on at least one body part of the individual and generate corresponding measured data indicative of said motion, wherein changes in the corresponding measured data being sensitive to at least first and second levels of first andsecond cognitive conditions of an individual associated with, respectively, fatigue and drunkenness characteristic factors defining together a combined effect on the operational cognitive condition of a vehicle's driver, said combined effect correlating with the changes in said first and second levels of the first and second cognitive conditions, said measured data being therefore indicative of changes in the operational cognitive condition; a controller comprising a data processor and analyzer configured and operable to analyze the measured data and upon identifying that said motion comprises one or more features corresponding to a predetermined thresholding condition of said combined effect, generate output data indicative thereof, thereby enabling detection of a critical operational cognitive state of the driver, irrespective of whether or not at least one of said first and second levels of the first and second cognitive conditions has reached at least one of first and second respective thresholds.

25. A control system for use in monitoring an operational cognitive state of an individual during certain activity of the individual, the control system being configured and operable as a computerized system comprising: a data processor and analyzer configured and operable to analyze input measured data indicative of motion originated on at least one body part of the individual, to analyze the measured data and upon identifying that said motion comprises one or more features corresponding to a predetermined thresholding condition, generate output data indicative thereof; a triggering utility configured and operable to be responsive to sensing data indicative of changes in at least one of first and second levels of first and second cognitive conditions of an individual associated with, respectively, at least first and second characteristic factors defining together a combined effect on the operational cognitive condition of an individual, said combined effect correlating with the changes in said first and second levels of the first and second cognitive conditions; and upon identifying that said at least one of the first and second levels of the first and second cognitive conditions has reached a respective first or second value corresponding to a certain fraction of the at least one of the first and second thresholds, generate a triggering signal to activate said data processor and analyzer to analyze said measured data,said control system thereby enabling detection of a critical operational cognitive state of the individual during said certain activity, irrespective of whether or not at least one of said first and second levels of the first and second cognitive conditions has reached at least one of first and second respective thresholds.

26. The control system according to claim 25, wherein said sensing data comprises at least one of the following: data indicative of motion originated on at least one body part of the individual, and directly sensed one or more physiological parameters of the individual.

27. A vehicle comprising said control system according to claim 25 or 26, thereby enabling monitoring of the operational cognitive state of an individual in relation to acting as a driver of the vehicle.

Images