Methods, devices and computer-readable storage medium for monitoring a driver's activity level

Abstract

Method comprising: Receiving eye movement data from a sensor (104) that monitors eye movements of a driver of a vehicle (102) at a processor (512); Calculating an eye movement activity index by the processor (512) using a substantially real-time recursive analysis of the eye movement data; wherein calculating the eye movement activity index includes: Calculating a first current analog value at a time by applying a first recursive filter to the eye movement data; Calculating a second current analog value at the time by applying a second recursive filter to the first current analog value; Scaling the second current analog value to a normalized value corresponding to the eye movement activity index at that time; Calculating a driver low activity indicator by the processor (512) based on the eye movement activity index; and Perform a task based on the low activity indicator.

Description

AREA OF REVELATION

[0001] This disclosure relates generally to driver assistance in motor vehicles and, more particularly, to methods and apparatus for monitoring a driver's activity level. GENERAL STATE OF THE ART

[0002] People who are drowsy, fall asleep, or otherwise become inattentive while driving a vehicle pose a serious threat to their own safety and that of those around them. With the advent of advanced driver assistance systems and autonomous vehicles, safety concerns posed by inattentive drivers are somewhat mitigated. However, most vehicles are not fully autonomous. Furthermore, the driver's attention or awareness of the vehicle and / or the surrounding conditions may be desirable regardless of the level of driver assistance provided by a vehicle. Prior art can be found in DE 103 55 221 A1 and DE 199 83 911 T5. SUMMARY

[0003] Methods and apparatus for monitoring a driver's activity level are disclosed. An example method includes receiving eye movement data from a sensor that monitors eye movements of a driver of a vehicle at a processor. The method includes calculating, by the processor, an eye movement activity index using a substantially real-time recursive analysis of the eye movement data. The method further includes calculating, by the processor, a low activity indicator of the driver based on the eye movement activity index. The method also includes performing a task based on the low activity indicator.

[0004] In another example, a tangible computer-readable storage medium includes instructions that, when executed, cause a machine to receive at least eye movement data from a sensor that monitors eye movements of a driver of a vehicle. The instructions also cause the machine to calculate an eye movement activity index using a substantially real-time recursive analysis of the eye movement data. The instructions further cause the machine to calculate a driver low activity indicator based on the eye movement activity index. The instructions also cause the machine to perform a task based on the low activity indicator.

[0005] An exemplary system includes a sensor for monitoring eye movements of a driver of a vehicle. The system further includes memory and a processor that executes instructions stored in the memory to calculate an eye movement activity index using a substantially real-time recursive analysis of the eye movement data received from the sensor. The processor also calculates a low activity indicator of the driver based on the eye movement activity index. The processor also performs a task based on the low activity indicator. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 illustrates an exemplary activity level monitoring system for a vehicle. Fig. Figure 2 illustrates graphs showing example eye movement data and a corresponding eye movement activity index calculated from this data. Fig. Figure 3 illustrates graphs showing different example eye movement data and a corresponding eye movement activity index calculated from this data. Fig. 4 is a flowchart illustrating an exemplary method for implementing the exemplary activity level monitoring system of Fig. 1 illustrates. Fig. 5 is a block diagram of an example processor system configured to execute example machine-readable instructions implemented at least in part by the example method of Fig. 4 to illustrate the exemplary activity level monitoring system of Fig. 1 to be implemented. DETAILED DESCRIPTION

[0006] Example methods and apparatus implemented according to the teachings disclosed herein enable the determination of an activity level of a driver of a vehicle. As used herein, a driver's activity level refers to the driver's level of attention or inattention to the operation of the vehicle and / or the surrounding conditions. In some examples, the activity level may be based on a driver's drowsiness, which may be determined based on eye movement data associated with detecting eyelid movements of the driver (e.g., changes between an open and closed state of the eyes). In particular, as discussed below, driver eyelid movements may be tracked and analyzed in substantially real-time using a recursive approach to calculate an eye movement activity index.Calculating the eye movement activity index in this way enables the analysis and detection of eye abnormalities (e.g., eyelid movements other than normal blinking) over time to assess the possibility of reduced levels of concentration or attention by the driver due to drowsiness or fatigue. In some examples, the eye movement activity index is calculated based on double exponential smoothing of the eye movement data. In some examples, the constants in the recursive filters can be adjusted or configured depending on the desired sensitivity of detecting a potentially low activity state (e.g., drowsy) of the driver.

[0007] Additionally or alternatively, in some disclosed examples, a driver's activity level may be based on the driver's workload when performing tasks associated with operating the vehicle. As used herein, a driver's workload refers to the driver's visual, physical, and / or cognitive demands associated with the primary activity of driving (steering, braking, accelerating, etc.) as well as secondary activities (e.g., interacting with the dashboard, instrument panel, center console, and / or other aspects of the vehicle (e.g., adjusting a seat position, opening / closing a window, etc.)).In some examples, driver workload is inferred based on vehicle operating data received from one or more vehicle sensors that monitor vehicle operation, including driver-vehicle interactions, vehicle conditions, and / or environmental conditions associated with the vehicle. A driver with a relatively high workload (e.g., indicated by frequently stopping and starting in a high-traffic area) is less prone to becoming distracted than a driver with a relatively low workload (e.g., indicated by a vehicle maintaining speed on an open highway). Accordingly, in some examples, a driver's workload is analyzed from vehicle operating data to calculate a driver workload activity index, which may indicate the driver's activity level.In some examples, the Driver Stress Activity Index is based on a long-term analysis of individual drivers' driving behavior to account for different behaviors of each driver.

[0008] In some examples, a low activity indicator may be calculated using both the eye movement activity index and the workload-based activity index to enhance the characterization and / or detection of a driver in a relatively low activity state. In some examples, the activity indices are weighted and summed using appropriate weighting factors to obtain a final value for the driver's low activity indicator. In some examples, other metrics may also be aggregated with appropriate weighting into the final calculation of the low activity indicator, which indicates the driver's overall activity level. The weighting factors for each activity index may be set or configured depending on the specific circumstance.For example, the closer the vehicle's driver assistance state is to full autonomy, the less reliable vehicle operation data will be in indicating driver workload. Accordingly, in some examples, the driver workload activity index may be weighted lower than the eye movement activity index. However, if the vehicle's control is primarily manual (e.g., driver assistance features are either unavailable or disabled), the vehicle operation data will likely be more relevant, so the driver workload activity index may be weighted higher than the eye movement activity index.

[0009] In some examples, value ranges for the driver low activity indicator are configured into different categories to manage information delivery and machine interaction with the driver. When the low activity indicator is within a range (e.g., reaches a threshold) corresponding to a low driver activity state (e.g., drowsy and / or otherwise inattentive), the driver may be warned or otherwise reminded to focus on the vehicle and / or surrounding conditions. Additionally or alternatively, in some examples, the low activity indicator may be provided to a driver assistance system of the vehicle to assume control of some or all of the vehicle's operations.Furthermore, if the vehicle is already in an autonomous or semi-autonomous driver assistance state, the driver assistance system may adjust how the vehicle is controlled and / or interacts with the driver in response to the low activity indicator calculated for the driver.

[0010] As disclosed herein, a driver's activity indices and / or the resulting low activity indicator may be collected and stored over time for reference by the driver and / or third-party organizations. For example, such information may be collected for driver training or insurance purposes. Additionally or alternatively, such information may be shared substantially in real time with individuals other than the driver (e.g., parents, owners of a rental car company, etc.) to provide additional knowledge of the driver's activity level, where there may be a reason for third parties to know this information (e.g., novice drivers, older drivers with health conditions, etc.).

[0011] Now referring more closely to the figures, Fig. 1 shows an exemplary activity level monitoring system 100 for a vehicle 102. In the illustrated example, the activity level monitoring system 100 includes an eye movement sensor 104 and an eye movement analysis unit 106 to analyze eye movement data collected by the eye movement sensor 104. Furthermore, the activity level monitoring system 100 of the illustrated example includes one or more vehicle sensors 108 and a driver strain analysis unit 110 to analyze vehicle operating data collected by the vehicle sensors 108. In some examples, the activity level monitoring system 100 includes a driver assistance system 112 to provide assistance to a driver of the vehicle 102. In the illustrated example, the eye movement analysis unit 106, the driver strain analysis unit 110, and the driver assistance system 112 each provide inputs to a driver activity level analysis unit 114.The driver activity level analysis unit 114 may provide inputs to a driver interface module 116, a remote communication module 118, and / or a driver assistance feedback module 120.

[0012] More specifically, in some examples, the eye movement sensor 104 may be an image sensor (e.g., a camera) built into the vehicle 102 (e.g., the steering wheel, dashboard, sun visor, etc.) with a direct line of sight toward the driver's eyes. In other examples, the eye movement sensor 104 may be worn by the driver (e.g., integrated into wearable eye-tracking glasses). The eye movement sensor 104 generates eye movement data corresponding to detected changes in the state of the driver's eyelids (e.g., between an open eye state and a closed eye state) substantially in real time. A typical blink of an eye lasts between approximately 0.1 and 0.4 seconds. Thus, in some examples, the eye movement sensor 104 has a significantly faster sampling rate. In some examples, the eye movement sensor 104 has a sampling rate of approximately 200 Hz (0.005 seconds).

[0013] In illustrated example from Fig. 1, the eye movement analysis unit 106 receives and analyzes eye movement data received from the eye movement sensor 104. In some examples, the eye movement analysis unit 106 analyzes the eye movement data to detect and / or determine blinks that do not correspond to a normal blink by the driver. That is, in some examples, the eye movement analysis unit 106 determines blinks with a duration that exceeds the upper limit for the duration of a typical blink (e.g., 0.4 seconds). Blinks that extend beyond the duration of a typical blink may indicate fatigue in drivers who have difficulty keeping their eyes open.

[0014] A single blink lasting longer than normal may be insufficient to determine with certainty that a driver is fatigued or drowsy. In addition to increasing blink duration, the frequency of blinks also typically increases as a person becomes fatigued. Accordingly, in some examples, the eye movement analysis unit 106 tracks blinks detected by the eye movement sensor 104 over time using recursive analysis to calculate an eye movement activity index, which can be used to estimate the driver's activity state (e.g., level of drowsiness) substantially in real time. In some examples, the eye movement analysis unit 106 applies a recursive filter to the eye movement data received from the eye movement sensor 104, having the following form: sk=α1sk−1+(1−α1)xk where x kis the digital value (0 or 1) for the kth sample of the eye movement data provided by the eye movement sensor 104; s k is the current exponentially calculated analog value of the recursive analysis (in a range from 0 to 1); s k-1 the previously calculated s k and α1 is a tunable constant (in a range from 0 to 1).

[0015] In some areas, the issued s k from equation 1 a second recursive filter which has the following form: bk=α2bk−1+(1−α2)sk where b k is the current exponentially calculated analog value of the second recursive analysis (in a range from 0 to 1); b k-1 the previously calculated b kand α2 is a second tunable constant (ranging from 0 to 1). Thus, in some examples, the recursive analysis implements a double exponential smoothing approach applied to the eye movement data.

[0016] The issued b k from equation 2 can be scaled by a scaling factor (max(b k )) to calculate the final continuous eye movement activity index (Eye Movement - EM_Index) value as follows: EM_Indexk=bKmax(bk)

[0017] In some examples, the scaling factor max(b k ) a maximum value for b k (e.g., 0.75), determined over a characteristic period. In other examples, the scaling factor may be configured as a fixed value (e.g., 1), regardless of the maximum calculated value for b k . By dividing b k by max(b k), as shown in Equation 3, the output values ​​from Equation 2 are scaled or normalized to yield the eye movement activity index with a value in a range from 0 to 1.

[0018] The value of the eye movement activity index at a particular time indicates a driver activity state associated with a level of drowsiness or fatigue at a particular time. In some examples, the higher the eye movement activity index (e.g., closer to 1), the lower the driver activity state. Thus, in some examples, as described more fully below, the activity level monitoring system 100 may initiate certain tasks (e.g., alert the driver to become more focused and alert) when the eye movement activity index reaches (e.g., exceeds) a threshold corresponding to a low activity state (e.g., drowsy).

[0019] As mentioned above, the constants or smoothing factors α1 and α2 in Equations 1 and 2 can be configured or adjusted according to specific circumstances, based on the desired time constant for the recursive analysis and the desired sensitivity in detecting a low driver activity level. For example, typical values ​​for the constants might be α1 = 0.99 and α2 = 0.98. However, if faster responses are desired (to warn drivers earlier that they may be becoming drowsy), the constants can be set to lower values ​​(e.g., α1 = 0.9 and α2 = 0.95). In some examples, the individual values ​​can be configured based on the driver's skill and experience (e.g., lower constants for novice drivers), the time of day (e.g., lower constants late at night when people are more likely to be tired), or any other factor.In some examples, the individual values ​​for the constants may be determined based on the driver assistance state of the vehicle (e.g., lower constants for faster alerts when driver awareness or attention is critical to the operation of the driver assistance system 112).

[0020] Specific examples of calculated values ​​for the eye movement activity index of drivers with different levels of drowsiness are shown in diagrams 200, 300 of the Fig. 2 and Fig. 3. In particular, the upper diagram represents 200 Fig. 2 represents the eye movement activity index, which is compared with the values ​​shown in the lower diagram 202 from Fig. 2. Likewise, the upper diagram 300 represents Fig. 3 represents the eye movement activity index, which is compared with the values ​​shown in the lower diagram 302 from Fig. 3 eye movement data.

[0021] In the illustrated examples, the eyelid movements were sampled at 200 Hz (0.005 seconds per sample). Thus, for example, diagram 200 corresponds to a time period of approximately one hour. The eye movement data shown in the lower diagrams 202, 302 have a value of 1 when the eye movement sensor 104 detects an eyelid closure that lasts longer than a typical blink (e.g., longer than 0.4 seconds) and has a value of 0 at all other times. That is, in some examples, typical blinks are excluded as inputs to Equation 1 of the recursive analysis, so that the focus of the analysis remains on eye movements that are specifically associated with driver fatigue. In other words, the eye movements shown in the lower diagrams 202, 302 of the Fig. 2 and Fig. 3 correspond to a successive number of samples of the eye movement sensor 104 associated with a closed eye state over a threshold time period (e.g., 0.4 seconds, 0.5 seconds, etc.) without an intervening state change (e.g., eyelid opening).

[0022] Thus, as can be seen from a comparison of the Fig. 2 and Fig. 3, the figures in the lower diagram 202 of the Fig. 2, the eye movement data of a first driver show significantly more extended eyelid closures with a higher frequency than the number of eyelid closures detected in a second driver, as shown in the lower diagram 302 of the Fig. 3. The different attention levels (e.g. activity state) of each of the Fig. 2 and Fig. 3 monitored drivers can be determined by comparing the upper graphs 200, 300 of the Fig. 2 and Fig. 3. In particular Fig. 2 indicates that the driver is in a state of relatively low activity or lacking attention and concentration, as the eye movement activity index is relatively high (reaching a peak value of approximately 0.9). Fig. 3 associated drivers, on the other hand, are more focused and attentive, as indicated by the relatively low eye movement activity index, which never exceeds 0.1 (reaching a peak of approximately 0.055). In some examples, as described more fully below, one or more eye movement activity index thresholds may be set that, when reached (e.g., exceeded), trigger certain actions, such as warning the driver to wake up or otherwise become more attentive.

[0023] Returning to Fig. 1, the vehicle sensors 108 of the activity level monitoring system 100 may be any type of sensor to detect driver-vehicle interactions, vehicle conditions, and / or environmental conditions associated with the vehicle. Such information generated by the vehicle sensors 108 is collectively referred to herein as vehicle operating data. In some examples, the vehicle operating data is analyzed by the driver workload analysis unit 110 to generate a driver workload activity index similar to the eye movement activity index described above. That is, the driver workload activity index may be another metric indicating the driver's activity level determined substantially in real time.However, unlike the eye movement activity index, which indicates a driver's activity level based on an inferred level of fatigue or drowsiness associated with collected eye movement data, the driver strain activity index indicates the driver's activity level based on the amount of demands on the driver's attention determined from vehicle operating data.

[0024] In general, the higher the number and / or complexity of stimuli to which a driver responds (indicating a relatively high level of workload), the less likely a driver is to be in a low activity state. In particular, driver workload (and an associated activity level) can be inferred from various information collected by vehicle sensors 108, including fluctuations in speed, acceleration, braking, steering, instrument panel and / or center console interactions, the vehicle's location, the amount of surrounding traffic, current weather conditions, etc.

[0025] As a specific example, vehicle sensors 108 associated with electronic stability control systems, including an anti-lock braking system and / or traction control system, may respond to conditions when the vehicle 102 is operating at or beyond its operating limits. In some examples, more visual, physical, and / or cognitive attention (e.g., higher workload) may be expected to be required from a driver to maintain control of a vehicle as the vehicle 102 approaches its operating limits. Accordingly, in some examples, the driver workload analysis unit 110 calculates the driver workload activity index based on how close a driver is to the operating limits of the vehicle 102.In some examples, the driver strain activity index may be scaled from 0 to 1, with values ​​closer to 0 indicating relatively high levels of driver concentration and activity, and values ​​closer to 1 indicating relatively low levels of driver activity (attention). In this way, the driver strain activity index may be compared and / or aggregated with the eye movement activity index, as described more fully below. Thus, in some examples, a relatively high strain determined when the vehicle 102 is operating near its limit results in a relatively low driver strain activity index. Further details on determining the limit and calculating an associated index are described in U.S. Patent 8,914,192, issued December 16, 2014, which is hereby incorporated by reference in its entirety.

[0026] As another example, more visual, physical, and / or cognitive attention (e.g., higher workload) may be expected from a driver when traffic and / or the vehicle's route involves frequent changes in vehicle speed (e.g., changes in accelerator pedal position and / or brake pedal use) and / or steering (e.g., lane changes, turns, switchbacks). Accordingly, in some examples, the driver workload analysis unit 110 calculates the driver workload activity index based on the amount of driver control actions. In some examples, the driver workload activity index decreases with increasing frequency, amount, duration, and / or variance of driver control actions to indicate a higher driver activity state (i.e., higher concentration).Further details on determining driver control actions and calculating an associated index are described in US Patent 8,914,192, which is already included above.

[0027] In other examples, more visual, physical, and / or cognitive attention (e.g., higher workload) may be expected from a driver as the number, frequency, and / or duration of the driver's interactions with the instrument panel and / or other vehicle interfaces increases. The interactions may be touch-activated and / or voice-activated. Specific example interactions include the driver using the windshield wiper controls, climate controls, volume controls, turn signals, window controls, power seat controls, the navigation system, etc. Accordingly, in some examples, the driver workload analysis unit 110 calculates the driver workload activity index based on vehicle operating data indicative of such driver-vehicle interactions. In some examples, an increase in driver-vehicle interactions corresponds to an increase in inferred driver workload.As explained above, increases in estimated driver strain correspond to lower values ​​of the Driver Strain Activity Index (indicating higher levels of driver concentration). Further details on determining driver-vehicle interactions and calculating an associated index are described in U.S. Patent 8,914,192, which is incorporated above.

[0028] In other examples, more visual, physical, and / or cognitive attention (e.g., higher workload) may be required from a driver as the forward distance between the driver's vehicle 102 and another vehicle (or other object) in front of the vehicle 102 decreases. Accordingly, in some examples, the driver workload analysis unit 110 calculates the driver workload activity index based on the forward distance of the vehicle 102, where a shorter forward distance corresponds to an increase in the driver's workload and thus a lower value of the driver workload activity index (indicating relatively high concentration levels). Further details on determining a forward distance and calculating an associated index are described in U.S. Patent 8,914,192, incorporated above.

[0029] In some examples, more than one of the boundary, driver control action, driver-vehicle interaction, and / or lead distance factors described above may be combined to determine the driver stress activity index. Further, other factors may be considered in addition to or instead of those set forth above. The driver stress activity index based on vehicle operating data collected from the vehicle sensors 108 associated with any of these various factors allows for an estimate of a driver's activity level that is independent of the eye movement activity index set forth above. As described further below, in some examples, both of these metrics are combined to calculate an overall low driver activity indicator.In this way, a more robust assessment of the driver is possible to take more situations into account and correctly detect when a driver enters a low activity state, requiring the driver to be warned and / or other actions to be initiated.

[0030] As in the illustrated example of Fig. 1, the example activity level monitoring system 100 includes the driver assistance system 112. In some examples, the driver assistance system 112 may operate the vehicle 102 in various driver assistance states associated with varying levels of autonomous control of the vehicle 102. In some examples, the driver assistance states of the vehicle 102 vary from a fully manual state (when the driver assistance system 112 is effectively inactive and the driver has full control of the vehicle) to a fully autonomous state (when the driver assistance system 112 operates the vehicle 102 based on the vehicle operating data received from the vehicle sensors 108 with little or no input from the driver).In some examples, there may also be one or more intermediate driver assistance states for the vehicle 102 associated with semi-autonomous control of the vehicle 102 by the driver assistance system 112.

[0031] The level of a driver's involvement in operating the vehicle 102 varies for each different machine assistance state of the vehicle 102. Thus, the importance of the driver's attention and / or awareness regarding the operation of the vehicle and the surrounding circumstances (i.e., the driver's activity level) may also vary depending on how much control of the vehicle 102 is accomplished by the driver assistance system 112.

[0032] In some examples, an indication of the driver assistance state of the vehicle 102 is provided by the driver assistance system 112 as an input to the driver activity level analysis unit 114. This input may be used by the driver activity level analysis unit 114 in conjunction with the eye movement activity index received from the eye movement analysis unit 106 and the driver strain activity index received from the driver strain analysis unit 110. In some examples, other measures of the driver's activity level may also be provided as inputs to the driver activity level analysis unit 114. In some examples, the driver activity level analysis unit 114 calculates an aggregate or overall driver low activity indicator. In some examples, the low activity indicator is scaled from 0 to 1, with higher values ​​corresponding to lower driver activity levels (e.g.,when the driver is drowsy or otherwise inattentive). Specifically, in some examples, the driver activity level analysis unit 114 calculates a weighted composite of the various activity metrics using the following formula:. LAI=∑i=1Nwiyi where LAI is the aggregated low driver activity indicator; N is the number of aggregated driver activity level indicators; y i the value of each key figure; and w i is the weighting factor assigned to each metric. For example, if the metrics considered include the Eye Movement Activity Index (EM_Index) and Driver Workload Activity Index (DW_Index) described above, the driver's overall Low Activity Indicator (LAI) can be expressed as follows: LAI=(EM_Index)w1+(DW_Index)w2 where w1 is the weighting factor associated with the eye movement activity index and w2 is the weighting factor associated with the driver strain activity index.

[0033] In some examples, each of the metrics may be weighted equally. In other examples, certain metrics may be weighted more heavily (e.g., higher) than others. Further, in some examples, the particular weighting factor assigned to the respective metrics may change under different circumstances. For example, different weights may be assigned to the respective metrics for different drivers (e.g., novice drivers, teen drivers, older drivers, etc.). In some examples, different weights may be assigned based on a time of day (e.g., the eye movement activity index may be weighted more heavily during late evening hours when there is a higher likelihood that the driver is tired).

[0034] In some examples, the value of the weighting factor assigned to each metric is determined based on the driver assistance state of the vehicle 102 provided by the driver assistance system 112. For example, if the vehicle 102 is in an autonomous driver assistance state, such that the driver assistance system 112 controls the speed, acceleration, braking, steering, etc. of the vehicle 102, feedback from the vehicle sensors 108 regarding these operations would not indicate increased demands on the driver's attention (e.g., strain). As such, the driver strain activity index calculated based on this data may not reflect the actual level of driver activity. Accordingly, in some examples, the eye movement activity index may be assigned a higher weight than the driver strain activity index.Accordingly, in some examples, the eye movement activity index may be assigned a higher weight than the driver strain activity index.

[0035] In some examples, the weighting factor for one of the metrics may be set to 1 and the weighting factor for other metrics may be set to 0, effectively associating the low driver activity indicator with a single metric. In some examples, a single metric may be designated as the low driver activity indicator without assigning the weighting factors according to Equation 4.

[0036] After calculating the low activity indicator, the driver activity level analysis unit 114 may provide the calculated low activity indicator to one or more of the driver interface module 116, the remote communication module 118, and / or the driver assistance feedback module 120. In some examples, the driver interface module 116 compares the low activity indicator to a threshold associated with a relatively low level of activity (e.g., a relatively high low activity indicator value). In some examples, the driver interface module 116 may perform certain tasks in response to the low activity indicator reaching (e.g., exceeding) the threshold. The tasks may include the driver interface module 116 generating an alert or reminder for the driver to awaken from a drowsy state and / or otherwise become more alert.In some such examples, the warning or reminder may be presented to the driver visually (e.g., via a screen or lights), audibly (e.g., via speakers), and / or haptically (e.g., via vibrations in the driver's seat, steering wheel, etc.). In some examples, different tasks (e.g., different types or durations of warnings) may be associated with different thresholds, each with a different value.

[0037] Similarly, the remote communication module 118 may compare the low activity indicator to a corresponding threshold associated with a relatively low level of activity and perform certain tasks when the threshold is reached. The threshold may be the same as or different from the threshold applied by the driver interface module 116. In some examples, the tasks may include the remote communication module 118 communicating an alert to an interested third party (e.g., parents of a teen driver, rental car owner, insurance company, etc.) about the driver's low level of activity. In some examples, the remote communication module 118 may communicate the low activity indicator to a remote third party regardless of a threshold so that the driver's activity level can be collected over time.

[0038] Likewise, the driver assistance feedback module 120 may compare the low activity indicator to a corresponding threshold associated with a relatively low level of activity and perform certain tasks when the threshold is reached. The threshold may be the same as or different from the threshold applied by either the driver interface module 116 or the remote communication module 118. In some examples, the tasks may include the driver assistance feedback module 120 providing feedback to the driver assistance system 112 to enable the driver assistance system 112 to adjust control of the vehicle 102 depending on the level of driver activity or attention represented by the low activity indicator.In some examples, the driver assistance system 112 may communicate with the driver interface module 116 to alert the driver based on the value of the low activity indicator and / or request feedback regarding the autonomous operation of the vehicle 102.

[0039] While an exemplary manner of implementing the activity level monitoring system 100 in Fig. 1, one or more of the Fig. 1 may be combined, divided, rearranged, omitted, eliminated, and / or implemented in any other manner. Furthermore, the exemplary eye movement sensor 104, the exemplary eye movement analysis unit 106, the exemplary vehicle sensors 108, the exemplary driver load analysis unit 110, the exemplary driver assistance system 112, the exemplary driver activity level analysis unit 114, the exemplary driver interface module 116, the exemplary remote communication module 118, the exemplary driver assistance feedback module 120, and / or more generally, the exemplary activity level monitoring system 100 may be Fig. 1 be implemented by hardware, software, firmware and / or any combination of hardware, software and / or firmware.Thus, for example, each of the example eye movement sensor 104, the example eye movement analysis unit 106, the example vehicle sensors 108, the example driver load analysis unit 110, the example driver assistance system 112, the example driver activity level analysis unit 114, the example driver interface module 116, the example remote communication module 118, the example driver assistance feedback module 120, and / or more generally, the example activity level monitoring system 100 could be implemented by one or more digital circuits, logic circuits, programmable processors, application specific integrated circuits (ASICs), programmable logic devices (PLDs), and / or field programmable logic devices (FPLDs).Where device or system claims of this patent read as covering only a software and / or firmware implementation, at least one of the example eye movement sensor 104, the example eye movement analysis unit 106, the example vehicle sensors 108, the example driver load analysis unit 110, the example driver assistance system 112, the example driver activity level analysis unit 114, the example driver interface module 116, the example remote communication module 118, and / or the example driver assistance feedback module 120 is / are hereby expressly defined to include a tangible computer-readable storage device or disk, such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blue Ray disk, etc., that stores the software and / or firmware. Further, the example activity level monitoring system 100 of FIG. Fig. 1 one or more elements, processes and / or devices in addition to or instead of those in Fig. 1 and / or may include more than one of any or all of the illustrated elements, processes, and devices.

[0040] A flowchart illustrating an exemplary method for implementing the exemplary activity level monitoring system 100 of Fig. 1 is shown in Fig. 4. In this example, the method may be implemented using machine-readable instructions comprising a program for execution by a processor, such as the processor 512 used in the exemplary processor platform 500 discussed below in conjunction with Fig. 5. The program may be embodied in software stored on a tangible computer-readable storage medium, such as a CD-ROM, a floppy disk, a hard disk, a digital versatile disk (DVD), a Blue Ray disk, or a memory associated with the processor 512, but alternatively, all of the program and / or portions thereof could be executed on a device other than the processor 512 and / or embodied in firmware or special-purpose hardware. Furthermore, although the exemplary program is described with respect to the Fig. Alternatively, many other methods may be used to implement the exemplary activity level monitoring system 100, as described in the flowchart illustrated in Figure 4. For example, the order of execution of the blocks may be changed and / or some of the described blocks may be changed, eliminated, and / or combined.

[0041] As mentioned above, the exemplary procedure can be Fig. 4 may be implemented using encoded instructions (e.g., computer- and / or machine-readable instructions) stored on a tangible computer-readable storage medium, such as a hard disk, flash memory, read-only memory (ROM), a compact disk (CD), a digital versatile disk (DVD), a cache, random access memory (RAM), and / or any other storage device or disk on which information is stored for any duration (e.g., for extended periods of time, permanently, temporarily, for temporarily buffering and / or caching the information). As used herein, the term tangible computer-readable storage medium is expressly defined to include any type of computer-readable storage device and / or disk and to exclude propagating signals and to exclude transmission media.As used herein, "tangible computer-readable storage medium" and "tangible machine-readable storage medium" are used interchangeably. Additionally or alternatively, the example processes of [the present invention] may be used. Fig. 4 may be implemented using encoded instructions (e.g., computer- and / or machine-readable instructions) stored on a non-transitory computer- and / or machine-readable medium, such as a hard disk, flash memory, read-only memory, compact disk, digital versatile disk, cache, random access memory, and / or any other storage device or disk on which information is stored for any duration (e.g., for extended periods of time, permanently, temporarily, for temporarily buffering and / or caching the information). As used herein, the term non-transitory computer-readable medium is expressly defined to include any type of computer-readable storage device and / or disk and to exclude propagating signals and to exclude transmission media.When the phrase "at least," as used herein, is used as the transitional term in a preamble of a patent claim, it is open-ended, just as the phrase "comprehensively" is open-ended.

[0042] The exemplary procedure from Fig. 4 begins at block 402, where the example eye movement sensor 104 tracks eye movements of a driver. The detected eye movements are used to generate eye movement data that is provided to the eye movement analysis unit 106. At block 404, the example eye movement analysis unit 106 calculates an eye movement activity index based on the eye movement data. In some examples, the calculation includes the application of recursive filters and a scaling factor, as described above in connection with Equations 1-3. At block 406, the example vehicle sensors 108 track the operation of the vehicle 102. The monitored operation of the vehicle generates vehicle operating data indicative of driver-vehicle interactions, states of the vehicle 102, and / or environmental conditions associated with the vehicle 102.At block 408, the example driver load analysis unit 110 calculates a driver load activity index based on the vehicle operating data.

[0043] At block 410, the example driver assistance system 112 determines the driver assistance state of the vehicle 102. At block 412, the example driver activity level analysis unit 114 calculates a low driver activity indicator. In some examples, the low activity indicator is a weighted composite of the eye movement activity index and the driver strain activity index. In some examples, the weighting factors associated with each activity index are based on the driver assistance state of the vehicle 102.

[0044] At block 414, the example driver activity level analysis unit 114 calculates whether the low activity indicator reaches a threshold. In some examples, the low activity indicator is scaled to have a value in a range of 0 to 1, with higher values ​​corresponding to a lower driver activity level. Accordingly, in some examples, the threshold is met when the low activity indicator is equal to or greater than the threshold. The threshold may have any suitable value (e.g., 0.5, 0.6, 0.75, 0.9, etc.). If the example driver activity level analysis unit 114 determines that the low activity indicator does not reach the threshold (i.e., is less than the threshold), control returns to block 402. If the low activity indicator does reach the threshold, control transfers to block 416.

[0045] At block 416, the example driver interface module 116 generates a warning and / or message for the driver. At block 418, the example remote communication module 118 communicates the low activity indicator to a remote third party. At block 420, the example driver assistance feedback module 120 provides the low activity indicator to the driver assistance system 112. In some examples, block 418 and / or block 420 may be implemented in response to the low activity indicator reaching a different threshold than in block 414. In some examples, block 418 and / or block 420 may be implemented regardless of the value of the low activity indicator. At block 422, the example method either continues by returning control to block 402 or the example method Fig. 4 ends.

[0046] Fig. 5 is a block diagram of an exemplary processor platform 500 capable of implementing the method of Fig. 4 to execute the activity level monitoring system 100 from Fig. 1. The processor platform 500 may be, for example, a server, a personal computer, a mobile device (e.g., a cell phone, a smartphone, a tablet such as an iPad™), a personal digital assistant (PDA), an Internet device, or any other type of computing device.

[0047] The processor platform 500 of the illustrated example includes a processor 512. In some examples, the processor 512 is configured to implement one or more of the example eye movement sensor 104, the example eye movement analysis unit 106, the example vehicle sensors 108, the example driver load analysis unit 110, the example driver assistance system 112, the example driver activity level analysis unit 114, the example driver interface module 116, the example remote communication module 118, and / or the example driver assistance feedback module 120. The processor 512 of the illustrated example is hardware. For example, the processor 512 may be implemented by one or more integrated circuits, logic circuits, microprocessors, or controllers from any desired family or manufacturer.

[0048] The processor 512 of the illustrated example includes a local memory 513 (e.g., a cache). The processor 512 of the illustrated example is in communication with a main memory, which includes a volatile memory 514 and a non-volatile memory 516, via a bus 518. The volatile memory 514 may be implemented by synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), RAMBUS dynamic random access memory (RDRAM), and / or any other type of random access memory device. The non-volatile memory 516 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 514, 516 is controlled by a memory controller.

[0049] The processor platform 500 of the illustrated example also includes an interface circuit 520. The interface circuit 520 may be implemented by any type of interface standard, such as an Ethernet interface, a Universal Serial Bus (USB), and / or a PCI Express interface.

[0050] In the illustrated example, one or more input devices 522 are connected to the interface circuit 520. The input device(s) 522 enable the user to input data and commands into the processor 512. The input device(s) 522 may be implemented, for example, by an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a trackpad, a trackball, an isopoint, and / or a voice recognition system.

[0051] One or more output devices 524 are also connected to the interface circuit 520 of the illustrated example. The output devices 524 may be implemented, for example, by display devices (e.g., a light-emitting diode (LED), an organic light-emitting diode (OLED), a liquid crystal display, a cathode ray tube (CRT) display, a touchscreen, a tactile output device, a light-emitting diode (LED), a printer, and / or a speaker). The interface circuit 520 of the illustrated example thus typically includes a graphics driver card, a graphics driver chip, or a graphics driver processor.

[0052] The interface circuit 520 of the illustrated example also includes a communication device, such as a transmitter, a receiver, a transceiver, a modem, and / or a network interface card, to enable data exchange with external machines (e.g., computing devices of any type) over a network 526 (e.g., an Ethernet connection, a Digital Subscriber Line (DSL), a telephone line, a coaxial cable, a cellular telephone system, etc.).

[0053] The processor platform 500 of the illustrated example also includes one or more mass storage devices 528 for storing software and / or data. Examples of such mass storage devices 528 include floppy disk drives, hard disk drives, CD drives, Blu-ray disk drives, RAID systems, and DVD drives.

[0054] Coded instructions 532 for implementing the procedure from Fig.4 may be stored in the mass storage device 528, in the volatile memory 514, in the non-volatile memory 516, and / or on a removable tangible computer-readable storage medium, such as a CD or DVD.

[0055] From the foregoing, it should be appreciated that the methods, apparatus, and articles of manufacture disclosed above enable the characterization and / or detection of a driver who is drowsy, inattentive, or otherwise in a low activity state based on a substantially real-time recursive analysis of the driver's eye movements. Furthermore, the constants used in the recursive analysis may be adjusted or configured to provide more precise and / or time-accurate monitoring depending on the particular circumstances in which the teachings disclosed herein are implemented. Furthermore, the eye movement activity index resulting from such recursive analysis may be combined with other activity level metrics (e.g., driver strain) that are appropriately weighted to enhance the determination of a driver's activity level.Weighting different metrics in this way further facilitates the configuration and / or tailoring of driver activity assessments depending on the specific circumstances. Such specific characterization of a driver enables the provision of warnings and / or information to the driver in a relevant and timely manner to enhance the driving experience. Additionally, information about the driver's activity level can be provided to remote third parties in essentially real-time and / or feedback can be provided to a driver assistance system of the vehicle to adjust vehicle operation.

[0056] Although certain exemplary methods, apparatus, and articles of manufacture have been disclosed herein, the scope of this patent is not so limited. Rather, this patent covers all methods, apparatus, and articles of manufacture that reasonably fall within the scope of this patent.

Claims

[1] Method comprising: Receiving eye movement data from a sensor (104) that monitors eye movements of a driver of a vehicle (102) at a processor (512); Calculating an eye movement activity index by the processor (512) using a substantially real-time recursive analysis of the eye movement data; wherein calculating the eye movement activity index includes: Calculating a first current analog value at a time by applying a first recursive filter to the eye movement data; Calculating a second current analog value at the time by applying a second recursive filter to the first current analog value; Scaling the second current analog value to a normalized value corresponding to the eye movement activity index at that time; Calculating a driver low activity indicator by the processor (512) based on the eye movement activity index; and Perform a task based on the low activity indicator. [2] The method of claim 1, further comprising performing the task in response to the low activity indicator reaching a threshold. [3] The method of claim 1, wherein the task includes providing a warning to the driver. [4] The method of claim 1, wherein the task includes providing the low activity indicator to a driver assistance system (112) of the vehicle (102). [5] The method of claim 1, wherein the task includes communicating the low activity indicator to a remote third party. [6] The method of claim 1, wherein the first recursive filter uses a first constant and the second recursive filter uses a second constant, wherein the first and second constants are adjusted based on a desired sensitivity of the eye movement activity index in detecting a potentially low activity state of the driver. [7] The method of claim 1, further comprising calculating the low activity indicator based on the eye movement activity index and a second activity metric, wherein the eye movement activity index is weighted by a first weighting factor, wherein the second activity metric is weighted by a second weighting factor. [8] The method of claim 7, further comprising adjusting the first and second weighting factors based on a driver assistance state of the vehicle (102). [9] The method of claim 7, wherein the second activity metric is a driver load activity index, the driver load activity index indicating an activity level of the driver inferred from an estimated load of the driver. [10] A tangible computer-readable storage medium containing instructions which, when executed, cause a machine to do at least: receive eye movement data from a sensor (104) that monitors eye movements of a driver of a vehicle (102); calculate an eye movement activity index using a substantially real-time recursive analysis of the eye movement data; wherein calculating the eye movement activity index comprises: Calculating a first current analog value at a time by applying a first recursive filter to the eye movement data; Calculating a second current analog value at the time by applying a second recursive filter to the first current analog value; Scaling the second current analog value to a normalized value corresponding to the eye movement activity index at that time; to calculate a driver low activity indicator based on the eye movement activity index; and perform a task based on the low activity indicator. [11] The storage medium of claim 10, wherein the instructions, when executed, further cause the machine to calculate the low activity indicator based on the eye movement activity index and a second activity metric, wherein the eye movement activity index is weighted by a first weighting factor, wherein the second activity metric is weighted by a second weighting factor. [12] The storage medium of claim 11, wherein the instructions, when executed, further cause the machine to adjust the first and second weighting factors based on a driver assistance state of the vehicle (102). [13] System comprising: a sensor (104) for monitoring eye movements of a driver of a vehicle (102); a memory; and a processor (512) that executes instructions stored in memory to: calculate an eye movement activity index using a substantially real-time recursive analysis of the eye movement data received from the sensor (104); wherein calculating the eye movement activity index comprises: Calculating a first current analog value at a time by applying a first recursive filter to the eye movement data; Calculating a second current analog value at the time by applying a second recursive filter to the first current analog value; Scaling the second current analog value to a normalized value corresponding to the eye movement activity index at that time; to calculate a driver low activity indicator based on the eye movement activity index; and perform a task based on the low activity indicator. [14] The system of claim 13, wherein the processor (512) performs the task in response to the low activity indicator reaching a threshold. [15] The system of claim 14, wherein the task includes providing a warning to the driver. [16] The system of claim 13, wherein the task includes providing the low activity indicator to a driver assistance system (112) of the vehicle (102). [17] The system of claim 13, wherein the task includes communicating the low activity indicator to a remote third party. [18] The system of claim 13, wherein the processor (512) calculates the low activity indicator based on the eye movement activity index and a second activity metric, wherein the eye movement activity index is weighted by a first weighting factor, wherein the second activity metric is weighted by a second weighting factor. [19] The system of claim 18, wherein the processor (512) sets the first and second weighting factors based on a driver assistance state of the vehicle (102).

Images