Method and system for quantitative evaluation of drowsiness based on behavioral annotations and consensus evaluation

Abstract

The present disclosure relates to a method and system for quantitatively evaluating human drowsiness based on behavioral annotations and consensus evaluation. The method includes receiving annotations from trained experts based on prerecorded video footage of a participant, determining a consensus annotation for behavioral events indicative of drowsiness, and converting these annotations into continuous behavioral metrics. These metrics are then consolidated into a unified drowsiness score by applying predetermined coefficients and summing the weighted contributions of each behavioral metric. The disclosed embodiments provide a technical solution for objectively quantifying drowsiness levels, which is particularly useful in fields such as automated driver state monitoring systems.

Description

METHOD AND SYSTEM FOR QUANTITATIVE EVALUATION OF DROWSINESSBASED ON BEHAVIORAL ANNOTATIONS AND CONSENSUS EVALUATIONTECHNICAL FIELD

[0001] The present disclosure relates generally to the field of human state detection and evaluation, and more particularly to a method and system for quantifying human drowsiness through behavioral observations and expert consensus evaluation.BACKGROUND

[0002] Human states, such as alertness or drowsiness, are often characterized by psychological and physiological parameters. Traditional approaches to defining and evaluating these states have been qualitative, relying on subjective self-assessment scales or expert interpretations of observable behaviors. For instance, individuals may be asked to rate their level of sleepiness using subjective scales like the Karolinska Sleepiness Scale (KSS), which can vary significantly based on individual perception and interpretation of the criteria. Such selfassessments are inherently subjective and can be influenced by the act of evaluation itself, potentially altering the state being measured.

[0003] In addition to self-assessment, expert behavioral evaluation has been employed to identify states of drowsiness. Trained experts may observe and interpret behavioral changes, such as increased frequency of blinking or yawning, as well as more subtle indicators like changes in facial expressions or involuntary movements. While these observations can provide valuable insights, they are subject to human error and the ambiguity inherent in interpreting complex human behaviors.

[0004] Objective factors, such as circadian rhythms and sleep deprivation, are known to influence states of alertness and drowsiness. These factors provide a more objective basis for evaluation but do not account for the real-time behavioral manifestations of drowsiness. Moreover, the integration of such objective factors with behavioral observations to provide a comprehensive and quantifiable measure of drowsiness remains a technical challenge.

[0005] The limitations of existing methods highlight the need for a more objective and quantifiable approach to evaluating human drowsiness. A method that reduces subjectivity and provides a standardized measure of drowsiness would be advantageous for various applications, including but not limited to, driver monitoring systems in the automotive industry, where accurate detection of driver fatigue may be advantageous.

[0006] The present disclosure seeks to address the aforementioned challenges by providing a technical solution that enables the quantitative evaluation of drowsiness based on behavioral annotations, consensus evaluation, and conversion of discrete behavioral events into continuous numerical values giving a differentiable time trajectory of drowsiness. The current disclosure aims to mitigate the subjectivity associated with current methods and provide a reliable and implementable way to establish numerical and differentiable ground truth for drowsiness, thereby enhancing the objectivity and accuracy of human state detection systems.SUMMARY

[0007] In one embodiment, a method for evaluating drowsiness in a participant comprises, receiving a plurality of annotations corresponding to a set of behavioral events indicative of drowsiness, wherein each annotation is associated with a respective timestamp and is generated by one of a plurality of trained experts based on analysis of prerecorded video footage of the participant. The method further comprises determining a consensus annotation for each of the set of behavioral events within a first temporal window from the plurality’ of annotations, to produce a set of consensus annotations corresponding to the set of behavioral events. Additionally, the method includes converting, within a second temporal window, the set of consensus annotations to a corresponding set of continuous behavioral metrics. The method also involves consolidating, within the second temporal window, the set of continuous behavioral metrics into a unified drowsiness score for the participant by calculating, for each behavioral metric, a weighted contribution to the unified drowsiness score by applying a predetermined coefficient, summing each weighted contribution of each behavioral metric to generate the unified drowsiness score for the second temporal window, and outputting the unified drowsiness score for the participant.

[0008] In another embodiment, a system for evaluating drowsiness in a participant is provided. The system comprises a processor and a non-transitoiy computer-readable medium storing instructions that, when executed by the processor, cause the system to receive a plurality of annotations corresponding to a set of behavioral events indicative of drowsiness, wherein each annotation is associated with a respective timestamp and is generated by one of a plurality of trained experts based on analysis of prerecorded video footage of the participant. The system is further configured to determine a consensus annotation for each of the set of behavioral events within a first temporal window from the plurality of annotations, to produce a set of consensus annotations corresponding to the set of behavioral events. The system is alsoadapted to convert, within a second temporal window, the set of consensus annotations to a corresponding set of continuous behavioral metrics; and consolidate, within the second temporal window, the set of continuous behavioral metrics into a unified drowsiness score for the participant by calculating, for each behavioral metric, a weighted contribution to the unified drowsiness score by applying a pre-determined coefficient, summing the weighted contribution of each behavioral metric to generate the unified drowsiness score for the second temporal window, and outputting the unified drowsiness score for the participant.

[0009] In yet another embodiment, a method for evaluating drowsiness in a participant over a duration of prerecorded video footage is disclosed. The method comprises receiving a plurality of annotations corresponding to a set of behavioral events indicative of drowsiness, wherein each annotation is associated with a respective timestamp and is generated by one of a plurality of trained experts based on analysis of the prerecorded video footage of the participant. The method further includes determining, within a plurality of first temporal windows spanning the duration of the prerecorded video footage, a consensus annotation for each of the set of behavioral events from the plurality of annotations, to produce a plurality of sets of consensus annotations corresponding to the set of behavioral events for each of the plurality of first temporal windows. The method also involves converting, within a plurality of second temporal windows corresponding to the plurality of first temporal windows, each set of consensus annotations to a corresponding set of continuous behavioral metrics, and consolidating, within each of the second temporal windows, the set of continuous behavioral metrics into a respective unified drowsiness score for the participant by calculating, for each behavioral metric within the set, a weighted contribution to the respective unified drowsiness score by applying a pre-determined coefficient; summing each weighted contribution of each behavioral metric within the set to generate the respective unified drowsiness score for each second temporal window, and outputting a time trajectory of drowsiness scores for the participant, the time trajectory7comprising the respective unified drow siness scores for each of the second temporal windows.

[0010] The disclosed embodiments provide a technical solution to the challenge of objectively quantifying drowsiness levels in participants, which is particularly useful in fields such as automated driver state monitoring systems. The methods and systems described herein leverage expert evaluations and consensus mechanisms to derive a quantifiable measure of sleepiness, thereby enhancing the reliability and objectivity of drowsiness assessments.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG. 1 is a schematic diagram illustrating an embodiment of a process for determining continuous drowsiness metrics from annotated video footage;

[0012] FIG. 2 is a block diagram depicting an annotation processing system;

[0013] FIG. 3 is a flowchart outlining a method for generating a unified drowsiness score based on expert annotations and consensus evaluation;

[0014] FIG. 4 is a flowchart of a method for determining consensus annotations from a plurality of expert annotations of drowsiness-related behavioral events;

[0015] FIG. 5 is a flowchart of a method for converting consensus annotations to continuous behavioral metrics for drowsiness evaluation;

[0016] FIG. 6 is a flowchart of a method for capturing expert annotations for drowsiness detection from video footage of a participant; and

[0017] FIG. 7 is a flowchart of a method for calculating a time trajectory of unified drowsiness scores from video footage annotations.

[0018] The drawings referred to here should not be understood as being drawn to scale unless specifically noted. Also, the drawings are often simplified and details or components omitted for clarity of presentation and explanation.DETAILED DESCRIPTION

[0019] Examples will be provided below for illustration. The descriptions of the various examples will be presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0020] The current disclosure provides an automated process for determining an objective and unified time trajectory' of drowsiness of a human subject. This enables creation of continuous and differentiable drowsiness metrics from discrete behavioral cues observed in video recordings, such as those obtained from eye tracker cameras during monotonous tasks. In some embodiments, the discrete behavioral events include eye closures, yawns, and head nods, within the video footage. The currently disclosed process aggregates annotations from a plurality of experts, which may be stored in CSV files as behavioral events with associated timestamps and durations. These events are subjected to a consensus evaluation that may operate in several modes, including OR, AND, and OR with a count threshold, over apredefined temporal window. In the OR mode, an event is included in the consensus annotation if at least one expert has identified it within the specified window. The AND mode requires all experts to agree on the event's occurrence for inclusion. The OR with count threshold mode includes events that are marked by a minimum number of experts, enhancing the robustness of the consensus. The result of this consensus evaluation is a unified markup that reflects a consensus view of the drowsy behavior events, which is used for further analysis.

[0021] Upon establishing the consensus annotations, the discrete events may be converted into continuous numerical metrics that provide a quantitative measure of drowsiness over time. This conversion determines metrics within fixed-duration windows, such as 30 or 120 seconds, throughout the video recording. The determination of continuous metrics may account for the real and synthetic timestamps of the events, as well as the window boundaries, to ensure accuracy. The continuous metrics include the number of events within a window, the asymmetric number of events, the duration of events, and the asymmetric duration, among others. These metrics are normalized by the window duration or the number of events to yield standardized and quantitative measures that can be used to assess the level of drowsiness across different time periods and conditions.

[0022] The current disclosure also teaches merging the continuous metrics into a unified drowsiness score for each temporal window, thereby providing a singular, quantifiable measure of the participant's drowsiness level. This unification is achieved by applying predetermined coefficients to each continuous metric, reflecting their relative contribution to the overall drowsiness state. The coefficients are empirically selected based on the significance of each behavioral event in indicating drowsiness. For instance, events such as closed eyes and head falls may carry a higher coefficient due to their strong correlation with drowsiness, while other events like mouth movement may be assigned a lower coefficient. Each continuous metric is multiplied by its respective coefficient, and the products are summed to generate the unified drowsiness score for the window. This score is then outputted, providing a singular, quantitative indication of the participant's drowsiness at that specific segment of the recording.

[0023] In one embodiment, a process for determining continuous drowsiness metrics from annotated video footage, as depicted in FIG. 1, may be performed by an annotation processing device, as shown in FIG. 2. The process for determining continuous drowsiness metrics from the annotated video footage may include one or more operations included in method 300, shown in FIG. 3, or 700, shown in FIG. 7. Methods 300 and 700 may determine consensus annotations by conducting one or more operations of method 400, shown in FIG. 4, and mayfurther convert the consensus annotations to continuous metrics by employing one or more operations of method 500, shown in FIG. 5. Further, a flowchart of a method for capturing expert annotations for drowsiness detection from video footage of a participant is illustrated in FIG. 6.

[0024] Referring to FIG. 1, a block diagram of a process 100 for generating a continuous behavioral metric based on expert annotations is shown. The process 100 begins with expert annotations 102, which comprises annotations from multiple experts, such as expert 104, expert 106, and expert 108, regarding discrete behavioral events indicative of drowsiness. These annotations are derived from the analysis of prerecorded video footage of a participant, where each expert annotates occurrences of a pre-determined set of behavioral events, such as blinking, yawning, and head nodding. The annotations may be made using various tools and guidelines to ensure consistency and accuracy. For instance, each of expert 104, expert 106, and expert 108 may annotate occurrences and durations of eye closures, yawning, and head nods by pressing a distinct key or button during occurrence of each respective behavioral event. Each expert's annotations are timestamped to facilitate subsequent evaluation. FIG. 1 shows each instance of an annotation of a discrete behavioral event indicated by an X, in a row of expert annotations 102 corresponding to one of expert 104, expert 106, and expert 108, and in a column of expert annotations 102 corresponding to a frame, or a plurality of contiguous frames, of the pre-recorded video footage.

[0025] The behavioral annotations produced by each of experts 104-108 is consolidated via consensus evaluation 110 into consensus annotations 114. The consensus evaluation 110 operates to convert the behavioral annotations provided by experts 104, 106, and 108 into a cohesive set of consensus annotations 114. The consensus evaluation 110 determines the consensus annotations 1 14 for each behavioral event within a defined first temporal window 112 from the plurality of annotations by identifying, for each behavioral event, instances of the event annotated by the experts within the first temporal window. A consensus rule is then applied to the identified instances, which specifies a concordance threshold that to be satisfied for an event to be included in the consensus annotations 114. For example, the consensus rule may include that a behavioral event be annotated by a majority of the experts to meet the threshold. In this manner, the consensus evaluation 110 ensures that the consensus annotations 114 reflect a level of agreement among the experts, thereby reducing the subjectivity' and potential for human error associated with individual annotations.

[0026] First temporal window 112 is a pre-determined time window for applying a consensus criterion or rule to expert annotations 102. This window may "slide" along the expert annotations to produce consensus annotations for each behavioral event of the pre-determined set of behavior events for a plurality of time windows. In one embodiment, the first temporal window 112 may span a duration of 3 seconds, aligning with the natural occurrence rate of drowsiness-related behaviors. In another embodiment, the duration of the first temporal window 112 may be adjusted based on the specific application or the sensitivity required for detecting drowsiness events.

[0027] Consensus annotations 114 are the result of the consensus evaluation 110. These annotations represent a standardized and consistent measure of the observed drowsiness- related behaviors. In one embodiment, the consensus annotations 114 may be further refined by applying machine learning algorithms to identify patterns and reduce noise in the data. In another embodiment, the consensus annotations 114 may be subjected to a quality control process to ensure that the consensus reached is representative of the participant's state.

[0028] The continuous metric determination 116 involves the conversion of consensus annotations 114 into a quantifiable and continuous scale, facilitating a nuanced analysis of drowsiness levels over time. In a specific embodiment, this process entails the determination of a duration or frequency of the annotated behavioral event type within the second temporal window 118. and normalizing this duration or frequency to a duration of the second temporal window 118 or an occurrence number of the event type within the second temporal window 118. The continuous metric determination 116 yields a continuous behavioral metric 120 for each of a plurality7of second temporal window s 118. In one embodiment, the second temporal window 118 may be longer than the first temporal window 112 to capture the cumulative effect of drowsiness behaviors over time. In another embodiment, multiple second temporal windows 118 may be used to track the progression of drowsiness in a participant over an extended period, such as during a long-duration task or monitoring session. By utilizing the specific windows in coordination as noted above, an efficient yet sufficiently accurate timing protocol can be provided that includes converting and transformation the acquired data to the behavioral metrics with efficient processing while still providing temporally relevancy yet reducing erratic determinations.

[0029] Continuous behavioral metric 120 is the output of the process 100, which provides a quantifiable and continuous measure of the behavior based on the expert consensus. Although FIG. 1 shows only one behavioral metric for simplicity, the current disclosure provides fordetermining a plurality of continuous behavior metrics in a manner similar to that shown in FIG. 1. In one embodiment, the continuous behavioral metric 120 may be represented as a drowsiness score that can be used to trigger alerts in a driver monitoring system. In another embodiment, the continuous behavioral metric 120 may be used in conjunction with other physiological or environmental data to provide a comprehensive assessment of a participant's state. The process 100 described herein offers a robust and reliable method for quantifying drowsiness based on behavioral observations and expert consensus, which can be applied in various fields, including automotive safety, clinical research, and occupational health monitoring.

[0030] Referring to FIG. 2, an annotation processing system 200 is depicted, in accordance with an exemplary embodiment. The annotation processing system 200 is configured to evaluate drowsiness in a participant by processing behavioral annotations derived from video footage. The system 200 may be particularly useful in applications such as driver monitoring systems, where accurate detection of drowsiness is advantageous.

[0031] The annotation processing system 200 includes an annotation processing device 202. The annotation processing device 202 may be incorporated into a larger monitoring system, such as a vehicle's driver monitoring system, or may operate as a standalone unit. In some embodiments, the annotation processing device 202 is communicably coupled to a video capture device 240, such as a camera system within a vehicle, which captures video footage of a participant for subsequent analysis.

[0032] The annotation processing device 202 comprises a processor 204 configured to execute machine-readable instructions stored in non-transitory memory 206. Processor 204 may be a single-core or multi-core processor, and the programs executed thereon may be configured for parallel or distributed processing. In some embodiments, the processor 204 may include individual components that are distributed throughout two or more devices, which may7be remotely located and / or configured for coordinated processing. In alternative embodiments, one or more aspects of the processor 204 may be virtualized and executed by remotely- accessible networked computing devices configured in a cloud computing configuration.

[0033] Non-transitory memory 206 stores several modules employed in the operation of the annotation processing system 200. The behavioral annotation module 208 is configured for processing and interpreting user annotations related to observed behaviors in video data. In one embodiment, the behavioral annotation module 208 may receive input from a plurality of trained experts who annotate instances of drowsiness-related behaviors, such as closed eyes orhead nods. In another embodiment, the behavioral annotation module 208 may utilize machine learning algorithms to automatically detect and annotate such behaviors in the video footage.

[0034] The consensus module 210 integrates multiple annotations to reach a consensus or reconcile differences in annotations among different annotators or sources. In one embodiment, the consensus module 210 may apply a rule-based approach to determine a consensus annotation within a first temporal window from the plurality of annotations. In another embodiment, the consensus module 210 may employ a machine learning model trained to recognize patterns in the annotations indicative of varying levels of drowsiness.

[0035] The continuous metric module 212 is adapted to convert the set of consensus annotations to a corresponding set of continuous behavioral metrics within a second temporal window. In one embodiment, the continuous metric module 212 calculates a number-based metric for each behavioral event, representing a count of occurrences normalized over the duration of the second temporal window. In another embodiment, the continuous metric module 212 calculates a duration-based metric, representing the total time duration of occurrences of the behavioral event normalized over the duration of the second temporal window.

[0036] Annotation data 214, stored within non-transitory memory7206, may include the raw annotations provided by the experts, the consensus annotations determined by the consensus module 210, and the continuous behavioral metrics generated by the continuous metric module 212. The annotation data 214 may be used to generate a unified drowsiness score for the participant.

[0037] User input device 250 allows users to interact with the annotation processing system 200, providing input and receiving feedback. In one embodiment, the user input device 250 may comprise a touchscreen interface through which users can manually annotate video footage. In another embodiment, the user input device 250 may include a keyboard and mouse, enabling users to navigate through the video footage and enter annotations.

[0038] Display device 230 presents visual information to the user, possibly including annotation data, video feeds, and system statuses. In some embodiments, the display device 230 may comprise a computer monitor that displays the video footage alongside the annotations and metrics. In alternative embodiments, the display device 230 may be a touchscreen integrated with the user input device 250, providing an interactive experience for the user. In some embodiments, display device 230 may located remotely from, and communicatively coupled to, annotation processing device 202.

[0039] The video capture device 240 captures video data that will be annotated by users or automatically by the system. In one embodiment, the video capture device 240 may be a camera installed within a vehicle to monitor the driver's state. In another embodiment, the video capture device 240 may be part of a laboratory setup for controlled observation studies related to drowsiness. In some embodiments, video capture device 240 may located remotely from, and communicatively coupled to, annotation processing device 202. In one example, annotation processing device 202 may be located in a first location, such as a data center, while video capture device 240 may be located in a second location, such as a vehicle cabin, or a laboratory.

[0040] In one example, annotation processing system 200 may employ video capture device 240 to acquire raw video data, which is then annotated by users through the user input device 250. The behavioral annotation module 208 processes these annotations, which are then integrated by the consensus module 210 to form a set of consensus annotations. The continuous metric module 212 converts the consensus annotations into continuous and differentiable metrics that are used to calculate the drowsiness score, which is may be displayed to the user via the display device 230.

[0041] It should be understood that the annotation processing system 200 shown in FIG. 2 is for illustration, not for limitation. Another appropriate annotation processing system may include more, fewer, or different components. The disclosed embodiments provide a technical solution for objectively quantifying drowsiness levels in participants, which is particularly useful in fields such as automated driver state monitoring systems. The methods and systems described herein leverage expert evaluations and consensus mechanisms to derive a quantifiable measure of sleepiness, thereby enhancing the reliability and objectivity of drowsiness assessments.

[0042] Referring to FIG. 3, a flow-chart of a method 300 for determining a unified drowsiness score based on video footage and expert annotations is shown. The method 300 may be employed by a drowsiness evaluation system to quantify the level of drowsiness in a participant by analyzing behavioral events indicative of drowsiness as annotated by trained experts.

[0043] At operation 302, the system receives video footage of a participant along with expert annotations corresponding to a set of behavioral events indicative of drowsiness. In one embodiment, the video footage may be captured using a camera system, which may include a color camera, a black-and-white camera, or a near-infrared camera. The expert annotationsmay be generated by a plurality of trained experts who analyze the prerecorded video footage to identify instances of drowsy behavior, such as closed eyes, head falls, and yawns. Each annotation is associated with a respective timestamp indicating the occurrence of the behavioral event within the video footage. In an alternative embodiment, the annotations may be received from a centralized repository where they have been previously stored after being generated by the experts.

[0044] At operation 304, the system determines consensus annotations for each of the set of behavioral events within a first temporal window from the plurality of annotations, as detailed in FIG. 4. One embodiment involves identifying instances of each behavioral event annotated by the experts within the first temporal window and selecting the event for inclusion in the set of consensus annotations if the number of concurrences of the identified instances meets or exceeds a threshold number. Another embodiment may include using a machine learning model trained to recognize patterns in the annotations indicative of varying levels of drowsiness to establish consensus annotations.

[0045] Operation 306 involves converting the set of consensus annotations to a corresponding set of continuous behavioral metrics, as described in more detail in the description of FIG. 5. In one embodiment, the system calculates a number-based metric for each behavioral event within a second temporal window, representing a count of occurrences of the behavioral event normalized over the duration of the second temporal window. An alternative embodiment may calculate a duration-based metric for each behavioral event, representing the total time duration of occurrences of the behavioral event normalized over the duration of the second temporal window.

[0046] At operation 308, the system calculates weighted contributions of each continuous behavioral metric to a unified drowsiness score. One embodiment may involve applying predetermined coefficients to each behavioral metric based on their physiological relevance to drowsiness. For example, closed eyes may receive a higher coefficient than mouth movements. In an additional embodiment, the processor is further configured to apply different coefficients to the behavioral events based on pre-determined assessments of the contribution of each behavioral event in the formation of sleepy states. This means that each behavioral event is evaluated for its correlation with the development of drowsiness, and coefficients are adjusted accordingly to reflect their significance in the overall drowsiness assessment. Another embodiment may use asymmetric continuous behavioral metrics, where the asymmetric duration for each behavioral event within the second temporal window is determined based onits proximity to the end of the window. This approach allows for a dynamic assessment of behavior as it relates to the development of drowsiness, with more recent behaviors potentially being given greater weight if they are deemed more indicative of an imminent sleepy state.

[0047] Operation 310 sums the weighted contributions to determine the unified drowsiness score. In one embodiment, the system sums each weighted contribution of each behavioral metric to generate the unified drowsiness score for the second temporal window. An alternative embodiment may involve summing the contributions using a weighted average approach, where the weights are adjusted based on the frequency and duration of the behavioral events.

[0048] At operation 312, the system outputs the unified drowsiness score. This may include displaying the score on a user interface for review by a sleep expert or transmitting the score to a driver monitoring system in the automotive industry for real-time fatigue assessment. In another embodiment, the unified drowsiness score may be stored in a database for longitudinal tracking of the participant's drowsiness levels over time. Following operation 312, method 300 may end.

[0049] The method 300 provides a technical solution for objectively quantifying drowsiness levels based on behavioral annotations and consensus evaluation. This method is particularly useful in fields such as automated driver state monitoring systems, where accurate detection of driver fatigue is advantageous. The disclosed embodiments leverage expert evaluations and consensus mechanisms to derive a quantifiable measure of sleepiness, thereby enhancing the reliability and objectivity of drowsiness assessments.

[0050] Referring to FIG. 4, a flow chart of a method 400 for determining consensus annotations from a plurality of expert annotations of behavioral events indicative of drowsiness is shown. The method 400 enables aggregating expert observations to derive a more objective and quantifiable measure of drowsiness by addressing inconsistencies between human annotators.

[0051] At 402, the method 400 commences by receiving a plurality of annotations for behavioral events indicative of drowsiness. Each annotation is associated with a respective timestamp and is generated by one of a plurality of trained experts based on analysis of prerecorded video footage of the participant. The annotations correspond to a set of behavioral events, such as semi-closed eyes, closed eyes, reseating, yawns, rubbing eyes, head falls, frequent eye closures, forcible blinks, mouth movement, and head shake. In one embodiment, the annotations are received from a system configured to analyze video footage, which marksdrowsiness behavior events. The annotations are stored in a non-transitory computer-readable medium for subsequent processing.

[0052] At 404, the method 400 includes identifying instances of each behavioral event within a first temporal window. This operation involves parsing the received annotations to locate instances of the predefined behavioral events that have been marked by the experts within a specified time frame. In one embodiment, the temporal window is set to a duration that aligns with the average complex sensorimotor reaction of a person.

[0053] At 406, the method 400 determines the consensus annotation for each of the set of behavioral events by applying a consensus rule. The consensus rule establishes a predetermined threshold number of occurrences required for each behavior type within the set of behavioral events to qualify for inclusion in the consensus annotations. The method counts the number of annotations corresponding to each behavior type within the first temporal window from among multiple such windows. Upon the count of annotations for a behavior type meeting or exceeding the established threshold number, that behavior type is included in the set of consensus annotations for first temporal window. In this way. behavior types with a predetermined level of concurrence among annotations are included in the consensus annotations, thereby enhancing the consistency and reliability' of the consensus annotations derived from the data.

[0054] At 408. the method 400 involves selecting behavioral events satisfying the consensus rule for inclusion in consensus annotations for the first temporal window. This operation includes the aggregation of instances that have met the concordance threshold into a set of consensus annotations. In one embodiment, the selection process involves merging the marked event sequences from a plurality of independent experts to form a unified representation of the discrete drowsiness behavioral events within the first temporal window.

[0055] At 410, the method 400 includes storing the set of consensus annotations in non- transitory memory'. The stored consensus annotations serve as a more consistent foundation for deriving continuous drowsiness metrics in further analysis or as input to a system for evaluating drowsiness. Following operation 410, method 400 may end.

[0056] Referring to FIG. 5, a flowchart of a method 500 for determining continuous drowsiness metrics from consensus annotations is shown. The method 500 enables quantifying human drowsiness based on behavioral observations and expert consensus evaluation, as time- continuous and differentiable numerical values.

[0057] At operation 502, the method 500 includes receiving a set of consensus annotations for behavioral events indicative of drowsiness. These annotations are derived from a plurality of trained experts who have analyzed prerecorded video footage of a participant. Each annotation is associated with a respective timestamp, reflecting the moment when a particular drowsiness- related behavior was observed by the expert. The consensus annotations are determined within a first temporal window from the plurality of annotations. In one embodiment, the consensus annotations may be determined by applying a threshold number of concurrences among the expert annotations. In another embodiment, a machine learning model trained to recognize patterns in the annotations indicative of vary ing levels of drowsiness may be utilized to determine the consensus annotations.

[0058] Proceeding to operation 504, the method 500 includes determining the window start frame and end frame for a second temporal window. This operation is establishes the temporal boundaries within which continuous drowsiness metrics will be calculated. In one embodiment, the start and end frames of the window may be determined by real timestamps from the file. In another embodiment, synthetic frame numbers and timestamps may be calculated in cases where data problems exist, such as missing, duplicated, or zeroed timestamps. The duration of the second temporal window may, in some embodiments, but larger than the duration of the first temporal window used to determine the consensus annotations. In some embodiments, the duration of the second temporal window may be in the range of 30 to 120 seconds.

[0059] At operation 506, the method 500 calculates the window duration in seconds based on the frame rate. In one example, the duration of one frame may be 1 / 60 seconds, and the duration of the second temporal window may be determined by multiplying a number of frames contained within the second temporal window by 1 / 60 to produce the duration in seconds of the second temporal window. In some embodiments, the window duration is calculated by subtracting the timestamp of the window start frame from the timestamp of the window end.

[0060] Operation 508 identifies the real boundaries of each behavioral event within the second temporal window. This operation is enables determining the extent to which each drowsiness- related event occurs within the window. The event can be entirely within the window7, partially enter the window, or completely cover and go beyond the window. In one embodiment, the real boundaries are determined by the frame numbers corresponding to the start and end of each event. In another embodiment, if the event extends beyond the window, the portion ofthe event within the second temporal window may be utilized for continuous metric calculation at operation 510.

[0061] At operation 510, the method 500 computes a continuous metric for each behavioral event within the second temporal window, thereby transforming discrete consensus annotations into a temporal continuum reflective of the subject's state. Recognizing that human states are processes rather than discrete points, this operation eschews binary event representations in favor of a more nuanced, time-windowed approach. The continuous metrics are derived through formulas that consider the frequency, duration, and temporal distribution of events, thereby encapsulating the dynamic nature of drowsiness. In one embodiment, the continuous metrics are calculated using a methodology that accounts for the number of events, their respective durations, and the overall window duration. In another embodiment, asymmetric continuous metrics are employed to differentially weight events based on their temporal occurrence within the second window, thereby enhancing the metric's sensitivity to the progression of drowsiness. Operation 510 may encompass a variety of sub-operations, including but not limited to operations 512, 514. and 516, each tailored to optimize the metric's efficacy in capturing the gradations between low, medium, and high levels of drowsiness. This customization acknowledges the non-linear relationship between drowsiness events and their window ed representations, as well as the context-dependent effectiveness of different window conversion types for the same set of metrics. In one embodiment, the continuous metrics are calculated using formulas that take into account the number of events, the duration of events, and the window duration. In another embodiment, asymmetric continuous metrics are calculated for events to give more weight to events based on their occurrence within the second temporal window . Operation 510 may include one, all, or a subset of operations 512. 514, and 516, described below.

[0062] Operation 512 calculates a number-based metric for each behavioral event normalized over the window duration. This metric represents the count of occurrences of the behavioral event within the second temporal window normalized by the window's duration. In one embodiment, the number-based metric is calculated by dividing the number of events by the window duration. In another embodiment, the number-based metric may be adjusted to account for asymmetric occurrences of events within the window, e.g., by w eighting the event more heavily based on how far (in time) the occurrence of the event is from either the start, or end, of the second temporal window. In one embodiment, the number-based metric may be determined according to the below equation:

[0063] Where, in the above equation, N equals the number of events within the second temporal window, W represents the duration of the second temporal window. For an individual event, event_start and event_end define the frames at which the event begins and concludes, and event start in wnd indicates the frame at which an event commences within the second temporal window, and event_end_in_wnd denotes the frame at which the event finishes within the second temporal window (e.g., if the event extends beyond the start and end of the second temporal window, event start in wnd will equal the start frame of the second temporal window, and event_end_in_wnd will equal the end frame of the second temporal window).

[0064] In another embodiment, the number-based metric may be determined with an asymmetric weighting of events within the second temporal window, giving more weight to events occurring closer to an end frame of the second temporal window; as given by the below equation:

[0065] Where, in the above equation, N equals the number of events within the second temporal window , W represents the duration of the second temporal window, and x represents the current frame of the video. The variables wnd start and wnd end correspond to the starting and ending frames of the second temporal window, respectively. For an individual event, event_start and event_end define the frames at which the event begins and concludes, and event start in wnd indicates the frame at which an event commences within the second temporal window, and event end in wnd denotes the frame at which the event finishes within the second temporal windows

[0066] Operation 514 calculates a duration-based metric for each behavioral event normalized over the window duration. This metric represents the total time duration of occurrences of the behavioral event within the second temporal window; normalized by the window's duration. In one embodiment, the duration-based metric is calculated by summing the durations of all occurrences of the event within the window and dividing by the w indow duration. In another embodiment, the duration-based metric may be adjusted to account for partial events that only partially occur within the window. In one embodiment, the duration-based metric may be determined according to the below equation:

[0067] Where, in the above equation, N equals the number of events within the second temporal window, W represents the duration of the second temporal window, and F is the duration of one frame, also measured in seconds. For an individual event, event_start_in_wTid indicates the frame at which an event commences within the second temporal window, and event_end_in_wnd denotes the frame at which the event finishes within the second temporal window.

[0068] In another embodiment, the duration-based metric may be determined with an asymmetric weighting of events within the second temporal window, giving more weight to events occurring closer to an end frame of the second temporal window, as given by the below equation:

[0069] Where, in the above equation. N equals the number of events within the second temporal window, W represents the duration of the second temporal window, and x represents the current frame of the video. F is the duration of one frame, also measured in seconds. The variables wnd_start and wnd_end correspond to the starting and ending frames of the second temporal window, respectively. For an individual event, event start in wnd indicates the frame at which an event commences within the second temporal window, and event end in nd denotes the frame at which the event finishes within the second temporal window.

[0070] Operation 516 calculates a ratio-based metric for each behavioral event relative to the occurrence count. This metric represents the total time duration of occurrences of the behavioral event in relation to the count of occurrences within the second temporal window . In one embodiment, the ratio-based metric is calculated by dividing the duration-based metric by the number-based metric. In another embodiment, the ratio-based metric may be calculated according to the below equation:

[0071] Where, in the above equation, N equals the number of events within the second temporal window, and F is the duration of one frame, also measured in seconds. For an individual event, event_start and event_end define the frames at which the event begins andconcludes and event start in wnd indicates the frame at which an event commences within the second temporal window, and event_end_in_wnd denotes the frame at which the event finishes within the second temporal window.

[0072] In another embodiment, the ratio-based metric may be determined with an asymmetric weighting of events within the second temporal window, giving more weight to events occurring closer to an end frame of the second temporal window, as given by the below equation:

[0073] Where, in the above equation, N equals the number of events within the second temporal window, x represents the current frame of the video, and F is the duration of one frame, also measured in seconds. The variables wnd start and wnd end correspond to the starting and ending frames of the second temporal window, respectively. For an individual event, event_start and event_end define the frames at which the event begins and concludes and event start in wnd indicates the frame at which an event commences within the second temporal window, and event_end_in_wnd denotes the frame at which the event finishes within the second temporal window.

[0074] Finally, at operation 518, the method 500 stores the continuous behavioral metrics in non-transitory memory. This operation ensures that the calculated metrics are preserved for further analysis, reporting, or integration into a larger system for monitoring and evaluating drowsiness. In one embodiment, the continuous behavioral metrics are stored in a CSV file format, with each metric being associated with its respective window and event type. In another embodiment, the metrics may be stored in a database system that allows for efficient retrieval and analysis of the data.

[0075] The method 500, as described, provides a comprehensive and systematic approach to converting discrete behavioral events into continuous numerical metrics. The approach of method 500 offers a technical solution to the challenge of objectively quantifying drowsiness levels in participants, and converts the consensus drowsiness annotations into a form usable by downstream applications which rely on continuous and numerical data (e.g., machine learning models or other automated fatigue monitoring systems).

[0076] Referring to FIG. 6. a flowchart of a method 600 for annotating behavioral events indicative of drowsiness in a participant using video footage, is shown. The method 600 enables consistent evaluation of drowsiness by analyzing behavioral events indicative ofdrowsiness as observed in video footage of a participant. The method 600 leverages the expertise of trained annotators to identify and mark drowsiness -related behaviors, which are then used by one or more systems and methods disclosed herein to generate a unified drowsiness score for the participant.

[0077] At operation 602, the method 600 includes acquiring video footage of a participant for the purpose of drowsiness detection. This video footage is acquired from a source, such as a camera system, which may include color, black-and-white, or near-infrared cameras. In one embodiment, the video footage may be captured using an eye tracker camera system during monotonous tasks to ensure that the behaviors associated with drowsiness are likely to manifest. In another embodiment, the video footage may be stored in a centralized repository where it can be accessed for subsequent analysis.

[0078] Proceeding to operation 604, method 600 includes retrieving a predetermined list or database of behavioral events that are indicative of drowsiness. These behavioral events may include, but are not limited to, eye closures, yawns, head nods, and other facial or body movements that have been empirically associated with drowsiness. In one embodiment, the list of behavioral events is derived from a comprehensive ethogram that categorizes various drowsiness-related behaviors with associated hotkeys for annotation purposes. In another embodiment, the list may be dynamically updated based on ongoing research and findings in the field of drowsiness detection.

[0079] At operation 606, an annotation session is initiated for multiple trained experts. These experts are typically sleep specialists or individuals with experience in recognizing drowsiness-related behaviors. In some embodiments, the annotation session may be conducted using a software platform, such as BORIS (Behavioral Observation Research Interactive Software), which allows for the systematic marking of behavioral events as they occur in the video footage. In one embodiment, the annotation session is configured to accommodate two or more experts to ensure that the annotations are robust and consistent. In another embodiment, the experts may annotate the video footage independently without interaction to prevent bias and to maintain the integrity of the annotations.

[0080] Operation 608 involves capturing annotations from the experts corresponding to the behavioral events indicative of drow siness. The experts view the video footage and mark the occurrence of each predetermined behavioral event using the software platform. Each annotation is associated with a respective timestamp and behavioral event type. In one embodiment, the annotations are captured with an accuracy of 1-2 seconds, aligning with theaverage complex sensorimotor reaction time of a person. In another embodiment, the annotations may be stored in a format that allows for easy retrieval and analysis, such as CSV files.

[0081] At operation 610, the annotations are aligned with the corresponding unique timestamps assigned to each frame of the video footage. This synchronization process ensures that the marked behavioral events are accurately represented in the context of the video sequence. The events are saved in conjunction with the specific timestamps of the video frames, maintaining the integrity of the labeling process and congruence between the labels and video data. This alignment is enables the subsequent utilization of the labeled data.

[0082] Referring to FIG. 7, a flowchart of a method 700 for determining a unified drowsiness score based on behavioral events is shown. The method 700 may be employed by a drowsiness evaluation system to quantify the level of drowsiness in a participant by analyzing behavioral events indicative of drowsiness as annotated by trained experts.

[0083] At operation 702, the system receives a plurality of annotations for behavioral events indicative of drowsiness, wherein each annotation is associated with a respective timestamp and is generated by one of a plurality of trained experts based on analysis of prerecorded video footage of the participant. The annotations correspond to a set of behavioral events, such as semi-closed eyes, closed eyes, reseating, yawns, rubbing eyes, head falls, frequent eye closures, forcible blinks, mouth movement, and head shake. In one embodiment, the annotations are received from a system configured to analyze video footage, which marks drowsiness behavior events. The annotations are stored in a non-transitory computer-readable medium for subsequent processing.

[0084] At operation 704, the system determines consensus annotations for each of the set of behavioral events within a plurality of first temporal windows from the plurality of annotations. This operation involves parsing the received annotations to locate instances of the predefined behavioral events that have been marked by the experts within specified time frames. The first temporal windows may be established over the full duration of annotations, by. in some embodiments, using a sliding window approach, where each window is advanced incrementally over the timeline of the annotations. In one embodiment, the temporal windows may slide in one-second increments, allowing for a fine-grained analysis of the consensus among experts over time. The consensus rule applied within each first temporal window specifies a concordance threshold that must be satisfied by the identified instances for a behavioral event to be considered for inclusion in the consensus annotations. In oneembodiment, the consensus rule requires that the number of concurrences of the identified instances meets or exceeds a threshold number. This operation increases consistency among the annotations, thereby enhancing the reliability of the consensus annotations. The result is a series of consensus annotations that reflect a dynamic and temporal consensus view of the drowsy behavior events, which is used for further analysis.

[0085] Proceeding to operation 706. the system converts the set of consensus annotations to a corresponding set of continuous behavioral metrics within a plurality of second temporal windows. This operation transforms the discrete consensus annotations into continuous metrics that can be used to quantify the level of drowsiness over time. In one embodiment, the second temporal windows may correspond to the plurality of first temporal windows, ensuring that the continuous metrics are calculated over the same intervals used to establish consensus. In another embodiment, the second temporal windows may each be of greater duration than the first temporal windows. In one embodiment, the continuous metrics are calculated using formulas that take into account the number of events, the duration of events, and the window duration. In another embodiment, asymmetric continuous metrics are calculated for events to give more weight to events based on their occurrence be closer to a start of a second temporal window. The use of multiple second temporal windows allow for the creation of a time trajectory' of continuous metrics that reflects the participant's drowsiness over the entire duration of the annotations. This trajectory can then be used to assess changes in drowsiness levels and identify patterns or trends in the participant's state over time, or as ground truth data for training a deep learning model to automatically infer drowsiness from video footage.

[0086] Operation 708 involves calculating a weighted contribution for each behavioral metric using pre-determined coefficients. The coefficients are empirically selected based on the significance of each behavioral event in indicating drowsiness. For instance, events such as closed eyes and head falls may carry a higher coefficient due to their strong correlation with drowsiness, while other events like mouth movement may be assigned a lower coefficient. Each continuous metric is multiplied by its respective coefficient, and the products are summed to generate the unified drowsiness score for the window. This score is then outputted, providing a clear and objective indication of the participant's drowsiness at that specific segment of the recording.

[0087] At operation 710, the system sums the weighted contributions to determine the unified drowsiness score within each of the plurality of second temporal windows. In one embodiment, the system sums each weighted contribution of each behavioral metric to generate the unifieddrowsiness score for each of the plurality of second temporal windows. An alternative embodiment may involve summing the contributions using a weighted average approach, where the weights are adjusted based on the frequency and duration of the behavioral events.

[0088] At operation 712, the system outputs a time trajectory of unified drowsiness scores. In some embodiments, this may include displaying the score on a user interface for review by a sleep expert or transmitting the score to a driver monitoring system in the automotive industry for real-time fatigue assessment. In another embodiment, the unified drowsiness score may be stored in a database for longitudinal tracking of the participant's drowsiness levels over time. Following operation 712, method 700 may end.

[0089] Aspects of the present disclosure are described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / acts specified in the flowchart and / or block diagram block or blocks. Such processors may be, without limitation, general purpose processors, special-purpose processors, application-specific processors, or field-programmable processors.

[0090] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.

Claims

CLAIMS:

1. A method for evaluating drowsiness in a participant, the method comprising: receiving a plurality of annotations corresponding to a set of behavioral events indicative of drowsiness, wherein each annotation is associated with a respective timestamp and is generated by one of a plurality of trained experts based on analysis of prerecorded video footage of the participant; determining a consensus annotation for each of the set of behavioral events within a first temporal window from the plurality7of annotations, to produce a set of consensus annotations corresponding to the set of behavioral events; converting, within a second temporal window, the set of consensus annotations to a corresponding set of continuous behavioral metrics; and consolidating, within the second temporal window, the set of continuous behavior metrics into a unified drowsiness score for the participant by : calculating, for each behavioral metric, a weighted contribution to the unified drowsiness score by applying a pre-determined coefficient; summing each weighted contribution of each behavioral metric to generate the unified drowsiness score for the second temporal window; and outputting the unified drowsiness score for the participant.

2. The method according to claim 1, wherein determining the consensus annotation for each of the set of behavioral events within the first temporal window from the plurality of annotations, comprises: identifying, for each behavioral event, instances of the behavioral event annotated by the plurality of trained experts within the first temporal window; and selecting the behavioral event for inclusion in the set of consensus annotations for the first temporal window responsive to a number of concurrences of the identified instances meeting or exceeding a threshold number of concurrences.

3. The method according to claim 1, wherein determining the consensus annotation for each of the set of behavioral events comprises: establishing a threshold number of occurrences for each behavior type within the set of behavioral events;counting a number of annotations for each behavior type within the first temporal window; and including the behavior type in the set of consensus annotations for the first temporal window if the counted number of annotations for that behavior type meets or exceeds the threshold number.

4. The method of claim 1, wherein the set of behavioral events comprises at least one of the following: semi-closed eyes, closed eyes, reseating, yawns, rubbing eyes, head falls, frequent eye closures, forcible blinks, mouth movement, and head shake.

5. The method of claim 1. wherein converting the set of consensus annotations to the corresponding set of continuous behavioral metrics comprises calculating a number-based metric for each behavioral event within the second temporal window, the number-based metric representing a count of occurrences of the behavioral event normalized over a duration of the second temporal window.

6. The method of claim 1, wherein converting the set of consensus annotations to the corresponding set of continuous behavioral metrics comprises calculating a duration-based metric for each behavioral event within the second temporal window, the duration-based metric representing a total time duration of occurrences of the behavioral event normalized over a duration of the second temporal window.

7. The method of claim 1, wherein converting the set of consensus annotations to the corresponding set of continuous behavioral metrics comprises calculating a ratio-based metric for each behavioral event within the second temporal window, the ratio-based metric representing a total time duration of occurrences of the behavioral event in relation to a count of occurrences of the behavioral event within the second temporal window.

8. The method of claim 1, wherein the converting of the set of consensus annotations to the corresponding set of continuous behavioral metrics within the second temporal window further comprises: calculating asymmetric continuous behavioral metrics for each behavioral event, wherein the asymmetric continuous behavioral metrics are calculated by:determining an asymmetric duration for each behavioral event within the second temporal window, wherein the asymmetric duration is based on a proximity of the behavioral event to an end of the second temporal window; and calculating an asymmetric metric value for each behavioral event by dividing the asymmetric duration by a total number of frames in the second temporal window or by the number of occurrences of the behavioral event within the second temporal window.

9. A system for evaluating drowsiness in a participant, the system comprising: a processor; and a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the system to: receive a plurality of annotations corresponding to a set of behavioral events indicative of drowsiness, wherein each annotation is associated with a respective timestamp and is generated by one of a plurality of trained experts based on analysis of prerecorded video footage of the participant; determine a consensus annotation for each of the set of behavioral events within a first temporal window from the plurality of annotations, to produce a set of consensus annotations corresponding to the set of behavioral events; convert, within a second temporal window, the set of consensus annotations to a corresponding set of continuous behavioral metrics; and consolidate, within the second temporal window, the set of continuous behavioral metrics into a unified drowsiness score for the participant by: calculating, for each behavioral metric, a weighted contribution to the unified drowsiness score by applying a pre-determined coefficient; summing the weighted contribution of each behavioral metric to generate the unified drowsiness score for the second temporal window; and outputting the unified drowsiness score for the participant.

10. The system of claim 9, wherein the processor is further configured to receive and process input from a camera system comprising at least one of a color camera, a black-and- white camera, and a near-infrared camera, the camera system being operable to capture the prerecorded video footage of the participant for analysis by the plurality of trained experts.

11. The system of claim 9, wherein the processor is further configured to apply different coefficients to the behavioral events based on pre-determined assessments of a contribution of each behavioral event in formation of sleepy states.

12. The system of claim 9, wherein the processor is further configured to utilize a machine learning model to determine the consensus annotation from the plurality of annotations, the machine learning model being trained to recognize patterns in the annotations indicative of varying levels of drowsiness.

13. The system of claim 9, wherein the processor is further configured to generate a visual representation of the unified drowsiness score over time, the visual representation including a graphical display of weighted contributions of each behavioral event to the unified drowsiness score.

14. The system of claim 9, wherein the processor is further configured to display a annotation feedback to the plurality of trained experts, the feedback including a comparison of individual expert annotations against the consensus annotation.

15. A method for evaluating drowsiness in a participant over a duration of prerecorded video footage, the method comprising: receiving a plurality of annotations corresponding to a set of behavioral events indicative of drowsiness, wherein each annotation is associated with a respective timestamp and is generated by one of a plurality of trained experts based on analysis of the prerecorded video footage of the participant; determining, within a plurality of first temporal windows spanning the duration of the prerecorded video footage, a consensus annotation for each of the set of behavioral events from the plurality of annotations, to produce a plurality of sets of consensus annotations corresponding to the set of behavioral events for each of the plurality of first temporal windows; converting, within a plurality of second temporal windows corresponding to the plurality of first temporal windows, each set of consensus annotations to a corresponding set of continuous behavioral metrics; and consolidating, within each of the second temporal windows, the set of continuous behavioral metrics into a respective unified drowsiness score for the participant by:calculating, for each behavioral metric within the set, a weighted contribution to the respective unified drowsiness score by applying a pre-determined coefficient; summing each weighted contribution of each behavioral metric within the set to generate the respective unified drowsiness score for each second temporal window; and outputting a time trajectory of drowsiness scores for the participant, the time trajectory comprising the respective unified drowsiness score of each of the second temporal windows.

16. The method according to claim 15, wherein determining the consensus annotation for each of the set of behavioral events within the plurality of first temporal windows from the plurality of annotations, comprises: for a respective first temporal window' from the plurality of first temporal windows: identifying, for each behavioral event, instances of the event annotated by the plurality of trained experts w ithin the respective first temporal window; applying a consensus rule to the identified instances, wherein the consensus rule specifies a concordance threshold to be satisfied by the identified instances; and selecting the behavioral event for inclusion in the set of consensus annotations for the respective first temporal window when the identified instances meet or exceed the concordance threshold.

17. The method according to claim 15, wherein determining the consensus annotation for each of the set of behavioral events comprises: establishing a threshold number of occurrences for each behavior type within the set of behavioral events; counting the number of annotations for each behavior type within a respective first temporal w indow' of the plurality' of first temporal windows; and including the behavior type in the set of consensus annotations for the respective first temporal window if the counted number of annotations for that behavior type meets or exceeds the threshold number.

18. The method according to claim 15, wherein the set of behavioral events comprises at least one of the following: semi-closed eyes, closed eyes, reseating, yawns, rubbing eyes, head falls, frequent eye closures, forcible blinks, mouth movement, and head shake.

19. The method according to claim 15, wherein converting each set of consensus annotations to the corresponding set of continuous behavioral metrics within the plurality of second temporal windows comprises calculating a number-based metric for each behavioral event within each of the plurality of second temporal windows, the number-based metric representing a count of occurrences of the behavioral event normalized over a duration of a respective second temporal window from the plurality of second temporal windows.

20. The method according to claim 15, wherein converting each set of consensus annotations to a corresponding set of continuous behavioral metrics within the plurality of second temporal windows comprises calculating a duration-based metric for each behavioral event within a respective second temporal window of the plurality of second temporal windows, the duration-based metric representing a total time duration of occurrences of the behavioral event normalized over a duration of the respective second temporal window.

Images