Methods and systems for visual detection of seizures in animals using supervised machine learning

Abstract

The disclosure includes systems and methods for detecting a seizure in one or more animals. An example system includes a camera configured to capture video data of one or more animals, a trained machine-learning (ML) model configured to determine one or more aspects of a seizure in one or more animals based on the video data, and a controller configured to determine these aspects using the ML model during a given time period based on the video data. Some example devices may also include determining a univariate scale indicative of the one or more aspects of a seizure, determining the presence of a seizure during a given time interval, or determining the likelihood of an animal having a seizure during a given time interval. Methods of training an ML model to detect seizures in one or more animals are also provided.

Description

Methods and Systems for Visual Detection of Seizures in Animals Using Supervised Machine LearningCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Patent Application No. 63 / 652,739 filed May 29, 2024, the contents of which are incorporated herein by reference in their entirety.STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0002] This invention was made with government support under DA048634 and AG078530 awarded by the National Institutes of Health (NIH). The government has certain rights in the invention.BACKGROUND

[0003] Neurological seizures are caused by abnormally synchronous brain activity that can result in changes in muscle tone, such as twitching, stiffness, limpness, or rhythmic jerking. These behavioral manifestations are evident with visual inspection and the most widely used seizure scoring systems in preclinical models, such as the Racine scale in rodents, use these behavioral patterns in semi quantitative seizure intensity scores. However, visual inspection is time-consuming, low-throughput, and partially subjective, and there is a need for rigorously quantitative approaches that are scalable. Facilitating research into seizures in mice is relevant because of high homology between mouse and human neural circuits underlying seizures. Accordingly, there exists a need for methods and systems for visual detection of seizures in mice using supervised machine learning.SUMMARY

[0004] Embodiments of the present disclosure include methods and systems for visual detection of seizures in animals using supervised machine learning.

[0005] In a first aspect, a system for detecting aspects of a seizure in one or more animals is provided. The system includes at least one camera configured to capture video data of one or more animals. The system also includes a trained machine-learning (ML) model configured to determine one or more aspects of a seizure in one or more animals based on the video data recorded by the at least one camera. The system additionally includes a controllerhaving at least one processor and a memory configured to store program instructions. The processor is operable to execute the program instructions to carry out operations. The operations include receiving, by the at least one camera, video data indicative of a view of one or more animals. The operations also include determining, by the ML model, one or more aspects of a seizure in one or more animals during a given time period. The operations yet further include outputting, by the controller, the one or more aspects of a seizure in one or more animals during the given time period.

[0006] In a second aspect, a method of detecting aspects of a seizure in one or more animals is provided. The method includes capturing, by at least one camera, video data of one or more animals. The method also includes receiving, at a controller having at least one processor and a memory configured to store program instructions, the video data. The method yet further includes determining, by the controller, and by a trained ML model, one or more aspects of a seizure in one or more animals at a given time period. The method additionally includes outputting, by the controller, the one or more aspects of a seizure in one or more animals.

[0007] In a third aspect, a method of training a machine learning (ML) model to determine one or more aspects of a seizure in one or more animals is provided. The method includes receiving annotated video training data. The annotated video training data includes annotations corresponding to known aspects of a seizure in one or more animals during given time periods. The method also includes extracting, by a controller having at least one processor, pose-estimation and body segmentation data for one or more animals in the annotated video training data. The method yet further includes determining, by the controller, one or more classifiers that indicate at least one aspect of a seizure in one or more animals during a given time period. The method also includes training, by supervised machine learning, and by the one or more classifiers and the annotated video training data, a machine learning model to determine one or more aspects of a seizure in one or more animals during the given time period.

[0008] These as well as other aspects, advantages, and alternatives will become apparent to those of ordinary skill in the art by reading the following detailed description with reference where appropriate to the accompanying drawings. Further, it should be understood that the description provided in this summary section and elsewhere in this document is intended to illustrate the claimed subject matter by way of example and not by way of limitation.BRIEF DESCRIPTION OF THE FIGURES

[0009] Figure 1 is an illustration of a system for detecting seizures in one or more animals, according to an example embodiment.

[0010] Figure 2 is an illustration of a system for detecting seizures in one or more animals based on video data, according to an example embodiment.

[0011] Figure 3 is an illustration of a system for determining supervised or heuristic features of a seizure in one or more animals, according to an example embodiment.

[0012] Figure 4 is a chart illustrating a univariate scale for a seizure in one or more animals, according to an example embodiment.

[0013] Figure 5 is a chart illustrating the detection of seizures in one or more animals across one or more time bins, according to an example embodiment.

[0014] Figure 6 is an illustration of a method of detecting a seizure in one or more animals, according to an example embodiment.

[0015] Figure 7 is an illustration of a method of training a machine learning model to detect a seizure in one or more animals, according to an example embodiment.DETAILED DESCRIPTION

[0016] Examples of methods and systems are described herein. It should be understood that the words “exemplary,” “example,” and “illustrative,” are used herein to mean “serving as an example, instance, or illustration.” Any embodiment or feature described herein as “exemplary,” “example,” or “illustrative,” is not necessarily to be construed as preferred or advantageous over other embodiments or features. Further, the exemplary embodiments described herein are not meant to be limiting. It will be readily understood that certain aspects of the disclosed systems and methods can be arranged and combined in a wide variety of different configurations.

[0017] It should be understood that the below embodiments, and other embodiments described herein, are provided for explanatory purposes, and are not intended to be limiting.I. Overview

[0018] This disclosure relates to supervised machine learning approaches to develop automated classifiers to predict seizure severity directly from non-invasive video data. Using the pentylenetetrazole (PTZ)-induced seizure model in mice, video-only classifiers are trainedto predict ictal events, i.e. that occur during the middle phase of a seizure. These events are combined to predict a univariate seizure intensity for a given recording session, as well as timevarying seizure intensity scores. Systems and methods described herein beneficially allow for seizure events and overall intensity to be rigorously quantified directly from video of mice in various environments using supervised learning approaches. Accordingly, described embodiments enable high-throughput, non-invasive, and standardized seizure scoring for downstream applications such as neurogenetics and therapeutic discovery.

[0019] The community standard seizure severity score for pre-clinical models is theRacine scale, which is an ordinal scale from one to seven denoting progressively more pronounced behavioral manifestations of seizures, from whisker trembling (Racine score = 1) to a tonic-clonic seizure followed by tonic extension and possibly respiratory arrest or death (Racine score = 7). In order to enrich for seizure events in mice, seizures were induced using the convulsant pentylenetetrazole (PTZ), a gamma-aminobutyric acid (GABAA) receptor antagonist. It will be understood that other types of convulsants or medications for inducing specific rodent behaviors or conditions are possible and contemplated. By training robust, automated classifiers to predict seizure severity using video data alone, methods and systems described herein show that supervised methods can accurately detect specific seizure events and that these events can be combined to create a seizure intensity score.

[0020] To create robust training data with many seizures, mice were provided PTZ at varying doses. Mice were monitored in an open field during PTZ-induced seizures that were scored according to the Racine scale by an expert observer. This created human-annotated training data for seizure intensity.

[0021] In order to automate the seizure detection, a method was developed. First, a pose-estimation and body-segmentation time series was extracted from raw behavioral videos using deep neural network models. Second, from the extracted time series, classifiers were developed for six characteristic behavioral seizure manifestations: freezing (i.e., behavioral arrest, score of 1), Straub tail (score of 4), leg splaying (score of 4), circling (score of 5), side seizure and wild jumping (score of 6). Third, and finally, the classifier outputs were summarized as a set of features (e.g., “time spent in side seizure”) that were used in an ordinal regression model to predict seizure severity. To detect seizure intensity, two analyses were carried out. In a first analysis, seizure severity was predicted over the course of an entire session. In a second analysis, time-varying seizure intensity was predicted.

[0022] It is important to note that the described approach is not a surrogate for Racinescoring per se, in that not all Racine score features are classified. In particular, some of the semiological features in the Racine scale, such as facial twitches and whisker trembling, are extremely subtle and are not as easily detected by overhead video.

[0023] Instead, features that involve relatively large changes detectable at the level of posture or locomotion are used. This approach dramatically simplifies the video recording set up to be in line with those for widely adopted behavioral tasks and, in particular, does not require special equipment such as a depth camera. Furthermore, a computational ethology approach has the potential to enable detection of multiple behaviors simultaneously. It will be understood that systems and methods described here can be utilized to combine seizure detection with other behaviors such as homeostatic behaviors, social behaviors, and more to holistically phenotype an animal. Furthermore, unsupervised and supervised methods can be used in a complementary assay to provide rich partitioning of ictal and interictal behaviors. Taken together, the described systems and methods can robustly detect, from video data alone, the major semiological features of mouse behavioral seizures. The resulting models provide reproducible quantitative scores that may be beneficial for downstream applications, such as drug screening studies, quantitative genetics, and correlating seizure phenotypes with behavioral comorbidities.

[0024] In an example embodiment, a system for detecting seizures in one or more animals (e.g., mice, rats, and / or other animals) may include at least one camera configured to capture video data of one or more animals. The camera may be a visual spectrum camera, an infrared or ultraviolet camera, or one or more motion detectors or lidars. The field of view of the video data may include a top-down view, an open-field view, a cage, an enclosure, a view in a natural environment, or another view of the one or more animals. The system may also include a trained machine learning (ML) model configured to determine one or more aspects of a seizure in one or more animals based on the video data recorded by the at least one camera. The trained machine learning model may be trained on the video data recorded by the at least one camera, video data from another source, or other training data. The trained machine learning model may include a supervised machine learning model, such as a supervised latent variable model. The system may also include a controller having at least one processor and a memory configured to store program instructions. In such scenarios, the processor is operable to execute the program instructions to carry out operations. The operations include receiving, by the at least one camera, video data indicative of a view (e.g., an open-field view, a cage, an enclosure, etc.) of one or more animals. The operations also include determining, by the MLmodel, one or more aspects of a seizure in one or more animals during a given time period. The operations yet further include outputting, by the controller, the one or more aspects of a seizure in one or more animals during the given time period.

[0025] In a second example embodiment, a method of detecting aspects of a seizure in one or more animals may include capturing, by at least one camera configured to record video data, video data of one or more animals. The camera may be a visual spectrum camera, an infrared or ultraviolet camera, or one or more motion detectors or laser sensors / lidars. A field of view of the video data may include a top-down view, an open-field view, a cage, an enclosure, a view in a natural environment, or another view of the one or more animals. The method may also include receiving, at a controller having at least one processor and a memory configured to store program instructions, the video data. The video data may be segmented into one or more time intervals. The time intervals may be of equal length, or they may differ in length. The video data and / or time intervals may also be associated with one or more annotations that include information about the video data or the one or more animals. The method may also include determining, by the controller, and by a trained ML model, one or more aspects of a seizure in one or more animals during a given time period. The trained ML model may be a supervised machine learning model, such as a supervised latent variable model. The trained ML model may be trained on the video data, previous video data, or other data. The method may also include outputting, by the controller, the one or more aspects of a seizure in one or more animals during the given time period. Outputting the one or more aspects of a seizure in one or more animals may include annotating the video data with the one or more aspects of a seizure in one or more animals. The one or more aspects of a seizure may also be output to an external computer system, a network, or a graphical user interface.

[0026] In some example embodiments, a method of training a machine learning (ML) model to determine one or more aspects of a seizure in one or more animals is provided. The method includes receiving annotated video training data. The annotated video training data includes annotations. The annotations may include known aspects of a seizure in one or more animals at given time periods. The known aspects of a seizure may be determined by a trained ML model, by human observation, or by other means. The annotated video training data may also be divided into one or more time segments. The method may include extracting, by a controller having at least one processor, pose-estimation and body segmentation data for one or more animals in the video training data. The pose-estimation and body segmentation data may be determined by methods such as keypoint tracking, ellipse fit, or a trained ML model.The method also includes determining, by the controller, one or more classifiers that indicate at least one aspect of a seizure in one or more animals during a given time period. The one or more aspects of a seizure in one or more animals may comprise one or more classifiers. The classifiers may also comprise annotations, data, and other observations about the one or more animals. The method may also include training, by supervised machine learning, and by the one or more classifiers and the annotated video training data, a machine learning model to determine one or more aspects of a seizure in one or more animals during the given time period. The supervised machine learning process used may include use of a supervised latent variable model.II. Example Systems

[0027] Figure 1 is an illustration of a system 100 for detecting seizures in one or more animals 10, according to an example embodiment. Figure 1 illustrates a system 100 having a camera 110, a controller 150 having one or more processors 152 and memory 154, and a trained machine learning model 120. The camera 110 is configured to capture video data 112 of one or more animals 10. The video data 112 may be an “open-field” view of the one or more animals 10. The open-field view of the one or more animals 10 may be a top-down view of the one or more animals 10 in an environment. The environment may be a natural environment, or an environment such as a container, a cage (e.g., a “home cage”), an enclosure, and / or another type of controlled laboratory setup. Other camera views of the video data 112 are possible. The video data 112 may also be recorded using a depth camera, or other type of camera capable of 3D imaging. The video data 112 may also be captured using laser sensors (e.g., lidars), motion detectors, or other methods of non-visual spectrum imaging. In an example embodiment, the video data 112 can include captured video of multiple animals co-housed in the same home cage. In such scenarios, systems and methods described herein can detect if and when at least one of the animals experiences a seizure.

[0028] In some example embodiments, the video data 112 may be further processed into annotated video data 122, which may include one or more annotations to a behavior or characteristic displayed by an animal 10 in the video data 112. The annotated behaviors or characteristics may include, but are not limited to: motions of body parts of the animal 10, sounds emitted, eye movements, behavioral patterns, skin and / or fur markings, gait and posture measurements, body mass, flexibility, grooming behaviors, biological age, pain states, or other exterior indicia. The annotations may include machine-readable data containing informationabout the behaviors or appearances of the one or more animals 10. The annotations may also include human-readable language.

[0029] The annotated video data 122 may be processed by annotation and classification software 130 configured to annotate the video data 112. This annotation and classification software 130 may utilize various methods of video annotation to create the annotated video data 122. In an example embodiment, the annotation and classification software 130 utilizes image-processing techniques such as scale-invariant feature transform (SIFT) and edge detection to detect a plurality of features in frames of the video data 112 to create the annotated video data 122. In another example embodiment, the annotation and classification software 130 may use a neural network or other trained machine learning model to create the annotated video data 122. For example, a neural network may be used to create a segmentation mask of the animal to produce an ellipse fit of the animal 10 at each frame of the video data 112. This allows for the movements and behavior of the animal 10 to be tracked over the duration of the video data 112.

[0030] In some example embodiments, the annotated video data 122 may also track the movements of the one or more animals 10 in the environment. In some example embodiments, the speed and position of each of the one or more animals 10 may be recorded at different time periods across the video data 112. These data may then be used to define periods of motion and inactivity in each of the one or more animals 10. Distances and speeds traveled by the one or more animals 10 may also be calculated, and used to create the annotated video data 122.

[0031] While some embodiments described herein involve abstraction (e.g., animal pose estimation and / or keypoint estimation, and segmentation), such methods are not strictly necessary. For example, in some scenarios, supervised analysis of seizures can be performed on raw video. It will be understood that other ways to train a machine learning model to detect animal seizures are possible and contemplated.

[0032] The trained machine learning model 120 may be used to establish one or more aspects of a seizure 124 in the one or more animals 10. In some example embodiments, the one or more aspects of a seizure 124 may include one or more behaviors of the one or more animals 10 that are indicative of a seizure. Non-limiting examples of such behaviors are wild jumping, leg splaying, and repetitive circling movements. These behaviors may be established as one or more aspects of a seizure 124 by the trained machine learning model 120, which may be configured to recognize these behaviors. In some example embodiments, the one or more aspects of a seizure 124 may also include scores, ratings, or other classifiers of a seizure in theone or more animals 10. A classifier included in the one or more aspects of a seizure 124 may be determined by the trained machine learning model 120, and may comprise a number of observed characteristics that are indicative of a seizure. For example, a “Straub tail” classifier may be determined by the trained machine learning model 120 in the annotated video data 122, and it may be indicative of a pattern of movements in an animal wherein the tail stiffens, rises quickly, and jerks down. Other types of classifiers relating to animal seizure movements, such as side seizure, leg splaying, and wild jumping among others, are possible and contemplated.

[0033] In some example embodiments, the trained machine learning model 120 may predict the likelihood or severity of a seizure in the one or more animals 10. This prediction or likelihood may be included in the one or more aspects of a seizure 124. Based on the classifiers and behaviors in the annotated video data 122 the trained machine learning model 120 may be able to establish predictions of seizure events in the one or more animals 10. These predictions may be indicative of the likelihood of a presence of a seizure, or indicative of the potential severity of a seizure. In some example embodiments, the predictions may also be based on a time-segmented version of the annotated video data 122. In such scenarios, the annotated video data 122 may be divided into a plurality of time intervals. The trained machine learning model 120 may be able to predict a severity or presence of a seizure in the one or more animals 10 during a given time interval.

[0034] It will be understood that the trained machine learning model 120 can accept other information about the one or more animals 10 with the intent of providing more accurate seizure prediction and identification. For example, such other information can include nonvideo information, which may include, among other possibilities, genetic information, coat color, animal size, and other visually-identifiable traits. In some embodiments, the trained machine learning model 120 can be provided genetic information (e.g., mouse strain) about the animals housed in the home cage. As an example, mouse strains C57BL / 6J and C57BL / 6NJ can be used and the machine learning model 120 can provide predictions based at least in part on the strain of mouse. Other genetic strains of the one or more animals 10 are possible and contemplated.

[0035] The one or more aspects of a seizure 124 may be outputted by the trained machine learning model 120 as output 126. The output 126 may include data about the one or more aspects of a seizure 124, graphical representations of the established aspects, and / or predictions about a presence or severity of a seizure in the one or more animals 10. The output 126 may also include a further annotated version of the annotated video data 122 withannotations including the one or more aspects of a seizure 124 determined by the trained machine learning model 120.

[0036] Figure 2 is an illustration of a system for detecting seizures in one or more animals based on video data, according to an example embodiment. The system 200 may include a camera 110 and that is configured to capture video data 112 of one or more animals 202. The one or more animals 202 may be located in a laboratory environment or another environment in captivity. The one or more animals 202 may also be located their natural habitats or other environments. The one or more animals 202 may all be of one species or breed of animal, or they may be different species. The video data 112 may be provided to a controller 150 that is similar to the controller illustrated in Figure 1. Based on the video data 112 and / or annotated video data 122, a trained machine learning model 120 may be used to determine one or more aspects of a seizure 124. The controller may then, by a network connection, graphical display, or other means, create outputs 126 indicative of the one or more aspects of a seizure 124, the video data 112, and / or the annotated video data 122.

[0037] Figure 3 is an illustration of a system for determining supervised or heuristic features of a seizure in one or more animals, according to an example embodiment. The system 300 includes one or more animals 202, which may display behaviors, have external characteristics, and move around their environments. From these behaviors, characteristics, and movements, the system 300 may extract keypoint data 306, segmented video data 308, and ellipse data 310. These data may be defined by a supervised trained machine learning model, imaging techniques such as SIFT, and / or neural networks or other machine learning models.

[0038] The keypoint data 306 may be determined by placing one or more keypoints on different sections of the bodies of the one or more animals 202. These keypoints may be determined in the video data based on external features, markings placed onto the one or more animals 202, or other methods. By tracking the motion of the one or more keypoints over time, keypoint data 306 may be defined and used to track the movements, positions, and behaviors of the one or more animals 202. This can be used to create a “skeleton” for each of the one or more animals 202, allowing for easier tracking of the movements, positions, and behaviors of the one or more animals 202.

[0039] The segmented video data 308 may be determined by segmenting the recorded video data of the one or more animals 202 into time intervals. The time intervals may be equal time segments, or they may be of different lengths. In some embodiments, the time intervals can include durations of a few milliseconds, a few seconds, minutes, hours, or longer. It willbe understood that a duration of one or more time intervals can vary and / or be selected based on, for example, one or more specific seizure-related movements or behaviors.

[0040] The ellipse data 310 may be determined by producing an ellipse fit of each of the one or more animals 202 at each frame of the video data. The ellipse fit can be used for ellipse tracking of each of the one or more animals 202 over time. The ellipse data 310 may therefore include position, movement, and speed information for each of the one or more animals 202 in the environment, as well as position, movement, and speed information for individual movements of body parts of the one or more animals 202.

[0041] In some example embodiments, various measurements were derived from ellipse tracking of mice. For example, tracking was used to produce locomotor activity and anxiety features. Freezing behavior was heuristically derived by taking the average speed of the nose, base of head, and base of tail points at each frame, and finding periods of at least 3 seconds where the average speed of the mouse was less than 0.01 cm / sec. For detecting tight circling events, the angle the mouse is facing and a rate of change of the angle are measured. Once these changes in direction exceed 360 degrees in either direction, such a behavior is segmented out that as a circle event. The distance traveled by a mouse during that event is also measured. If the mouse has traveled more than 6 cm, then the mouse was walking around the arena. Distances less than 6 cm indicate a tight circling event. To adjust for the different durations that animals were observed during experiments with varying PTZ doses, each feature was standardized by the time (in minutes) that the animal was taken out of the experimental scenario.

[0042] To estimate the seizure intensity for a mouse at each time point of the assay, a hierarchical version of cumulative link models may be fit, namely, cumulative linear mixed models (CLMM). More specifically, repeated measures for each animal in each time bin are utilized by treating them as random effects and treating features such as tail jerk, leg splaying, side seizure, wild jumping, freezing, and circling as fixed effects. In some examples, leave- out-one-animal cross validation (LOOCV) may be used to validate an ordinal mixed effects model. In various embodiments, the estimated model coefficients for both fixed and random effects may be used to estimate the instantaneous per minute seizure intensity for the left-out animal. In such scenarios, the ordinal package may be used to fit the cumulative linear mixed model.

[0043] Additionally or alternatively, system 300 can utilize raw video 314 without applying substantial processing in order to determine seizure intensity, duration, or otheraspects of a seizure.

[0044] Figure 4 is a chart illustrating a univariate scale for a seizure in one or more animals, according to an example embodiment. In some example embodiments, the system may establish a univariate scale configured such that the univariate scale is indicative of the one or more aspects of a seizure 124. The chart 400 shows an example of the univariate scale.

[0045] The univariate scale may be configured such that it is a numerical scale indicative of all of the one or more aspects of a seizure 124. The univariate scale may also be indicative of the classifiers, features, or other data points used by the system to determine the one or more aspects of a seizure 124. In one example embodiment, the univariate seizure scale may provide a numerical representation of the intensity of a seizure, as shown in the chart 400. In some other example embodiments, the univariate seizure scale may group ranges of the numerical representations of the intensity of a seizure into one or more bins, shown by the no seizure bin 402, the low intensity bin 404, the medium intensity bin 406, and the high intensity bin 408.

[0046] In some example embodiments, the system may establish the univariate scale using a supervised latent variable model. A supervised latent variable model is a type of statistical model used in machine learning where some underlying, hidden variables (latent variables) are inferred from observed variables, and these inferred features are then used to predict an outcome variable, which is supervised (i.e., it has known target values). The supervised latent variable model may be trained based on the video data 112, the annotated video data 122, or other data collected by the system 100. The supervised latent variable model may also be trained based on data on seizures in animals from other sources. Training may comprise using an ordinal regression model, or another statistical mode, to estimate the probability of each of the one or more animals 202 having a seizure of different intensities. For example, each of the one or more animals 202 may be given a probability indicative of the probability that it has no seizure, a low intensity seizure, a medium intensity seizure, or a high intensity seizure. Based on the data, a trained supervised latent variable model may be created. In some examples, such trained models can beneficially uncover complex, hidden structures in the video data that may be highly predictive of seizure, enhancing both the contextual understanding and prediction capabilities of such behaviors.

[0047] Figure 5 is a chart illustrating the detection of seizures in one or more animals across one or more time bins, according to an example embodiment. In some example embodiments, the video data 112 or the annotated video data 122 may be divided into one ormore time bins 502, shown in the chart 500. The time bins 502 may be of equal length, or they may vary in length. In some example embodiments, the time bins 502 may be defined by the system 100 according to events or observations at specific times in the video data 112.

[0048] In some example embodiments, the system 100 may determine a presence of a seizure during one or more of the time bins 502. The chart 500 shows a spike in intensity in a plurality of seizure time bins 504. The seizure time bins 504 are time bins in which a seizure was present in the one or more animals 202. The system may determine the presence of a seizure by determining the one or more aspects of a seizure 124. In some example embodiments, the system 100 will also determine an intensity of a seizure in one or more of the time bins 502. These determinations may be included in the annotated video data 122, and they may associate the one or more aspects of a seizure 124 with time bins or frames of the video data 112.

[0049] Example systems and methods described herein are able to determine various aspects of animal seizure. First, embodiments include determining, during a given portion of video content, whether a seizure has occurred in at least one animal within the field of view. Second, examples can determine during which temporal segment (e.g., during which one minute time bin) the seizure has occurred in the animal and the intensity of the seizure. These two main functions can be conducted over long time periods (e.g., hours, days, weeks, or longer). In an example scenario, video information can be gathered from multiple mice in a home cage situation. Such information can be analyzed by example systems and methods so as to identify which mouse had a seizure, the intensity of the seizure, and the time at which the seizure occurred.III. Example Methods

[0050] Figure 6 is an illustration of a method of detecting a seizure in one or more animals, according to an example embodiment. In various embodiments, method 600 can be utilized to determine and output one or more aspects of a seizure 124 as described herein. The method 600 may include various blocks and / or steps. In various embodiments, the blocks and / or steps can be repeated, omitted, and / or carried out in different orders than that illustrated in Figure 6. In some examples, one or more blocks or steps of method 600 can be carried out, in full or in part, by system 100.

[0051] Block 602 includes capturing, by at least one camera (e.g., camera 110) configured to record video data, video data (e.g., video data 112) of one or more animals. Insome example embodiments, the camera may have an open-field view and / or a top-down view of the one or more animals. The camera may additionally or alternatively include a view of a cage, a portion of a cage, or another type of enclosure. In some other example embodiments, the camera may be configured to capture visual, infrared, ultraviolet, or other video data. The camera may also be a system of motion detectors or laser sensors configured to capture motion data of the one or more animals.

[0052] Block 604 includes receiving, at a controller (e.g., controller 150) having at least one processor (e.g., processor 152) and a memory (e.g., memory 154) configured to store program instructions, the video data. The video data may be digital video data. The video data may be captured by the camera detailed in block 602, or by another method.

[0053] Block 606 includes determining, by the controller, and by a trained ML model (e.g., trained machine learning model 120), one or more aspects of a seizure (e.g., one or more aspects of a seizure 124) in one or more animals during a given time period. The trained ML model may be trained based on the video data. The trained ML model may also be trained based on data collected from other sources. In some example embodiments, training is performing using a supervised latent variable model.

[0054] Block 608 includes outputting, by the controller, the one or more aspects of a seizure in one or more animals during the given time period. The one or more aspects of a seizure can be provided in a graphical format, such as a chart, line graph, or table. The one or more aspects of a seizure may also be associated with the video data captured by the camera, creating annotated video data (e.g., annotated video data 122). These outputs (e.g., outputs 126) may be provided to an external computing system, a network connection, or to another system. The outputs may also be provided to a machine learning model and used to train the machine learning model to determine one or more aspects of a seizure.

[0055] Figure 7 is an illustration of a method of training a machine learning model to detect a seizure in one or more animals, according to an example embodiment. In various embodiments, method 700 can be utilized to train a machine learning model so as to provide the trained machine learning model 120 as described herein. The method 700 may include various blocks and / or steps. In various embodiments, the blocks and / or steps can be repeated, omitted, and / or carried out in different orders than that illustrated in Figure 7.

[0056] Block 702 includes receiving annotated video training data, wherein the annotations comprise known aspects of a seizure (e.g., one or more aspects of a seizure 124) in one or more animals at given time periods. The annotated video training data may be theannotated video data 122. It may also be an annotated version of the video data 112. In some example embodiments, the annotated video training data may be from external sources, or based on video data previously captured from one or more animals.

[0057] Block 704 includes extracting, by a controller having at least one processor, pose-estimation and body segmentation data for one or more animals in the video training data. In some example embodiments, the pose-estimation and body segmentation data may be extracted by one or more deep neural network models configured to extract pose-estimation and body segmentation data for one or more animals in video data. The deep neural network models may be trained based on the video data of the one or more animals, or they may have been previously trained. In some other example embodiments, the pose-estimation and body segmentation data may be based on keypoint tracking of the bodies of the one or more animals. In other example embodiments, the pose-estimation and body segmentation data may be based on an ellipse fit of the one or more animals.

[0058] Block 706 includes determining, by the controller, one or more classifiers that indicate at least one aspect of a seizure in one or more animals during a given time period. These classifiers may be indicative of various behaviors, movements, actions, and physical conditions of the one or more animals. In some example embodiments, the classifiers may be used to determine one or more aspects of a seizure in one or more animals.

[0059] Block 708 includes training, by supervised machine learning, and by the one or more classifiers and the annotated video training data, a machine learning model to determine one or more aspects of a seizure in one or more animals during the given time period. In some example embodiments, training may be performed with a supervised latent variable model.

[0060] Additionally or alternatively, method 700 may include determining, based on the video training data, a representation of one or more animals. In other words, method 700 may include determining keypoints, two- or three-dimensional shapes, skeletons, and / or other representations of one or more animals in the video training data. Furthermore, block 708 may include training the machine learning model based on the determined representation of the one or more animals. It will be understood that other ways to train the machine learning model are possible and contemplated.IV. Conclusion

[0061] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying figures. In thefigures, similar symbols typically identify similar components, unless context indicates otherwise. The illustrative embodiments described in the detailed description, figures, and claims are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein. In various examples, extracting the pose-estimation for one or more animals need not be required in the context of method 700 and block 704.

[0062] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

CLAIMSWe claim:

1. A system for detecting aspects of a seizure in one or more animals, the system comprising: at least one camera configured to capture video data of one or more animals; a trained machine-learning (ML) model configured to determine one or more aspects of a seizure in one or more animals based on the video data recorded by the at least one camera; a controller having at least one processor and a memory configured to store program instructions, wherein the processor is operable to execute the program instructions to carry out operations, the operations comprising: receiving, by the at least one camera, video data indicative of a view of one or more animals; determining, by the ML model, one or more aspects of a seizure in one or more animals during a given time period; and outputting, by the controller, the one or more aspects of a seizure in one or more animals during the given time period.

2. The system of claim 1, wherein the operations further comprise: determining labeled physical behavior data of the one or more animals labeled by a trained machine learning model configured to label physical behavior data of an animal based on the video data; and transmitting, to the ML model, the labeled physical behavior data.

3. The system of claim 1, wherein the operations further comprise establishing a univariate scale, based on the one or more aspects of a seizure in the one or more animals during the given time period, wherein the univariate scale is indicative of the one or more aspects of a seizure.

4. The system of claim 3, wherein the univariate scale is established using a machine learning model trained by a supervised latent variable model based on the one or more aspects of a seizure in one or more animals.

5. The system of claim 1, wherein the video data is divided into a plurality of intervals indicative of a time period, wherein a seizure during a given time interval is indicated by at least one of the one or more aspects of a seizure in one or more animals.

6. The system of claim 5, wherein at least one of the one or more aspects of a seizure in one or more animals is indicative of an intensity of a seizure present during a given interval.

7. The system of claim 1, wherein the operations further comprise determining, based on the one or more aspects of a seizure in one or more animals during the given time period, a likelihood of a given animal having a seizure during the given time period.

8. The system of claim 7, wherein determining the likelihood of a given animal having a seizure during the given time period is based on ordinal regression of a continuous latent variable indicative of the one or more aspects of a seizure in one or more animals.

9. The system of claim 1, wherein the operations further comprise determining, based on the one or more aspects of a seizure in one or more animals during the given time period, one or more categories of seizure intensity assigned to each of the one or more animals.

10. The system of claim 1, wherein the video data is indicative of an open-field view, a cage view, or an enclosure view including one or more animals.

11. A method of detecting aspects of a seizure in one or more animals, the method comprising: capturing, by at least one camera, video data of one or more animals;receiving, at a controller having at least one processor and a memory configured to store program instructions, the video data; determining, by the controller, and by a trained ML model, one or more aspects of a seizure in one or more animals during a given time period; and outputting, by the controller, the one or more aspects of a seizure in one or more animals during the given time period.

12. The method of claim 11, further comprising determining, based on the video data and a trained machine learning model configured to label physical behavior of an animal, labeled physical behavior data of the one or more animals; and transmitting, to the ML model, the labeled physical behavior data.

13. The method of claim 11, further comprising establishing, based on the one or more aspects of a seizure in the one or more animals during a given time period, a univariate scale configured such that the univariate scale is indicative of the one or more aspects of a seizure.

14. The method of claim 13, wherein the univariate scale is established using a machine learning model trained by a supervised latent variable model based on the one or more aspects of a seizure in one or more animals.

15. The method of claim 11, further comprising dividing the video data into a plurality of intervals indicative of a time period, and wherein at least one of the one or more aspects of a seizure in one or more animals is indicative of a presence of a seizure during a given interval.

16. The method of claim 15, wherein at least one of the one or more aspects of a seizure in one or more animals is indicative of an intensity of a seizure present during a given interval.

17. The method of claim 11, further comprising determining, by the controller, based on the one or more aspects of a seizure in one or more animals during the given time period, a likelihood of a given animal having a seizure during the given time period.

18. A method of training a machine learning (ML) model to determine one or more aspects of a seizure in one or more animals, the method comprising: receiving annotated video training data, wherein the annotated video training data comprises annotations corresponding to known aspects of a seizure in one or more animals at given time periods; extracting, by a controller having at least one processor, pose-estimation and body segmentation data for one or more animals in the annotated video training data; determining, by the controller, one or more classifiers that indicate at least one aspect of a seizure in one or more animals during a given time period; and training, by supervised machine learning, and by the one or more classifiers and the annotated video training data, a supervised machine learning model to determine one or more aspects of a seizure in one or more animals during the given time period.

19. The method of claim 18, wherein the one or more classifiers comprise a univariate scale configured such that the univariate scale is indicative of the one or more classifiers, and wherein training the supervised machine learning model is based on the univariate scale.

20. The method of claim 18, wherein training the supervised machine learning model further comprises training based on 1) holistic phenotype data of at least one of the one or more animals; or 2) social behavior data of at least one of the one or more animals.

Images