Determining visual frailty index using machine learning models

Abstract

The systems and methods described herein provide techniques for determining a visual frailty score by processing video data of a subject. Various features may be used to determine the visual frailty score, including, but not limited to, spinal mobility features, gait measurements, behavioral features, and body composition data. The various features may be extracted from the video data using different techniques. One or more machine learning models may be used to process the various features to determine the visual frailty score.

Description

[Technical field]

[0001] Related Applications This application claims the benefit under 35 U.S.C. §119(e) of U.S. Provisional Application No. 63 / 187,892, filed May 12, 2021, the disclosure of which is incorporated herein by reference in its entirety.

[0002] The invention relates in some aspects to determining a subject's visual frailty index by processing video data using a machine learning model. Government support

[0003] This invention was made with government support under grants DA041668 and DA048634 awarded by the National Institute on Drug Abuse, and AG38070 awarded by the National Institute on Aging. The Government has certain rights in this invention. [Background technology]

[0004] Aging is the final process that affects all living systems. In contrast to chronological aging, biological aging occurs at different rates for different individuals. In humans, aging comes with increased health problems and mortality, but some individuals live long and healthy lives, while others die early from diseases and disorders. More precisely, heterogeneity in mortality risk and health status has been observed among individuals within age cohorts [Non-Patent Document 1, Non-Patent Document 2]. The concept of frailty has been used to quantify this phenomenon of heterogeneity and is defined as a state of increased vulnerability to adverse health outcomes [Non-Patent Document 3]. Identifying frailty is clinically important, since frail individuals have a higher risk of disease and disability, worse health outcomes due to the same disease, and even different symptoms of the same disease [Non-Patent Document 2].

[0005] The Frailty Index (FI) is a widely used approach to quantify frailty [Non-Patent Document 1] and is superior to other methods [Non-Patent Document 4]. In this method, individuals are scored for a set of multiple, age-related health impairments to generate a cumulative score. Each impairment must have the following characteristics: it must be health-related, its population must increase with age, and its population must not saturate too quickly [Non-Patent Document 5]. The presence and severity of each health impairment is scored as 0 if absent, 0.5 if partially present, or 1 if present. A compelling finding about the FI is that the exact health impairments scored may differ between the indices, but they still exhibit similar characteristics and utility [Non-Patent Document 5]. That is, two sufficiently large FIs with different numbers and selections of impairments show similar average rates of disability accumulation with age and the same maximum possible FI score. More importantly, both FIs robustly predict an individual's risk of adverse health outcomes, hospitalization, and death. This property of the FI is advantageous because it allows researchers to pull data from various large medical databases to support large-scale studies. It also suggests that frailty is a legitimate phenomenon and that the FI is a valid way to quantify it, given the complexity of aging. Not only do different people age at different rates, but they also age in different ways. That is, one person may have severe problems with mobility but a sharp memory, while another person may have a healthy heart but a weak immune system, and so on. Both may be equally frail, but this would only become evident if various health impairments were sampled. Indeed, the FI score outperforms other developed indices such as molecular marker tracking and frailty phenotyping in efficiently predicting mortality risk and health status [Non-Patent Documents 6-8]. Some FIs have been adapted for use in mice using both a variety of behavioral and physiological indices as index items [Non-Patent Documents 2, 4, 9], but there are no adequate methods to assess frailty and predict mortality risk and health in animal models and humans. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Mitnitsky A. et al., Scientific World Journal, 1, 323-36 (September 2001) [Non-Patent Document 2] Whitehead JC et al., Journal of Gerontology, Series A: Biomedical Sciences and Medical Sciences, 69, 621-632 (2014) [Non-Patent Document 3] Lockwood K et al., CMAJ 150, 489-495 (1994) [Non-Patent Document 4] Schultz MB et al., Nature Communications 11, 1-12 (2020) [Non-Patent Document 5] Searle SD et al., BMC Geriatrics 8, 24 (2008) [Non-Patent Document 6] Schultz M. B. et al., Nature Communications 11, 1-12 (2020) [Non-Patent Document 7] Kim S. et al., Geroscience 39, 83-92 (January 2017) [Non-Patent Document 8] Kojima G. et al., Age and Aging 47, 193-200, (2017) [Non-Patent Document 9] Parks R. et al., Journal of Gerontology, Series A, Biological and Medical Sciences, 67, 217-27 (March 2012) Summary of the Invention [Problem to be solved by the invention]

[0007] According to one aspect of the invention, a computer-implemented method is provided, the method including receiving video data representing video capturing movement of a subject, using the video data to determine a spinal mobility feature of the subject for a duration of the video, and processing at least the spinal mobility feature using at least one machine learning model to determine a visual frailty score for the subject. In some embodiments, determining the spinal mobility feature of the subject for a duration of the video includes determining a plurality of spinal measurements, each spinal measurement of the plurality of spinal measurements corresponding to one video frame of the video data, and using the plurality of spinal measurements to determine the spinal mobility feature. In some embodiments, determining the spinal mobility feature of the subject for a duration of the video includes determining, for each video frame of the video data, a first distance between the subject's head and the subject's tail, determining a second distance between the subject's mid-back and a midpoint between the head and tail, determining an angle formed between the subject's head, tail, and mid-back, and determining the spinal mobility feature of the video frame to include the first distance, the second distance, and the angle. In some embodiments, determining the spinal mobility characteristic of the subject during the duration of the video includes determining a distance between the mid-back of the subject and a midpoint between the subject's head and the subject's tail for each video frame of the video data. In some embodiments, the method also includes processing the video data using at least an additional machine learning model to determine posture estimation data that tracks a position of at least the subject's head, the subject's tail, and the subject's mid-back during the duration of the video, and using the posture estimation data to determine the spinal mobility characteristic. In some embodiments, the method also includes processing the video data to determine posture estimation data that tracks a position of at least 12 body parts of the subject during the duration of the video, using the posture estimation data to determine the characteristics of the subject, and processing the characteristics using at least one machine learning model to determine a visual frailty score.In some embodiments, the method also includes determining a physical characteristic of the subject, the physical characteristic corresponding to at least one of a length of the subject, a width of the subject, and a distance between the hind legs of the subject, and processing the physical characteristic using at least one machine learning model to determine a visual frailty score. In some embodiments, the method also includes determining a number of rear events that occur during the duration of the video, determining a rear stand length for each rear stand event, and processing the number of rear stand events that occur and the rear stand length for each rear stand event using at least one machine learning model to determine a visual frailty score. In some embodiments, the method also includes processing the video data using at least one machine learning model to determine ellipse fit data for the subject during the duration of the video, using the ellipse fit data to determine features of the subject, and processing the features using at least one machine learning model to determine a visual frailty score. In some embodiments, determining the subject's spinal mobility features during the duration of the video includes determining a first set of video frames representative of ambulatory movement by the subject, determining a first set of spinal mobility features for the first set of video frames, determining a second set of video frames representative of non-ambulatory movement by the subject, and determining a second set of spinal mobility features for the second set of video frames, wherein the spinal mobility features include the first set of spinal mobility features and the second set of spinal mobility features. In some embodiments, the first set of spinal mobility features corresponds to a distance between the subject's mid-back and a midpoint between the subject's head and tail, and the second set of spinal mobility features corresponds to an angle formed between the subject's head, tail, and mid-back. In some embodiments, the method also includes using the video data to determine gait measures for the subject during the duration of the video, and processing the gait features using at least one machine learning model to determine a visual frailty score for the subject.In some embodiments, the method also includes processing the video data to determine point data tracking movement of a set of a plurality of body parts of the subject during the duration of the video; using the point data to determine a plurality of stance phases and a corresponding plurality of swing phases represented in the video data; determining a plurality of stride intervals represented in the video data based on the plurality of stance phases and the plurality of swing phases; and using the point data to determine a gait measurement based on each stride interval of the plurality of stride intervals. In some embodiments, the method also includes determining a first transition from a first stance phase of the plurality of stance phases and from a first swing phase of the plurality of swing phases based on a toe off event of the subject's left hindlimb or the subject's right hindlimb; determining a second transition from a second swing phase of the plurality of swing phases to a second stance phase of the plurality of stance phases based on a foot contact event of the left hindlimb or the right hindlimb; and determining a gait measurement using the first transition and the second transition. In some embodiments, the method also includes processing the video data to determine point data tracking movement of a set of body parts of the subject during the duration of the video, the set of body parts including a left hind leg and a right hind leg, and determining gait measurements includes using the point data to determine a step length for each stride interval, the step length representing a distance traveled by the right hind leg beyond a ground contact position of the immediately preceding left hind leg; using the point data to determine a stride length to be used for each stride interval, the stride length representing a distance traveled by the left hind leg during each stride interval; and using the point data to determine a step width for each stride interval, the step width representing the distance between the left hind leg and the right hind leg.In some embodiments, the method also includes processing the video data to determine point data tracking movement of a set of body parts of the subject during a duration of the video, the set of body parts including a base of the tail, and determining the gait measures includes using the point data to determine velocity data of the subject based on movement of the base of the tail for each stride interval. In some embodiments, the method also includes processing the video data to determine point data tracking movement of a set of body parts of the subject during a duration of the video, the set of body parts including a base of the tail, and determining the gait measures includes using the point data to determine a set of velocity data of the subject based on movement of the base of the tail during a set of frames representing a stride interval of the stride interval, and determining a stride velocity for the stride interval by averaging the set of velocity data. In some embodiments, the method also includes processing the video data to determine point data tracking movement of a set of body parts of the subject during a duration of the video, the set of body parts including a right hind leg and a left hind leg, and determining the gait measures includes using the point data to determine a first stance duration representative of an amount of time that the right hind leg is in contact with the ground during a stride interval; determining a first load factor based on the first stance duration and the duration of the stride interval; using the point data to determine a second stance duration representative of an amount of time that the left hind leg is in contact with the ground during a stride interval; determining a second load factor based on the second stance duration and the duration of the stride interval; and determining an average load factor for the stride interval based on the first load factor and the second load factor.In some embodiments, the method also includes processing the video data to determine point data tracking movement of a set of body parts of the subject during the duration of the video, the set of body parts including the base of the tail and the base of the neck, and determining the gait measurements includes using the point data to determine a set of vectors connecting the base of the tail and the base of the neck during a set of frames representing a stride interval of a plurality of stride intervals, and using the set of vectors to determine an angular velocity of the subject relative to the stride interval. In some embodiments, the method also includes processing the video data to determine point data tracking movement of a set of body parts of the subject during a duration of the video, the set of body parts including a mid-spine of the subject, and stride intervals associated with the set of frames of the video data, and determining the gait measurement includes using the point data to determine a displacement vector for the stride interval, the displacement vector connecting the mid-spine represented in a first frame of the set of frames and the mid-spine represented in a last frame of the set of frames. In some embodiments, the set of body parts also includes a nose of the subject, and determining the metrics data includes using the point data to determine a set of lateral displacements of the nose relative to the stride interval based on a perpendicular distance of the nose from the displacement vector for each frame of the set of frames. In some embodiments, the lateral displacement of the nose is further based on a body length of the subject. In some embodiments, determining the gait measurement further includes determining a tail tip displacement phase offset by performing an interpolation using a set of lateral displacements of the nose to generate a smooth curve of lateral nasal displacement versus stride interval, determining at what point during the stride interval maximum nasal displacement occurred using the smooth curve of lateral nasal displacement, and determining a percent stride position representing the percent of the stride interval completed when maximum nasal displacement occurred.In some embodiments, the method also includes processing the video data to determine point data tracking movement of a set of body parts of the subject during the duration of the video, the set of body parts further including a base of the subject's tail, and determining the gait measurements includes using the point data to determine a set of lateral displacements of the base of the tail relative to stride interval based on a perpendicular distance of the base of the tail from a displacement vector for each frame in the set of frames. In some embodiments, determining gait measurements further includes determining a tail base displacement phase offset by performing an interpolation using the set of lateral displacements of the tail base to generate a smooth curve of lateral displacement of the tail base versus stride interval, using the smooth curve of lateral displacement of the tail base to determine at what point during the stride interval a maximum displacement of the tail base occurred, and determining a percent stride position representing a percentage of the stride interval completed when the maximum displacement of the tail base occurred. In some embodiments, the method also includes processing the video data to determine point data tracking movement of a set of body parts of the subject during a duration of the video, the set of body parts including the subject's tail tip, and determining gait measurements includes using the point data to determine a set of lateral displacements of the tail tip versus stride interval based on a perpendicular distance of the tail tip from a displacement vector for each frame in the set of frames. In some embodiments, determining the gait measurements also includes determining a tail tip displacement phase offset by performing an interpolation using the set of lateral displacements of the tail tip to generate a smooth curve of lateral displacement of the tail tip versus stride interval, using the smooth curve of lateral displacement of the tail tip to determine at what point during the stride interval the maximum displacement of the tail tip occurred, and determining a percent stride position representing the percent of the stride interval completed when the maximum displacement of the tail tip occurred. In some embodiments, the method also includes processing the video data to determine point data tracking movement of a set of body parts during the duration of the video, the set of body parts including one or more of the nose, base of the neck, mid-spine, left hind leg, right hind leg, base of the tail, mid-tail, and tip of the tail, using the point data to determine features of the subject, and processing the features using at least one machine learning model to determine a visual frailty score.In some embodiments, the method also includes processing the video data using an additional machine learning model to identify a likelihood that the subject exhibits a grooming behavior for the multiple video frames of the video data, and determining a visual frailty score using the likelihood that the subject exhibits the grooming behavior. In some embodiments, the method also includes processing the video data using an additional machine learning model to identify a likelihood that the subject exhibits a predetermined behavior for the multiple video frames of the video data, and determining a visual frailty score using the likelihood of the subject exhibiting the predetermined behavior. In some embodiments, the method also includes determining a set of rotated video frames by rotating the first set of video frames of the video data; processing the first set of video frames using a first machine learning model configured to identify a likelihood of the object exhibiting the predetermined behavioral action; determining a first probability of the object exhibiting the predetermined behavioral action in a first video frame of the first set of video frames, the first video frame corresponding to the first duration of the video data, based on processing the first set of video frames with the first machine learning model; processing the set of rotated frames using the first machine learning model; and determining a second probability of the object exhibiting the predetermined behavioral action in a second video frame of the rotated set of video frames, the second video frame corresponding to the first duration of the video data, based on processing the set of rotated video frames with the first machine learning model; and identifying a first label for the first video frame using the first probability and the second probability, wherein the first label indicates that the object exhibits the predetermined behavioral action.In some embodiments, the method also includes processing the first set of video frames using a second machine learning model configured to identify a likelihood of the object exhibiting the predetermined behavioral action; determining a third probability of the object exhibiting the predetermined behavioral action in the first video frames based on processing the first set of video frames with the second machine learning model; processing the rotated set of video frames using the second machine learning model; determining a fourth probability of the object exhibiting the predetermined behavioral action in the second video frames based on processing the rotated set of video frames with the second machine learning model; and identifying the first label using the first probability, the second probability, the third probability, and the fourth probability. In some embodiments, the method also includes determining a set of reflected video frames by reflecting the first set of video frames, processing the set of reflected video frames using a first machine learning model, determining a third probability of the subject exhibiting a predetermined behavioral action in a third video frame of the set of reflected frames, the third video frame corresponding to the first duration of the first video frames, based on processing the set of reflected video frames with the first machine learning model, and identifying the first label using the first probability, the second probability, and the third probability. In some embodiments, the subject is a mouse, and the predetermined behavior includes a grooming behavior including at least one of paw licking, one-sided face washing, bilateral face washing, and flank licking. In some embodiments, the first set of video frames represents a portion of video data for a period of time, and the first video frame is a last time frame of the period.In some embodiments, the method also includes identifying a second set of video frames from the video data, determining a second rotated set of video frames by rotating the second set of video frames, processing the second set of video frames using the first machine learning model, determining a third probability of the subject exhibiting the predetermined behavioral action in a third video frame of the second set of video frames based on processing the second set of video frames with the first machine learning model, processing the second rotated set of video frames using the first machine learning model, and determining a fourth probability of the subject exhibiting the predetermined behavioral action in a fourth video frame of the rotated set of frames based on processing the second rotated set of video frames with the first machine learning model, where the fourth video frame corresponds to the third video frame, and identifying a second label for the fourth video frame using the third probability and the fourth probability, the second label indicating that the subject exhibits the predetermined behavioral action. In some embodiments, the first machine learning model is a machine learning classifier. In some embodiments, the method also includes processing the video data to determine gait measurements of the subject during the duration of the video, processing the video data to determine behavioral data identifying portions of the video in which the subject exhibits a predetermined behavior, and processing the spinal mobility features, gait measurements, and behavioral data using at least one machine learning model to determine a visual frailty score. In some embodiments, the video captures the movement of the subject in an open field arena with a monocular view. In some embodiments, the method also includes determining a physical state of the subject using the visual frailty score. In some embodiments, the physical state is frail. In some embodiments, the frailty is a disease or condition. In some embodiments, the physical state is a pre-frail state. In some embodiments, the subject is a mammal, optionally a mouse.

[0008] According to another aspect of the present invention, there is provided a method of assessing a physical condition of a subject, comprising using the computer-implemented method of any one of the embodiments of the previous aspects to determine a visual frailty score of the subject. In some embodiments, the physical condition is frail. In some embodiments, the physical condition is pre-frail. In some embodiments, the physical condition is a disease or condition. In some embodiments, the subject is a mammal, optionally a mouse.

[0009] According to another aspect of the present invention, there is provided a method for determining the presence of an effect of a candidate compound on a frailty condition, the method comprising: obtaining a first visual frailty score for a subject, the obtaining means comprising a computer-implemented method according to any one of claims A1-A39, the subject having a frailty condition or an animal model exhibiting a frailty condition; administering the candidate compound to the subject; obtaining a post-administration visual frailty score for the subject; and comparing the first gait measurement value with the post-administration visual frailty score value, the difference between the first visual frailty score and the post-administration visual frailty score identifying an effect of the candidate compound on the frailty condition. In some embodiments, an improvement in the visual frailty score, indicative of a lower degree of frailty, identifies the candidate compound as promoting regression of the frailty condition. In some embodiments, a post-administration visual frailty score that is statistically equal to the first visual frailty score identifies a candidate compound that inhibits progression of the frailty condition in the subject. In some embodiments, the method also comprises additionally testing the effect of the compound in treating the frailty condition. In some embodiments, the subject is a mammal, optionally a mouse.

[0010] According to another aspect of the present invention, there is provided a method for identifying the presence of an effect of a candidate compound on a frailty condition, comprising administering a candidate compound to a subject having a frailty condition or to a subject that constitutes an animal model for a frailty condition, and obtaining a visual frailty score for the subject, the means for obtaining including the computer-implemented method embodiment of any of the above-mentioned aspects of the present invention, and comparing the obtained visual frailty score with a visual frailty score of a control, the difference between the obtained visual frailty score and the visual frailty score of the control identifying the presence of an effect of the candidate compound on the frailty condition. In some embodiments, an improvement in the visual frailty score indicating less frailty in the subject administered with the candidate compound compared to the frailty score of the control identifies the candidate compound as promoting regression of the frailty condition in the subject. In some embodiments, the visual frailty score obtained in the subject administered with the candidate compound is statistically equivalent to the control frailty score, identifying that the candidate compound inhibits the progression of the frailty condition in the subject. In some embodiments, the subject is a mammal, optionally a mouse.

[0011] According to another aspect of the invention, there is provided a system including at least one processor and at least one memory including instructions that, when executed by the at least one processor, cause the system to: receive video data representative of video capturing motion of a subject, use the video data to determine a spinal mobility feature of the subject for a duration of the video, and process the at least one spinal mobility feature using at least one machine learning model to determine a visual frailty score for the subject. In some embodiments, the instructions for causing the system to determine a spinal mobility feature of the subject for a duration of the video further cause the system to determine a plurality of spinal measurements, each spinal measurement of the plurality of spinal measurements corresponding to one video frame of the video data, and determine the spinal mobility feature using the plurality of spinal measurements. In some embodiments, the instructions for causing the system to determine a feature of spinal mobility of the subject during the duration of the video cause the system to determine, for each video frame of the video data, a first distance between the subject's head and the subject's tail, determine a second distance between the subject's mid-back and a midpoint between the subject's head and tail, determine an angle formed between the subject's head, tail, and mid-back, and determine a feature of spinal mobility of the video frame including the first distance, the second distance, and the angle. In some embodiments, the instructions for causing the system to determine a feature of spinal mobility of the subject during the duration of the video further cause the system to determine, for each video frame of the video data, a distance between the subject's mid-back and a midpoint between the subject's head and the subject's tail. In some embodiments, the at least one memory includes further instructions that, when executed by the at least one processor, cause the system to process the video data using at least one machine learning model to determine posture estimation data that tracks a position of at least the subject's head, the subject's tail, and the subject's mid-back during the duration of the video, and use the posture estimation data to determine a feature of spinal mobility.In some embodiments, the at least one memory also includes further instructions that, when executed by the at least one processor, cause the system to process the video data to determine pose estimation data that tracks one of the at least 12 body parts of the subject during the duration of the video, determine characteristics of the subject using the pose estimation data, and process the characteristics using at least one machine learning model to determine a visual frailty score. In some embodiments, the at least one memory includes further instructions that, when executed by the at least one processor, cause the system to determine physical characteristics for the subject, the physical characteristics corresponding to at least one of a length of the subject, a width of the subject, and a distance between the subject's hind legs, and process the physical characteristics using the at least one machine learning model to determine a visual frailty score. In some embodiments, the at least one memory includes further instructions that, when executed by the at least one processor, cause the system to: determine a number of rear events occur during a duration of the video, determine a rear length for each rear event, and process the number of rear events occur and the rear length for each rear event using at least one machine learning model to determine a visual frailty score. In some embodiments, the at least one memory includes further instructions that, when executed by the at least one processor, cause the system to: process the video data using at least an additional machine learning model to determine ellipse fit data for a subject during a duration of the video, use the ellipse fit data to determine features of the subject, and process the features using the at least one machine learning model to determine a visual frailty score.In some embodiments, the instructions for causing the system to determine spinal mobility features of the subject during the duration of the video further cause the system to determine a first set of video frames representative of ambulatory movement by the subject, determine a first set of spinal mobility features for the first set of video frames, determine a second set of video frames representative of non-ambulatory movement by the subject, and determine a second set of spinal mobility features for the second set of video frames, wherein the spinal mobility features include the first set of spinal mobility features and the second set of spinal mobility features. In some embodiments, the first set of spinal mobility features corresponds to a distance between the subject's mid-back and a midpoint between the subject's head and tail, and the second set of spinal mobility features corresponds to an angle formed between the subject's head, tail, and mid-back. In some embodiments, the at least one memory also includes further instructions that, when executed by the at least one processor, cause the system to determine gait measures of the subject using the video data during the duration of the video, and process the gait features using at least one machine learning model to determine a visual frailty score for the subject. In some embodiments, the at least one memory also includes further instructions that, when executed by the at least one processor, cause the system to process the video data to determine point data tracking movement of a set of body parts of the subject during the duration of the video; use the point data to determine a number of stance phases and a number of swing phases represented in the video data; determine a number of stride intervals represented in the video data based on the number of stance phases and the number of swing phases; and use the point data to determine a gait measurement for the subject based on each stride interval of the number of stride intervals.In some embodiments, the at least one memory includes further instructions that, when executed by the at least one processor, cause the system to determine a first transition from a first stance phase of the multiple stance phases and from a first swing phase of the multiple swing phases based on a toe off event of the subject's left hindlimb or the subject's right hindlimb, determine a second transition from a second swing phase of the multiple swing phases to a second stance phase of the multiple stance phases based on a foot contact event of the left hindlimb or the right hindlimb, and determine a gait measurement using the first transition and the second transition. In some embodiments, the at least one memory includes further instructions that, when executed by the at least one processor, cause the system to: process the video data to determine point data tracking movement of a set of body parts of the subject during the duration of the video, the set of body parts including a left hind leg and a right hind leg, and determining gait measurements includes using the point data to determine a step length for a stride interval, the step length representing a distance the right hind leg travels beyond a ground contact position of a previous left hind leg; use the point data to determine a stride length to be used for each stride interval, the stride length representing a distance traveled by the left hind leg during each stride interval; and use the point data to determine a step width for each stride interval, the step width representing the distance between the left hind leg and the right hind leg. In some embodiments, the at least one memory, when executed by the at least one processor, includes further instructions for processing the system to process the video data to determine point data tracking movement of a set of body parts of the subject during the duration of the video, the set of body parts including the base of the tail, and for determining gait measurements, the instructions for causing the system to use the point data to determine velocity data for the subject based on movement of the base of the tail relative to stride intervals.In some embodiments, the at least one memory, when executed by the at least one processor, includes further instructions that cause the system to process the video data to determine point data tracking movement of a set of body parts of the subject during the duration of the video, the instructions causing the system to determine gait measurements using the point data to determine a set of velocity data for the subject based on movement of the base of the tail during a set of frames representing a stride interval among the stride intervals, and to determine a stride velocity for the stride interval by averaging the set of velocity data. In some embodiments, the at least one memory, when executed by the at least one processor, includes further instructions that cause the system to: process the video data to determine point data tracking movement of a set of body parts of the subject during a duration of the video, the set of body parts including a right hind leg and a left hind leg; and determine a gait measurement using the point data to determine a first stance duration representing an amount of time that the right hind leg is in contact with the ground during a stride interval; determine a first load factor based on the first stance duration and the duration of the stride interval; use the point data to determine a second stance duration representing an amount of time that the left hind leg is in contact with the ground during a stride interval; determine a second load factor based on the second stance duration and the duration of the stride interval; and determine an average load factor for the stride interval based on the first load factor and the second load factor.In some embodiments, the at least one memory, when executed by the at least one processor, includes further instructions that cause the system to process the video data to determine point data tracking movement of a set of body parts of the subject during the duration of the video, the set of body parts including the base of the tail and the base of the neck, further causing the system to use the point data to determine a set of vectors connecting the base of the tail and the base of the neck during a set of frames representing a stride interval of the plurality of stride intervals, and to determine an angular velocity of the subject relative to the stride interval using the set of vectors. In some embodiments, the at least one memory, when executed by the at least one processor, further includes instructions for processing the video data to determine point data tracking movement of a set of body parts of the subject during a duration of the video, the set of body parts including a mid-spine of the subject, and stride intervals associated with a set of frames of the video data, and the instructions for determining gait measures further include instructions for processing the system to determine a displacement vector for the stride interval using the point data, the displacement vector connecting the mid-spine represented in a first frame in the set of frames and the mid-spine represented in a last frame in the set of frames. In some embodiments, the set of body parts also includes the nose of the subject. The instructions for determining gait measurements further cause the system to determine metrics data using the point data, which further causes the system to determine a set of lateral displacements of the nose for each stride interval based on a perpendicular distance of the nose from a displacement vector for each frame of the set of frames. In some embodiments, the lateral displacement of the nose is further based on a body length of the subject. In some embodiments, the instructions for determining gait measurements further cause the system to determine a tail tip displacement phase offset by performing an interpolation using the set of lateral displacements of the nose to generate a smooth curve of lateral displacement of the nose versus stride interval, using the smooth curve of lateral displacement of the nose to determine when during the stride interval the maximum displacement of the nose occurred, and determining a percent stride position representing the percent of the stride interval completed when the maximum displacement of the nose occurred. In some embodiments, the at least one memory, when executed by the at least one processor, includes further instructions for causing the system to process the video data to determine point data tracking movement of a set of body parts of the subject during the duration of the video, the set of body parts of the subject further including the base of the subject's tail, and determining gait measurements further includes instructions for causing the system to use the point data to determine a set of lateral displacements of the base of the tail relative to stride interval based on a perpendicular distance of the base of the tail from a displacement vector for each frame in the set of frames.In some embodiments, the instructions to cause the system to determine the gait measurements further cause the system to determine a tail base displacement phase offset by performing an interpolation using a set of lateral displacements of the tail base to generate a smooth curve of the tail base lateral displacement versus stride interval, using the tail base lateral displacement smooth curve to determine at what point during the stride interval the maximum tail base displacement occurred, and determining a percent stride position representing the percent of the stride interval completed when the maximum tail base displacement occurred. In some embodiments, the at least one memory, when executed by the at least one processor, includes further instructions that cause the system to process the video data to determine point data tracking movement of a set of body parts of the subject during a duration of the video, the set of body parts of the subject including the subject's tail tip, and determine gait measurements, further causing the system to use the point data to determine a set of lateral displacements of the tail tip relative to stride interval based on a perpendicular distance of the tail tip from a displacement vector for each frame in the set of frames. In some embodiments, the instructions to cause the system to determine the gait measurement further cause the system to determine a tail tip displacement phase offset by performing an interpolation using a set of lateral displacements of the tail tip to generate a smooth curve of tail tip lateral displacement versus stride interval, using the tail tip lateral displacement smooth curve to determine at what point during the stride interval the maximum tail tip displacement occurred, and determining a percent stride position representing the percent of the stride interval completed when the maximum tail tip displacement occurred.In some embodiments, the at least one memory includes further instructions that, when executed by the at least one processor, cause the system to process the video data to determine point data tracking movement of a set of body parts of the subject during a duration of the video, the set of body parts including one or more of the nose, base of the neck, mid-spine, left hind leg, right hind leg, base of the tail, mid-tail, and tip of the tail, determine features of the subject using the point data, and process the features using at least one machine learning model to determine a visual frailty score. In some embodiments, the at least one memory includes further instructions that, when executed by the at least one processor, cause the system to process the video data using an additional machine learning model to identify a likelihood that the subject exhibits grooming behavior for a plurality of video frames of the video data, and determine a visual frailty score using the likelihood that the subject exhibits grooming behavior. In some embodiments, the at least one memory includes further instructions that, when executed by the at least one processor, cause the system to process the video data using an additional machine learning model to identify a likelihood of a subject exhibiting a predetermined behavior for a plurality of video frames of the video data, and determine a visual frailty score using the likelihood of the subject exhibiting the predetermined behavior.In some embodiments, the at least one memory further includes instructions that, when executed by the at least one processor, cause the system to: determine a rotated set of video frames by rotating the first set of video frames of the video data; process the first set of video frames using a first machine learning model configured to identify a likelihood of a subject exhibiting a predetermined behavioral action; determine a first probability of the subject exhibiting the predetermined behavioral action in a first video frame of the first set of video frames, the first video frame corresponding to a first duration of the video data, based on processing the first set of video frames with the first machine learning model; process the rotated set of frames using the first machine learning model; and determine a second probability of the subject exhibiting the predetermined behavioral action in a second video frame of the rotated set of video frames, the second video frame corresponding to the first duration of the video data, based on processing the rotated set of video frames with the first machine learning model; and use the first probability and the second probability to identify a first label for the first video frame, wherein the first label indicates that the subject exhibits the predetermined behavioral action. In some embodiments, the at least one memory includes further instructions that, when executed by the at least one processor, cause the system to process a first set of video frames using a second machine learning model configured to identify a likelihood of an object exhibiting a predetermined behavioral action; determine a third probability of the object exhibiting the predetermined behavioral action in the first video frames based on processing the first set of video frames with the second machine learning model; process a set of rotated video frames using the second machine learning model; determine a fourth probability of the object exhibiting the predetermined behavioral action in the second video frames based on processing the rotated set of video frames with the second machine learning model; and identify a first label using the first probability, the second probability, the third probability, and the fourth probability.In some embodiments, the at least one memory includes further instructions that, when executed by the at least one processor, cause the system to: determine a set of reflected video frames by reflecting the first set of video frames; process the set of reflected video frames using a first machine learning model; determine a third probability of the subject exhibiting a predetermined behavioral action in a third video frame of the set of reflected frames, the third video frame corresponding to a first duration of the first video frames, based on processing the set of reflected video frames with the first machine learning model; and identify a first label using the first probability, the second probability, and the third probability. In some embodiments, the subject is a mouse, and the predetermined behavior includes a grooming behavior including at least one of licking paws, washing one side, washing both sides, and licking flanks. In some embodiments, the first set of video frames represents a portion of video data for a period of time, and the first video frame is a last time frame of the period.In some embodiments, the at least one memory further comprises instructions that, when executed by the at least one processor, cause the system to: identify a second set of video frames from the video data, determine a second rotated set of video frames by rotating the second set of video frames, process the second set of video frames using the first machine learning model, determine a third probability of an object exhibiting a predetermined behavioral action in a third video frame of the second set of video frames based on processing the second set of video frames with the first machine learning model, process the second rotated set of video frames using the first machine learning model, and determine a fourth probability of an object exhibiting a predetermined behavioral action in a fourth video frame of the rotated set of frames based on processing the second rotated set of video frames with the first machine learning model, where the fourth video frame corresponds to the third video frame, and use the third probability and the fourth probability to identify a second label of the fourth video frame, the second label indicating that the object exhibits the predetermined behavioral action. In some embodiments, the first machine learning model is a machine learning classifier. In some embodiments, the at least one memory includes further instructions that, when executed by the at least one processor, cause the system to process the video data to determine gait measurements for the subject during a duration of the video, process the video data to determine behavioral data identifying portions of the video in which the subject exhibits a predetermined behavior, and process the spinal mobility features, gait measurements, and behavioral data using at least one machine learning model to determine a visual frailty score. In some embodiments, the video captures the movement of the subject in an open field arena with a monocular view. In some embodiments, the at least one memory includes further instructions that, when executed by the at least one processor, cause the system to determine a physical state of the subject using the visual frailty score. In some embodiments, the physical state is frail.In some embodiments, the physical state is a pre-frail state.In some embodiments, the subject is a mammal, optionally a mouse. [Brief description of the drawings]

[0012] For a more complete understanding of the present disclosure, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which: [Figure 1] FIG. 1 is a conceptual diagram of a system for determining a subject's behavior, according to an embodiment of the present disclosure. [Diagram 2] 1 is a flowchart illustrating a process for determining various data about an object using point data derived from video data, according to an embodiment of the present disclosure. [Diagram 3] 1 is a flowchart illustrating a process for determining morphological data of an object using ellipse data derived from video data according to an embodiment of the present disclosure. [Figure 4] 1 is a flowchart illustrating a process for determining behavioral data of a subject using video data according to an embodiment of the present disclosure. [Diagram 5] 5 is a flowchart illustrating a process for determining a visual frailty score using one or more data determined according to the process of FIGS. 2-4, according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a block diagram conceptually illustrating example components of a device in accordance with an embodiment of the present disclosure. [Figure 7] FIG. 2 is a block diagram conceptually illustrating exemplary components of a server according to an embodiment of the present disclosure. [Figure 8A]8A is a schematic diagram showing an automated visual frailty index (vFI) pipeline and a graph showing scores from a manual frailty index for mice. The schematic in FIG. 8A shows a pipeline for creating an automated visual frailty index (vFI). Top-down video of each mouse's open field was processed by a tracking and segmentation network and a pose estimation network. The resulting frame-by-frame ellipse fit and 12-point pose coordinates were further processed to create a per-video metric for the mouse. Each mouse was also manually frailty-indexed to generate an FI score. Video features of each mouse were used to model its FI score. [Figure 8B] FIG. 8B is a schematic showing the automated visual frailty index (vFI) pipeline and a graph depicting scores from the manual frailty index of mice. FIG. 8B shows a scatter plot of FI scores by age. The black line shows a piecewise linear fit to the data. Error bars show standard deviation. Males, lighter dots; females, darker dots. [Figure 8C] FIG. 8C is a schematic showing graphs representing scores from the automated visual frailty index (vFI) pipeline and the manual frailty index for mice. FIG. 8C shows a scatter plot of FI scores by age from each scorer (scorer 1, darkest dots; scorer 2, lighter dots; scorer 3, lighter dots; scorer 4, lightest dots). [Figure 9A] 9A shows a graph illustrating correlations between video metrics. A close wrap of points around the diagonal line indicates a high correlation between the mean and median or IQR, and the standard deviation for the respective indices. The graph in FIG. 9A shows the correlation between the mean / mean (x-axis) and median (y-axis) of the video gait metrics. The diagonal line corresponds to maximum correlation (i.e., 1). [Figure 9B]9A shows a graph depicting correlations between video metrics. A tight wrap of points around the diagonal indicates a high correlation between the mean and median or IQR, and the standard deviation for the respective indices. The graph in FIG. 9B shows the correlations between the interquartile range (IQR, x-axis) and standard deviation (Stdev, y-axis) video gait metrics. The diagonal corresponds to maximum correlation (i.e., 1). [Figure 10A] Video images, schematics, and graphs depicting aspects of morphological and behavioral measurements taken for scoring the visual frailty index are shown. Figures 10A-I show sample features used in vFI. Figure 10A shows a single frame of top-down open-field video. [Figure 10B] Video images, schematics, and graphs depicting aspects of morphological and behavioral measurements taken for scoring of the visual frailty index are shown. Figures 10A-I show sample features used in vFI. Figure 10B provides morphological features from ellipse fitting and hindlimb distance measurements performed on the mouse frame by frame. Consider the major and minor axes of the ellipse fitting as the length and width, respectively. [Figure 10C] Video images, schematics, and graphs depicting aspects of morphological and behavioral measurements taken for scoring the visual frailty index are shown. Figures 10A-I show sample features used in vFI. Figure 10C shows that the median ellipse fit width and median hindlimb distance taken across all mouse frames are highly correlated with FI scores. [Figure 10D] Video images, schematics, and graphs depicting aspects of morphological and behavioral measures taken for scoring the visual frailty index are shown. Figures 10A-I show sample features used in vFI. Figure 10D shows spatial, temporal, and whole-body coordination characteristics of gait used to create the metrics [Shepherd K. et al., Cell Reports 38, 110231 (January 2022)]. [Figure 10E]Video images, schematics, and graphs depicting aspects of morphological and behavioral measurements taken for scoring of the visual frailty index are shown. Figures 10A-I show sample features used in vFI. Figure 10E shows median step width with interquartile range of tip-to-tail lateral displacement taken across all strides of mice, which are highly correlated with FI scores. [Figure 10F] Video images, schematics, and graphs depicting aspects of morphological and behavioral measurements taken for scoring of the visual frailty index are shown. Figures 10A-I show sample features used in vFI. Figure 10F shows spine mobility measurements taken in each frame. dAC is the distance between points A and C (base of head and base of tail, respectively) normalized to body length, dB is the distance of point B from the midpoint of line AC (mid-back), and aABC is the angle formed by points A, B, and C. When the mouse spine is straight, dAC and aABC are at their maximum values, while dB is at its minimum value. When the mouse spine is curved, dB is at its maximum value and dAC and aABC are at their minimum values. [Figure 10G] Video images, schematics, and graphs depicting aspects of morphological and behavioral measurements taken for scoring the visual frailty index are shown. Figures 10A-10I show sample features used in vFI. Figure 10G shows the median dB taken across all mouse frames, and the median dB taken across frames where the mouse is not walking shows a correlation with FI score. [Figure 10H] Video images, schematics, and graphs depicting aspects of morphological and behavioral measurements taken for scoring of the visual frailty index are shown. Figures 10A-I show sample features used in vFI. Figure 10H shows a wall rearing event. The contours of the open field walls are captured, a 5 pixel buffer is added (edge ​​lines), and the threshold is marked. The mouse's nose point is tracked in each frame. A wall rearing event is defined by the point of the nose completely crossing the wall threshold. [Figure 10I]Video images, schematics, and graphs depicting aspects of morphological and behavioral measurements taken for scoring the visual frailty index are shown. Figures 10A-10I show sample features used in vFI. Figure 10I shows the total number of rearing events, and the number of rearing events during the first 5 minutes of open field footage shows some correlation with FI score. [Figure 11A] Graphs showing specific analysis of FI scores by age and sex are shown, as well as graphs representing comparisons of FI metrics for males and females. Figure 11A provides the distribution of FI scores for males and females when the data was divided into four age groups of equal range. The x points represent the midpoint of each age group range. Significant differences in the distribution of scores for males and females for that age group were determined by the Mann-Whitney U test. [Figure 11B] Graphs showing specific analysis of FI scores by age and sex, as well as graphs depicting a comparison of FI metrics for males and females are shown. Figure 11B shows the Pearson correlation of FI items with age for males compared to females. [Figure 11C-D] Graphs showing specific analysis of FI scores by age and sex, and graphs showing comparison of FI metrics for males and females are shown. Figure 11C shows the Pearson correlation of video metrics with FI scores for males compared to females. Figure 11D shows the Pearson correlation of video metrics with age for males compared to females. o Open field, ambulation, and genetic manipulation keys apply to Figure 11C and Figure 11D. [Figure 12A]

[0046] Figure 12A provides an embodiment of prediction of age and frailty from image features. Figure 12A provides a graphical illustration showing the different models that were fitted. [Figure 12B] Provide an embodiment of predicting age and frailty from video features. Figure 12B shows that video features are more accurate than clinical frailty index items in predicting age. The performance of random forest models was compared using frailty parameters (FRIGHT) and video-generated features (vFRIGHT) in predicting age. [Figure 12C] An embodiment of prediction of age and frailty from video features is provided. FIG. 12A provides a graphical illustration showing different models that were fitted. FIG. 12C shows the performance of the ordinal regression model (classifier) ​​in terms of accuracy (accurately predicting the value of the frailty parameter in the test using the model trained on the training data). The black dotted line superimposed on the plot indicates the accuracy obtained if a value was inferred instead of using video features. It was found that the video features encode useful information that improves the ability of the model to accurately predict the frailty parameter value. [Figure 12D] An embodiment of predicting age and frailty from video features is provided. Figure 12D shows a comparison between four models (LR*, SVM, RF, XGB) predicting FI score from video features in terms of mean absolute error (MAE), and R2 shows that RF outperforms other models. [Figure 12E] An embodiment of prediction of age and frailty from video features is provided. FIG. 12A provides a graphical illustration showing the different models that were fitted. FIG. 12E shows the uncertainty in predicting age (column 1) and FI score (column 3) plotted as a function of age (weeks). The black curves show the less fit. These plots show that there is low uncertainty in predicting age and FI score for very young mice. The distribution of prediction interval (PI) widths is plotted, and it is found that the PI widths for predicting age are wider (increased prediction uncertainty) for mice belonging to the middle age group (M). Similarly, the PI widths for predicting FI score increase with the age of the data. [Figure 12F] An embodiment of prediction of age and frailty from video features is provided. Figure 12A provides a graphical illustration showing different models that were fitted. Video features Figure 12F shows the residuals vs. index and predicted FI scores vs. true for the training (columns 1 and 2) and test sets (columns 3 and 4) for the RF model. [Figure 12G]An embodiment of predicting age and frailty from video features is provided. Figure 12A provides a graphical illustration showing the different models that were fitted. Figure 12G shows the residuals vs. exponent and predicted age vs. true for the training (columns 1 and 2) and test sets (columns 3 and 4) for the RF model. [Figure 13A] Quantile regression modeling of vFI using generalized random forest is provided. Figure 13A shows the variable importance index for three quantile random forest models (low tail - Q.025, median - Q.50, high tail - Q.975). Low tail and high tail mice correspond to mice with low and high frailty scores, respectively. [Figure 13B] Quantile regression modeling of vFI using generalized random forest is provided. Figure 13B provides a marginal ALE plot that represents how much a feature affects the model prediction on average. For example, the average predicted FI score increases with increasing step width, but remains true for values ​​above 3 for mice belonging to the lower and upper tails. [Figure 13C] Quantile regression modeling of vFI using generalized random forests is provided. Figure 13C provides a plot depicting how strongly features interact with each other. [Figure 13D] Quantile regression modeling of vFI using generalized random forests is presented. Figures 13D-E show the ALE quadratic interaction plots of step width and step length 1 on predicted FI scores. (Figure 13E: width and length). Light colors represent higher than average and dark colors represent lower than average predictions when marginal effects from features are already taken into account. The plots in Figures 13D (or E) show a weak (or strong) interaction between step width and step length 1 (or width and length), respectively. Larger step width and step length 1 increase vFI scores. [Figure 13E]Quantile regression modeling of vFI using generalized random forests is presented. Figures 13D-E show the ALE quadratic interaction plots of step width and step length 1 on predicted FI scores. (Figure 13E: width and length). Light colors represent higher than average and dark colors represent lower than average predictions when marginal effects from features are already taken into account. The plots in Figures 13D (or E) show a weak (or strong) interaction between step width and step length 1 (or width and length), respectively. Larger step width and step length 1 increase vFI scores. [Figure 14-1] A list showing feature correlations with FI scores is provided. [Figure 14-2] A list showing feature correlations with FI scores is provided. [Figure 15-1] A list showing the correlations (Pearson) between vFI features and age is provided. [Figure 15-2] A list showing the correlations (Pearson) between vFI features and age is provided. [Figure 16] A list showing correlations between manual FI items and age is provided. [Figure 17-1] We provide the FI test sheet listing all items for manual frailty indexing. The text in light font is adapted from Whitehead JC et al., Journal of Gerontology, Series A: Biomedical Sciences and Medical Sciences, 69, 621-632 (2014). [Figure 17-2] We provide the FI test sheet listing all items for manual frailty indexing. The text in light font is adapted from Whitehead JC et al., Journal of Gerontology, Series A: Biomedical Sciences and Medical Sciences, 69, 621-632 (2014). [Figure 17-3]We provide the FI test sheet listing all items for manual frailty indexing. The text in light font is adapted from Whitehead JC et al., Journal of Gerontology, Series A: Biomedical Sciences and Medical Sciences, 69, 621-632 (2014). [Figure 18A] Graphs depicting estimates of scorer effects on clinical FI items. Figure 18A shows that the effect of the tester varies by FI item. [Figure 18B] 18A and 18B show graphs depicting estimates of scorer effects on the Clinical FI items. FIG. 18B shows the estimated random effects among the four scorers in the dataset. [Figure 19A] Graphs and plots of detailed modeling analysis are provided. Figure 19A shows the distribution of age across 643 data points (533 mice). Distribution of manual FIadj scores across 643 data points (533 mice). [Figure 19B] Provide detailed modeling analysis graphs and plots. Figure 19B shows the results related to determining the contribution of frailty parameters in predicting age. The feature importance of all frailty parameters is calculated, and it is determined that gait disorder, kyphosis, and hair bristles are the most highly contributing. [Figure 19C] Detailed modeling analysis graphs and plots are provided. Figure 19C shows the results showing that the random forest regression model performed better than other models with the lowest root mean square error (RMSE) (p<2.2e-16, F3,147=59.53) and the highest R2 (p<2.2e-16, F3,147=58.14) when compared using repeated measures ANOVA. [Figure 19D]Detailed modeling analysis graphs and plots are provided. Figure 19D shows that the vFRIGHT model performed better than the FRIGHT model, with lower RMSE (RMSEvFRIGHT=17.97±1.44, RMSEFRIGHT=20.62±4.78, p<6.1e-7, F1,49=32.84) and higher R2 (RMSEvFRIGHT=0.78±0.04, RMSEFRIGHT=0.76±0.07, p<2.1e-8, F1,49=44.54) compared to using repeated measures ANOVA. [Figure 19E] Detailed modeling analysis graphs and plots are provided. Figure 19E shows the Random Forest regression model for predicting FI scores for unseen future data, which performed better than all other models with the lowest root mean square error (RMSE) (p<8.3e-14, F3,147=26.62) and highest R2 (p<4.7e-14, F3,147=27.2). [Figure 19F] Detailed modeling analysis graphs and plots are provided. The plot in Figure 19F shows the distribution of counts for individual frailty parameters (0-1 of 3, 0.5-2 of 3, 1-3 of 3 in each set of 3 values) for many parameters such as nasal discharge, rectal prolapse, vaginal discharge, and diarrhea, with the proportion of 0 counts being 1 (p0=1). Similarly, dermatitis, cataracts, eye discharge and swelling, microphthalmia, corneal opacity, tail stiffness, and malocclusion have p0>0.95. [Figure 19G] Detailed modeling analysis graphs and plots are provided. Figure 19G shows residuals vs. index, predicted residuals vs. true for the training (rows 1 and 2) and test sets (rows 3 and 4) for a model predicting age using the frailty index term, for both the training and test data. [Figure 19H]We provide graphs and plots of the detailed modeling analysis. Figure 19H and Figure 19I show the results of out-of-bag (OOB) error-based 95% prediction intervals (PI) (grey lines), which quantify the uncertainty of the point estimates / predictions (grey dots). There was one interval per test mouse, and approximately 95% of the PI intervals contained the correct age (Figure 19I) and FI score (Figure 19H). The x-axis (test set index) was ordered by ascending (left to right) order of actual age / FI. [Figure 19I] We provide graphs and plots of the detailed modeling analysis. Figure 19H and Figure 19I show the results of out-of-bag (OOB) error-based 95% prediction intervals (PI) (grey lines), which quantify the uncertainty of the point estimates / predictions (grey dots). There was one interval per test mouse, and approximately 95% of the PI intervals contained the correct age (Figure 19I) and FI score (Figure 19H). The x-axis (test set index) was ordered by ascending (left to right) order of actual age / FI. [Figure 20] Results of a test of Simpson's paradox are presented. Simpson

[35] showed that statistical relationships observed in a population can be reversed in all of the subgroups that make up that population, leading to erroneous conclusions being drawn from population data. To test for the manifestation of Simpson's paradox in the data, the bimodal age distribution was split into two separate unimodal distributions (clusters): age < 70 weeks (L70) vs. age > 70 weeks (U70). The dependent variable (frailty) was then plotted against each of the independent variables / characteristics in the data, and simple linear regression models were fitted separately to each subgroup as well as to the aggregate data (data not shown). Correlation was quantified by measuring the slope of the linear fit of the characteristic (Y) with age (X). The slopes for L70, U70, and overall (all) were calculated, and the slopes were plotted in descending order for the characteristics with respect to their association to the model (age was predicted from these characteristics). For each set of three bars, the left bar is L70, the middle bar is U70, and the right bar is All. It was determined that Simpson's Paradox did not appear in any of the top 15 features in the data. [Figure 21A]We provide graphs from further experiments to test the model's performance and parameters. Figure 20A shows the results of a comparison of the performance of different feature sets, 1) age only, 2) video, and 3) age + video, in predicting frailty. Age only was used as a feature for linear (AgeL) and generalized additive nonlinear models (AgeG). The random forest model using video features (VideoRF) did not show a clear improvement on vFI prediction based on age alone, but the model (AllRF) showed a clear improvement in predictive performance. The model (AllRF) included video features + age with the lowest MSE (p<2.2e-16, F3,147=213.79, LMM post hoc pairwise comparison with AgeG, t147=-12.21, FDR adj. p<0.0001), lowest RMSE (p<2.2e-16, F3,147=172.88, LMM post hoc pairwise comparison with AgeG, t147=-14.12, FDR adj. p<0.0001), and highest R2 (p<2.2e-16, F3,147=171.12, LMM post hoc pairwise comparison with AgeG, t147=14.07, FDR adj. p<0.0001). This indicates that video features add important information about frailty that is not applicable to age alone. [Figure 21B]Graphs are provided from further experiments to test model performance and parameters. Figure 21B shows the results of an embodiment using selected animals with age and FI score having an inverse relationship, i.e., younger animals with higher FI score and older animals with lower FI score. Five (5) test sets were formed that included animals with these criteria, and random forest (RF) models were trained on the remaining mice. The model using only video features (VideoRF) outperformed all other models for these mice with the lowest MSE (p<1.6e-08, F3,12=91.07, LMM post hoc pairwise comparison with AgeG, t12=13.60, FDR adj. p<0.0001), lowest RMSE (p<1.6e-08, F3,12=93.88, LMM post hoc pairwise comparison with AgeG, t12=14.15, FDR adj. p<0.0001), and highest R2 (p<1.31e-08, F3,12=94.32, LMM post hoc pairwise comparison with AgeG, t12=14.10, FDR adj. p<0.0001). [Figure 21C] We provide graphs from further experiments to test the model's performance and parameters. Figure 21C shows the results of further investigation of the difference between age predictors and vFI predictors in terms of feature importance. Features lying along the diagonal are important for both age and vFI prediction. [Figure 21D] We provide graphs from further experiments to test the model's performance and parameters. Figure 21D shows the results of predicting FI scores from video features extracted from videos with shorter duration. We investigated the loss of accuracy in predicting age and FI scores using video features generated from videos with shorter duration (first 5 minutes and 20 minutes). A random forest model trained with features generated from 60 minutes of video was used as a baseline model for comparison. The loss of accuracy was reduced when shorter videos were used. [Figure 21E]We provide graphs from further experiments to test the model's performance and parameters. Figure 21E shows the results of a study to confirm how much training data is actually needed. Simulation tests were performed, with different percentages of total data allocated to training. As expected, there is a general downward (upward) trend in MAE, RMSE (R2, increasing the percentage of data allocated to training set). In fact, a smaller training set (<80% training) can achieve similar training performance. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Although aging of the renal system is uniform, biological aging is heterogeneous. Clinically, this heterogeneity is manifested in health status and mortality, differentiating healthy aging from unhealthy aging. Clinical frailty indices serve as important tools in aging to capture health status. Frailty indices have been adapted for use in mice and are effective predictors of mortality risk. To advance our understanding of biological aging, high-throughput approaches to preclinical testing are needed. However, currently, frailty indexing in mice is manual and relies on trained / expert manual scorers, limiting scalability and reliability in generating frailty indices.

[0014] The present disclosure relates to an automated visual frailty system that processes video data of a subject and generates a visual frailty score for the subject. The automated visual frailty system of the present disclosure (e.g., system 100 shown in FIG. 1) may use one or more machine learning based techniques to determine the subject's visual frailty score and may operate on video data from an open field assay. The automated visual frailty system may determine the subject's visual frailty score based on biological aging features extracted from the video data. In some embodiments, the automated visual frailty system may extract morphological features, gait and posture features, behavioral features, and other features from the video data that may be used to determine the visual frailty score. The automated visual frailty system may improve accuracy, reproducibility, scalability, and efficiency in generating a subject's frailty index.

[0015] The system 100 of the present disclosure may be operated using various components shown in FIG. 1. The system 100 may include an image capture device 101, a device 102, and one or more systems 105 connected via one or more networks 199. The image capture device 101 may be part of, included within, or connected to another device (e.g., device 600), and may further be a camera, or a high-speed video camera, or other type of device capable of capturing images or video. The device 101 may include motion detection sensors, infrared sensors, temperature sensors, ambient condition detection sensors, and other sensors configured to detect various characteristics / environmental conditions in addition to or in place of the image capture device. The device 102 may be a laptop, desktop, tablet, smartphone, or other type of computing device capable of displaying data, and may include one or more components described in connection with device 600 below.

[0016] The image capture device 101 may capture a video (or one or more images) of the subject and may transmit video data 104 representing the video to the system(s) 105 for processing as described herein. The video may include the subject's movements in an open field arena. In some cases, the video data 104 may correspond to images (image data) captured by the device 101 at a particular time interval, such that the images capture the subject's movements over a period of time. The system(s) 105 may include one or more components shown in FIG. 1 and may be configured to process the video data 104 to determine a visual frailty score for the subject. The system(s) 105 may generate a visual frailty score 162 corresponding to the subject. The system(s) 105 may transmit the visual frailty score 162 to the device 102 for output to a user to view the results of processing the video data 104.

[0017] In some embodiments, the video data 104 may include videos of two or more subjects, and the system(s) 105 may process the video data 104 to determine the characteristics and visual frailty score of each subject represented in the video data 104.

[0018] The system(s) 105 may be configured to determine various features from the video data 104 for the subject. To determine these features and to determine the visual frailty score, the system(s) 105 may include a number of different components. As shown in FIG. 1, the system(s) 105 may include a point tracking component 110, a gait and posture analysis component 120, an ellipse generation component 130, an open field analysis component 140, a grooming behavior analysis component 150, and a visual frailty analysis component 160. The system(s) 105 may include fewer or more components than those shown in FIG. 1. In some embodiments, these various components may be located on the same physical system 105. In other embodiments, one or more of the various components may be located on different / separate physical systems 105. Communication between the various components may be direct or over a network 199. Communication between the device 101, the system(s) 105, and the device 102 may occur directly or via a network 199.

[0019] In some embodiments, one or more components illustrated as part of the system(s) 105 may be located on the device 102 or on a computing device connected to the image capture device 102 (e.g., device 600).

[0020] At a high level, the system(s) 105 may be configured to process the video data 104 to determine point data (which may be referred to as posture estimation data in the examples below). Using the point data, the system(s) 105 may determine various features corresponding to the movement of the subject in the video, such as gait measurements, spine measurements, rearing events, rear limb measurements, etc. Details regarding the determination of the point data and various features from the point data are described below in connection with FIG. 2. The system(s) 105 may also be configured to determine ellipse data (which may be referred to as ellipse fit in the examples below). Using the ellipse data, the system(s) 105 may determine morphological data for the subject. Details regarding the determination of ellipse data and morphological data are described below in connection with FIG. 3. The system(s) 105 may also be configured to use the video data 104 to determine behavioral features of the subject. Details regarding the determination of behavioral features are described below in connection with FIG. 4. Using the determined features / data, the system(s) 105 may then determine the visual frailty score 162 of the subject. Details regarding determining the visual frailty score 162 are described below in connection with FIG.

[0021] 2 is a flow chart illustrating a process 200 for determining various data of an object using point data derived from video data, according to an embodiment of the present disclosure. One or more of the steps 200 may be performed in a different order / sequence than that shown in FIG. 2. One or more steps of the process 200 may be performed by the point tracking component 110 and / or the gait and posture analysis component 120.

[0022] At step 202, the point tracking component 110 may receive video data 104 representing the movement of an object. At step 204, the point tracking component 110 may process the video data 104 to determine point data 112 that tracks the movement of a set of multiple object body parts. The point tracking component 110 may be configured to identify various body parts of the object. These body parts may be identified using various point data such that a first point data may correspond to a first body part and a second point data may correspond to a second body part. The point data may, in some embodiments, be one or more pixel positions / coordinates (x, y) corresponding to the body parts. In this manner, the point data 112 may include multiple point data corresponding to multiple body parts. The point tracking component 110 may be configured to identify pixel positions corresponding to a particular body part in one or more video frames of the video data 104. The point tracking component 110 may track the movement of a particular body part during the duration of the video by identifying corresponding pixel positions in the video. The point data 112 may indicate the position of a particular body part in a particular frame of the video. The point data 112 may include the locations of all body parts identified and tracked by the point tracking component 110 across multiple frames of the video data 104. The point data 112 may also include a confidence score for the location of a particular body part in a particular video frame. The confidence score may indicate how reliable the point tracking component 110 is in determining that particular location. The confidence score may be a probability / likelihood that a particular body part is at that particular location.

[0023] In some embodiments, if the subject is a mouse, the point tracking component 110 may identify and track the following body parts: nose, left ear, right ear, base of the neck, left forelimb, right forelimb, mid-spine, left hind limb, right hind limb, base of the tail, mid-tail, and tip of the tail.

[0024] The point data 112 may be a vector, array, or matrix representing pixel coordinates of various body parts on multiple video frames. For example, the point data 112 may be [frame1={nose:(x1, y1); right hind leg:(x2, y2)}], [frame2={nose:(x3, y3); right hind leg:(x4, y4)}], etc. The point data 112 may include, for each frame, in some embodiments, coordinates of at least 12 pixels representing 12 parts / body parts of the subject that the point tracking component 110 is configured to track.

[0025] The point tracking component 110 may implement one or more pose estimation techniques. The point tracking component 110 may include one or more machine learning models configured to process the video data 104. In some embodiments, the one or more machine learning models may be neural networks, such as deep neural networks, deep convolutional neural networks, recurrent neural networks, etc. In other embodiments, the one or more machine learning models may be types of models other than neural networks. The ML models of the point tracking component 110 may be configured for 3D landmark-less pose estimation based on transfer learning with deep neural networks.

[0026] The point tracking component 110 may be configured to determine the point data 112 with high accuracy and precision since the visual frailty score 162 may be sensitive to errors in the point data 112. The point tracking component 110 may implement an architecture that maintains high resolution features throughout the machine learning model stack, thereby preserving spatial accuracy. In some embodiments, the architecture of the point tracking component 110 may include one or more transposed convolutions to create a match between the resolution of the heatmap output and the resolution of the video data 104. The point tracking component 110 may be configured to determine the point data 112 at near real-time speeds and may run a powerful GPU. The point tracking component 110 may be configured to facilitate modification and enhancement. In some embodiments, the point tracking component 110 may be configured to generate inferences at a fixed scale rather than processing at multiple scales, which may save computational resources and time.

[0027] In some embodiments, the video data 104 may track the movement of one object, and the point tracking component 110 may be configured not to perform any object detection techniques / algorithms to detect the object within the video frames. In other embodiments, the video data 104 may track the movement of two or more objects, and the point tracking component 110 may be configured to distinguish one object from another object within the video data 104 by performing object detection techniques.

[0028] In step 206, gait and posture analysis component 120 may process point data 112 to determine gait measurement data 122 of the target. Gait and posture analysis component 120 may use point data 112 to determine distances and / or angles between various target body parts.

[0029] Gait and posture analysis component 120 may determine distances between various body parts of the subject(s) and generate one or more distance vectors. Gait and posture analysis component 120 may determine, for each video frame of video data 104, a first distance between two (a first pair) of body parts, a second distance between another two (a second pair) of body parts, etc., and the first distance and the second distance may be included in the distance vector. In some embodiments, gait and posture analysis component 120 may determine a first distance feature vector representing the distance between a first pair of body parts for multiple video frames, a second distance feature vector representing the distance between a second pair of body parts for multiple video frames, etc. Each value of the first distance vector may represent the distance between the first pair of body parts for a different corresponding video frame of video data 104. In some embodiments, the distance vector may be included in gait measurement data 122 used to determine visual frailty score 162. In other embodiments, the distance vector may be used by gait and posture analysis component 120 to determine the data included in gait measurement data 122.

[0030] Gait and posture analysis component 120 may determine angles between various body parts of the subject(s) and generate one or more angle vectors. Gait and posture analysis component 120 may determine first angle data between three (a first trio) of body parts, second angle data between another three (a second trio) of body parts, etc. for a plurality of video frames. Gait and posture analysis component 120 may determine a first angle vector representing the angle between the first trio of body parts across a plurality of video frames, a second angle vector representing the angle between the second trio of body parts across a plurality of video frames, etc. Each value of the first angle vector may represent an angle between the first trio of body parts for a different corresponding video frame of video data 104. In some embodiments, the angle vectors may be included in gait measurement data 122 used to determine visual frailty score 162. In other embodiments, the angle vectors may be used by gait and posture analysis component 120 to determine the data included in gait measurement data 122.

[0031] In some embodiments, the gait and posture analysis component 120 may determine gait and posture metrics. As used herein, gait metrics may refer to metrics derived from the subject's foot movement. Gait metrics may include, but are not limited to, step width, step length, stride length, velocity, angular velocity, and limb loading factor. As used herein, posture metrics may refer to metrics derived from the subject's whole body movement. In some embodiments, the posture metrics may be based on the subject's nose and tail movement. The posture metrics may include, but are not limited to, nose lateral displacement, tail base lateral displacement, tail tip lateral displacement, nose lateral displacement phase offset, tail base displacement phase offset, and tail tip displacement phase offset. One or more of the gait and posture metrics may be included in the gait measurement data 122. In some embodiments, each of the gait and posture metrics may be provided to the visual frailty analysis component 160 as a separate input rather than as a collective input via the gait measurement data 122.

[0032] The gait and posture analysis component 120 may determine one or more of the gait and posture metrics for each stride. The gait and posture analysis component 120 may determine the stride interval(s) represented in the video frames of the video data 104. In some embodiments, the stride interval may be based on a stance phase and a swing phase. In an exemplary embodiment, an approach for detecting the stride interval is based on the cyclical structure of gait. During a stride cycle, each foot may have a stance phase and a swing phase. During the stance phase, the subject's foot supports the subject's weight and is in static contact with the ground. During the swing phase, the foot moves forward and is not supporting the subject's weight. The transition from the stance phase to the swing phase is referred to herein as a toe off event, and the transition from the swing phase to the stance phase is referred to as a foot contact event.

[0033] The gait and posture analysis component 120 may determine a number of stance and swing phases represented by the duration of the video data 104. In an exemplary embodiment, the stance and swing phases may be determined for the subject's hind limbs. The gait and posture analysis component 120 may calculate the velocity of the forelimbs and may infer that the foot is in the stance phase when the velocity falls below a threshold and may infer that the foot is in the swing phase when the velocity exceeds the threshold. The gait and posture analysis component 120 may determine that a foot contact event occurs in a video frame that causes a transition from the swing phase to the stance phase.

[0034] The gait and posture analysis component 120 may also determine the stride intervals represented in the period. The stride intervals may span multiple video frames of the video data 104. The gait and posture analysis component 120 may determine, for example, that a 10 second period has five stride intervals, and that one of the five stride intervals is represented in five consecutive video frames of the video data 104. In an exemplary embodiment, a left hind foot contact event may be defined as the event that separates / distinguishes the stride interval. In another exemplary embodiment, a right hind foot contact event may be defined as the event that separates / distinguishes the stride interval. In yet another exemplary embodiment, a combination of a left hind foot contact event and a right hind foot contact event may be used to define a separated stride interval. In some other embodiments, the gait and posture analysis component 120 may determine stance and swing phases for the forelimbs, may calculate foot velocity based on the forelimbs, and may further distinguish stride intervals based on foot contact events of the right and / or left forelimbs. In some other embodiments, the transition from stance to swing phase, i.e., toe off events, may be used to separate / distinguish stride intervals.

[0035] In some embodiments, it may be preferable to determine stride intervals based on hind leg contact events rather than forelimb contact events because the inference quality of the point data 112 for the forelimbs (as determined by the point tracking component 110) is less reliable in some cases. This may be a result of the difficulty in accurately locating the forelimbs because they are often more hidden than the hind legs in a plan view.

[0036] The gait and posture analysis component 120 may filter the determined stride intervals to determine which stride intervals to use to determine gait and posture metrics. In some embodiments, such filtering may remove erroneous or low confidence stride intervals. In some embodiments, criteria for removing stride intervals include, but are not limited to, low confidence point data estimates, physiologically unrealistic point data estimates, missing right hind foot contact events, and insufficient whole body velocity of the subject (e.g., velocity less than 10 cm / sec). In some embodiments, the filtering of stride intervals may be based on a confidence level in determining the point data 112 used to determine the stride intervals. For example, stride intervals determined with a confidence level below a threshold may be removed from the set of stride intervals used to determine gait and posture metrics. In some embodiments, the first and last strides are removed in a consecutive sequence of strides to avoid start and stop motions adding noise to the data to be analyzed. For example, a sequence of seven strides will have at most five strides used for analysis.

[0037] After determining the stride intervals represented in the video data 104, the gait and posture analysis component 120 may determine gait and posture metrics included in the gait measurement data 122. The gait and posture analysis component 120 may use the point data 112 to determine a step length for each of the stride intervals. In some embodiments, the point data 112 may be for the subject's left hind leg, left forelimb, right hind leg, and right forelimb. In some embodiments, the step length may be the distance between the left forelimb and the right hind leg for the stride interval. In some embodiments, the step length may be the distance between the right forelimb and the left hind leg for the stride interval. In some embodiments, the step length may be the distance the right hind leg travels beyond the ground contact position of the immediately preceding left hind leg.

[0038] The gait and posture analysis component 120 may use the point data 112 to determine the stride length for a stride interval. The gait and posture analysis component 120 may determine the stride length for each stride interval of the period. In some embodiments, the point data 112 may be for the left hindlimb, the left forelimb, the right hindlimb, and the right forelimb of the subject. In some embodiments, the stride length may be the distance between the left forelimb and the left hindlimb for each stride interval. In some embodiments, the stride length may be the distance between the right forelimb and the right hindlimb. In some embodiments, the stride length may be the total distance traveled by the left hindlimb for a stride from a toe-off event to a foot-contact event.

[0039] The gait and posture analysis component 120 may use the point data 112 to determine a step width for the stride interval. The gait and posture analysis component 120 may determine a step width for each stride interval for the period. In some embodiments, the point data 112 may be for the subject's left hind limb, left foreleg, right hind limb, and right foreleg. In some embodiments, the step width is the distance between the left foreleg and the right foreleg. In some embodiments, the step width is the distance between the left hind limb and the right hind limb. In some embodiments, the step width is the average lateral distance separating the hind limbs. This may be calculated as the length of the shortest line segment connecting the contact position of the right hind limb and a line connecting the toe-off position and subsequent foot-contact position of the left hind limb.

[0040] The gait and posture analysis component 120 may use the point data 112 to determine foot utterances for a stride interval. The gait and posture analysis component 120 may determine a foot velocity for each stride interval of the period. In some embodiments, the point data 112 may be for the left hind leg, the right hind leg, the left forelimb, and the right forelimb of the subject. In some embodiments, the foot velocity may be the velocity of one foot during a stride interval. In some embodiments, the foot velocity may be the subject's velocity and may be based on the base of the subject's tail.

[0041] Gait and posture analysis component 120 may use point data 112 to determine a stride velocity for a stride interval. Gait and posture analysis component 120 may determine a stride velocity for each stride interval of the period. In some embodiments, point data 112 may be for the base of the tail. In some embodiments, the stride velocity may be determined by determining a set of velocity data for the subject based on the movement of the base of the tail of the subject during a set of video frames representing a stride interval. Each velocity data in the set of velocity data may correspond to one video frame in the set of video frames. The stride velocity may be calculated by averaging (or otherwise combining) the set of velocity data.

[0042] The gait and posture analysis component 120 may use the point data 112 to determine a limb loading factor for each stride interval. The gait and posture analysis component 120 may determine a limb loading factor for each stride interval for the period. In some embodiments, the point data 112 may be for the right and left hind limbs of the subject. In some embodiments, the limb loading factor for the stride interval may be an average of the first loading factor and the second loading factor. The gait and posture analysis component 120 may determine a first stance time representing an amount of time that the right hind limb is in contact with the ground during the stride interval, and may then determine a first loading factor based on the first stance time and the length of the stride interval. The gait and posture analysis component 120 may determine a second stance time representing an amount of time that the left hind limb is in contact with the ground during the stride interval, and may then determine a second loading factor based on the second stance time and the length of the stride interval. In other embodiments, the limb loading factor may be based on the stance time and loading factor of the forelimb.

[0043] The gait and posture analysis component 120 may use the point data 112 to determine an angular velocity for each stride interval. The gait and posture analysis component 120 may determine an angular velocity for each stride interval for the period. In some embodiments, the point data 112 may relate to the base of the tail and the base of the neck of the subject. The gait and posture analysis component 120 may determine a set of vectors connecting the base of the tail and the base of the neck, where each vector in the set corresponds to one frame of the set of frames for the stride interval. The gait and posture analysis component 120 may determine an angular velocity based on the set of vectors. The vector may represent an angle of the subject, and the first derivative of the angle value may be the angular velocity for the frame. In some embodiments, the gait and posture analysis component 120 may determine the stride angular velocity by averaging the angular velocity for the frame for the stride interval.

[0044] Gait and posture analysis component 120 may determine the lateral displacement of the nose, tail tip, and tail base of the subject for each stride interval. Based on the lateral displacement of the nose, tail tip, and tail base, gait and posture analysis component 120 may determine a displacement phase offset for each subject body part. To determine the lateral displacement, gait and posture analysis component 120 may first determine a displacement vector for the stride interval using point data 112. Gait and posture analysis component 120 may determine a displacement vector for each stride interval in the period. In some embodiments, point data 112 may be for the mid-spine of the subject. A stride interval may span multiple video frames. In some embodiments, the displacement vector may be a vector connecting the mid-spine in the first video frame of the stride interval to the mid-spine in the last video frame of the stride interval.

[0045] Gait and posture analysis component 120 may use point data 112 and the displacement vector to determine the lateral displacement of the subject's nose for the stride interval. Gait and posture analysis component 120 may determine the lateral displacement of the nose for each stride interval in the period. In some embodiments, point data 112 may be for the mid-spine and nose of the subject. In some embodiments, gait and posture analysis component 120 may determine a set of lateral displacements of the nose, where each lateral displacement of the nose may correspond to one video frame of the stride interval. The lateral displacement may be the perpendicular distance of the nose from the displacement vector for the stride interval in each video frame. In some embodiments, gait and posture analysis component 120 may subtract the minimum distance from the maximum distance and divide it by the body length of the subject so that the displacement measured for the larger subject is equivalent to the displacement measured for the smaller subject.

[0046] The gait and posture analysis component 120 may use a set of lateral displacements of the nose over the stride interval to determine the nasal displacement phase offset. The gait and posture analysis component 120 may perform an interpolation using the set of lateral displacements of the nose to generate a smooth curve of lateral displacement of the nose over the stride interval, and then use the smooth curve of lateral displacement of the nose to determine when the maximum nasal displacement occurs during the stride interval. The gait and posture analysis component 120 may determine a percent stride position that represents the percent of the stride interval completed when the maximum nasal displacement occurs. In some embodiments, the gait and posture analysis component 120 may perform a cubic spline interpolation to generate a smooth curve for the displacement, and because of the cubic interpolation, the maximum displacement may occur at a time between video frames.

[0047] Gait and posture analysis component 120 may use point data 112 and the displacement vector to determine the lateral displacement of the base of the tail of the subject relative to the stride interval. Gait and posture analysis component 120 may determine the lateral displacement of the base of the tail for each stride interval in the period. In some embodiments, point data 112 may relate to the mid-spine and base of the tail of the subject. In some embodiments, gait and posture analysis component 120 may determine a set of lateral displacements of the base of the tail, where each lateral displacement of the base of the tail may correspond to one video frame of the stride interval. The lateral displacement may be the perpendicular distance of the base of the tail from the displacement vector for the stride interval in each video frame. In some embodiments, gait and posture analysis component 120 may subtract the minimum distance from the maximum distance and divide by the body length of the subject so that the displacement measured for the larger subject is equivalent to the displacement measured for the smaller subject.

[0048] The gait and posture analysis component 120 may use a set of multiple lateral displacements of the base of the tail versus stride interval to determine a tail base displacement phase offset. The gait and posture analysis component 120 may perform an interpolation using a set of multiple lateral displacements of the base of the tail to generate a smooth curve of the lateral displacement of the base of the tail versus stride interval, and then use the smooth curve of the lateral displacement of the base of the tail to determine when the maximum displacement of the nose occurred during the stride interval. The gait and posture analysis component 120 may determine a percent stride position that represents the percent of the stride interval completed when the maximum displacement of the base of the tail occurred. In some embodiments, the gait and posture analysis component 120 may perform a cubic spline interpolation to generate a smooth curve for the displacement, and because of the cubic interpolation, the maximum displacement may occur at a time between video frames.

[0049] Gait and posture analysis component 120 may use point data 112 and the displacement vector to determine the lateral displacement of the subject's tail tip relative to the stride interval. Gait and posture analysis component 120 may determine the lateral displacement of the tail tip for each stride interval in the period. In some embodiments, point data 112 may be for the subject's mid-spine and tail tip. In some embodiments, gait and posture analysis component 120 may determine a set of lateral displacements of the tail tip, where each lateral displacement of the tail tip may correspond to one video frame of the stride interval. The lateral displacement may be the perpendicular distance of the tail tip from the displacement vector for the stride interval in each video frame. In some embodiments, gait and posture analysis component 120 may subtract the minimum distance from the maximum distance and divide by the subject's body length so that the displacement measured for the larger subject is equivalent to the displacement measured for the smaller subject.

[0050] Gait and posture analysis component 120 may use the set of lateral displacements of the tail tip versus stride interval to determine the tail root displacement phase offset. Gait and posture analysis component 120 may perform interpolation using the set of lateral displacements of the tail tip to generate a smooth curve of lateral displacement of the tail tip versus stride interval, and then use the smooth curve of lateral displacement of the tail tip to determine when the maximum displacement of the nose occurred during the stride interval. Gait and posture analysis component 120 may determine a percent stride position that represents the percent of the stride interval completed when the maximum displacement of the tail tip occurred. In some embodiments, gait and posture analysis component 120 may perform cubic spline interpolation to generate a smooth curve for the displacement, and because of the cubic interpolation, the maximum displacement may occur at a time between video frames.

[0051] In some embodiments, the gait and posture analysis component 120 may include a statistical analysis component that may take the gait and posture metrics as input to perform some statistical analysis. The subject's build and the subject's speed may affect the subject's gait and / or posture metrics. For example, a faster moving subject may walk differently compared to a slower moving subject. As a further example, a subject with a larger body will walk differently compared to a subject with a smaller body. However, in some cases, differences in stride speed (compared to control subjects) may be a defining feature of gait and posture changes due to aging and frailty. The gait and posture analysis component 120 collects multiple repeated measurements for each subject (via the video data 104 and via the subject in an open area), with each subject having a different number of strides resulting in unbalanced data. Averaging over repeated strides results in one average value per subject, but can be misleading as it removes variability and introduces a false confidence level. At the same time, classical linear models do not distinguish between stable within-subject variability and between-subject fluctuations, which may bias the statistical analysis. To address these issues, the gait and posture analysis component 120, in some embodiments, employs linear mixed models (LMMs) to separate between-subject variability from genotype-based between-subject variability. In some embodiments, the gait and posture analysis component 120 may capture main effects such as subject size, genotype, age, etc., and may additionally capture random effects related to within-subject variability. The techniques of the present invention collect multiple repeated measurements at different ages of subjects, resulting in a nested hierarchical data structure. Examples of exemplary statistical models implemented in the gait and posture analysis component 120 are shown below as models M1, M2, and M3. These models follow standard LMM notation, where (genotype, body length, speed, test age) represent fixed effects, and (subject ID / test age) (where test age is nested within subjects) represents a random effect. M1: phenotype~genotype+test age+body length+(1|mouse ID / test age) M2: phenotype~genotype+test age+rate+(1|mouse ID / test age) M3: phenotype~genotype+test age+speed+body length+(1|mouse ID / test age)

[0052] Model M1 introduces age and body length as inputs, model M2 introduces age and speed as inputs, and model M3 introduces age, speed, and body length as inputs. In some embodiments, the models of gait and posture analysis component 120 do not include subject gender as an effect because gender may be highly correlated to subject length / build. In other embodiments, the models of gait and posture analysis component 120 may introduce subject gender as an input. Using point data 112 (determined by point tracking component 110) allows for determining subject build and speed for these models. Therefore, no additional measurements are required for these variables for the models.

[0053] One or more of the data included in the gait measurement data 122 may be circular variables (e.g., stride length, angular velocity, etc.), and the gait and posture analysis component 120 may implement functions of the linear variables using a circular-linear regression model. Linear variables such as body length and velocity may be included as covariates in the model. In some embodiments, the gait and posture analysis component 120 may implement a multivariate outlier detection algorithm at the individual subject level to identify subjects with injuries and developmental effects.

[0054] In some embodiments, gait measurement data 122 may include, for a subject, one or more of speed, velocity, angular velocity, step count, step length, step width, stride length, lateral displacement, limb loading factor, time symmetry, stride count relative to duration of footage, and distance covered. Table 1 provides a list of footage features and metrics used in one embodiment of the present invention.

[0055] Table 1 lists the video characteristics. [Table 1]

[0056] The angular velocity may be the first derivative of the subject's angle as determined by the vector connecting the base of the subject's tail to the base of its neck. The lateral displacement may be the difference between the minimum and maximum of the reference points (e.g., nose, base of tail, and tail tip) of the perpendicular distance from the subject's displacement vector of the stride for each frame of the stride, normalized by the subject's body length. The limb loading factor may be the amount of time that the paw is in contact with the ground divided by the total stride time calculated and averaged for each hind limb of the subject. The velocity may be determined using the base of the tail. The step length may be the distance traveled by the right hind limb past the previous contralateral limb contact point. The gait measurement data 122 may include two step lengths, one based on the left hind limb contact and the other based on the right hind limb contact. The step width may be the length of the shortest line segment connecting the right hind limb contact point and the left hind limb toe off and the subsequent foot contact point. Stride length may be the total distance traveled by the left hindlimb for a stride from toe off to foot contact. Time symmetry may be the difference in contact time between the left and right hindlimb divided by the total contact time. Strides may be the sum of all strides represented by the duration of the video data 104. Distance covered may be the sum of locomotor activity normalized by time spent in the open field arena.

[0057] As shown in FIG. 1, in some embodiments, the gait and posture analysis component 120 may also generate spine measurement data 124. Referring to FIG. 2, in step 208, the gait and posture analysis component 120 may process the point data 112 to determine the spine measurements of the subject. A spine mobility measurement(s) may be determined for each video frame of the video data 104. The spine mobility measurements of a video frame may include a number of different measurements. Using the point data 112, the gait and posture analysis component 120 may determine a first distance (dAC) between the base of the subject's head (point A) and the base of the subject's tail (point B). In some embodiments, the first distance may be normalized to the subject's body length. Using the point data 112, the gait and posture analysis component 120 may determine a second distance (dB) between the subject's mid-back (point B) and the midpoint of the line between the base of the head and the base of the tail (midpoint of line AC). Using the point data 112, the gait and posture analysis component 120 may determine the angle (aABC) formed by the base of the subject's head, base of the tail, and mid-back (points A, B, and C). The spine mobility measurements for a video frame may include the first distance, second distance, and angle discussed above. The gait and posture analysis component 120 may determine a spine mobility measurement for each video frame of the video data 104. The spine measurement data 124 may be a vector or matrix including the spine mobility measurements for each video frame of the video data 104.

[0058] When the subject's spine is straight, the first distance (dAC) and angle (aABC) may be at their maximum value and the second distance (dB) may be at their minimum value. When the subject's spine is flexed, the second distance (dB) may be at its maximum value while the first distance (dAC) and angle (aABC) may be at their minimum value. In determining the visual frailty score 162, the visual frailty analysis component 160 may consider the spine measurement data 124 over the entire duration of the video. The visual frailty analysis component 160 may be configured to identify that older subjects may have a lesser degree or less frequent bending of the spine due to reduced flexibility or spinal mobility. For each of the three spinal mobility measurements, the visual frailty analysis component 160 may determine the mean, median, standard deviation, minimum, and maximum of all video frames of the video data 104. In some embodiments, the visual frailty analysis component 160 may identify which video frames are non-ambulatory frames (i.e., frames in which the subject is not striding / walking). For such non-ambulatory frames, the visual frailty analysis component 160 may separately determine the mean, median, standard deviation, minimum, and maximum. The visual frailty analysis component 160 may be configured to identify a correlation between the spine measurement data 124 and the frailty of the subject. For example, in some cases, the median of the first distance (dAC) of the non-ambulatory frames and the median of the second distance (dB) of the non-ambulatory frames may increase (or decrease) with age.

[0059] As shown in FIG. 1, in some embodiments, the gait and posture analysis component 120 may also generate rearing event data 126. Referring to FIG. 2, in step 210, the gait and posture analysis component 120 may process the point data 112 to determine the rearing event data 126 for the subject. In some embodiments, a rearing event may be defined as when the subject's nose (or other body part) passes a threshold boundary / line on a wall of an open field. The gait and posture analysis component 120 may be configured to identify a threshold boundary on the wall and may use the point data 112 to identify when the subject's nose passes / is above the threshold boundary (e.g., a video frame of the video data 104). In other embodiments, a rearing event may be defined differently, for example, when the subject's paw passes a threshold boundary of a wall, when the subject spends a threshold amount of time in a corner of the open field, when the subject spends a threshold amount of time above the threshold boundary of a wall, etc.

[0060] The gait and posture analysis component 120 may be configured to use the coordinates of the boundary between the floor and the walls of the open field with a buffer of some pixels. Each time the subject's nose point passes through the buffer, this frame may be identified by the gait and posture analysis component 120 as including / representing a rearing event. Each uninterrupted series of video frames in which the subject exhibits a rearing event may be identified by the gait and posture analysis component 120 as a rearing behavior. In some embodiments, the gait and posture analysis component 120 may determine the total number of rearing behaviors, the average length of rearing behaviors, the number of rearing behaviors in the first few minutes (e.g., 5 minutes) of the video, and the number of rearing behaviors in the next few minutes (e.g., 5-10 minutes). The aforementioned measures may be included in the rearing event data 126. The visual frailty analysis component 160 may be configured to identify correlations between the rearing event data 126 and the frailty of the subject. For example, older / frail subjects may rear less (or more).

[0061] As shown in FIG. 1, in some embodiments, the gait and posture analysis component 120 may also generate hindlimb data 128. Referring to FIG. 2, at step 212, the gait and posture analysis component 120 may process the point data 112 to determine the hindlimb data 128. For each video frame of the video data 104, the gait and posture analysis component 120 may determine the distance between the coordinates of the hindlimb (from the point data 112). The hindlimb data 128 may be a vector including such distances for each of the video frames. The visual frailty analysis component 160 may determine the median, mean, standard deviation, maximum and / or minimum of the hindlimb distances for all video frames. The visual frailty analysis component 160 may be configured to identify a correlation between the hindlimb data 128 and the frailty of the subject. For example, the distance between the hindlimb of an elderly / frail subject may be smaller (or longer) than a control subject.

[0062] 1, in some embodiments, system(s) 105 may use ellipse data 132 and open field analysis component 140 to determine morphological data 142 of a subject. Figure 3 is a flow chart illustrating a process 300 for determining morphological data of a subject using ellipse data derived from video data, according to an embodiment of the present disclosure.

[0063] At step 302, the ellipse generation component 130 may process the video data 104 to determine ellipse data 132. In some embodiments, the ellipse generation component 130 may process the video data 104 to generate a segmentation mask that identifies an object in the video data 104 and then employ techniques to generate an ellipse fit / representation for the object. The ellipse generation component 130 may employ one or more techniques (e.g., one or more ML models) for object tracking in the video / image data and may be configured to identify the object (e.g., which pixels represent the object and which pixels represent the background). The segmentation mask generated by the ellipse generator 130 may identify object pixels (a set of pixels) that correspond to the object and may identify background pixels (another set of pixels separate and distinct from the object pixels) that correspond to the background. Using the segmentation mask, the ellipse generation component 130 may determine an ellipse fit. The ellipse fit may be an ellipse drawn around the body of the object. For different types of objects, system(s) 105 may be configured to determine different shape fits / representations (e.g., a circle fit, a rectangle fit, a square fit, etc.). Ellipse generation component 130 may determine an ellipse fit as a subset of object pixels. Ellipse data 132 may include this subset of pixels that corresponds to the ellipse fit. Ellipse generation component 130 may determine an ellipse fit for each video frame of video data 104. Ellipse data 132 may be a vector or matrix of ellipse fit pixels for all video frames of video data 104.

[0064] In some embodiments, the ellipse fit to the object may define several parameters of the object. For example, the ellipse fit may correspond to the position of the object and may include coordinates (e.g., x and y) that represent the pixel location (e.g., center of the ellipse) of the object in the video frame(s) of the video data 104. The ellipse fit may correspond to the length of the major axis and the length of the minor axis of the object. The ellipse fit may include the sine and cosine of the vector angle of the major axis. The angle may be defined relative to the direction of the major axis. The major axis may extend from the tip of the subject's head or nose to the end of the subject's body, such as the base of the tail. In some embodiments, the ellipse data 132 may include the aforementioned measurements for all video frames of the video data 104.

[0065] In some embodiments, the ellipse generation component 130 may use an encoder-decoder architecture to determine the segmentation mask from the video data 104. In some embodiments, the ellipse generation component 130 may use a neural network model to determine the ellipse fit from the video data 104.

[0066] The ellipse data 132 may also include a confidence score(s) of the ellipse generation component 130 in determining an ellipse fit for a video frame. The ellipse data 132 may alternatively include a probability or likelihood of an ellipse fit corresponding to the subject.

[0067] In some embodiments, ellipse generation component 130 may determine an ellipse fit of the object for each video frame of video data 104. The ellipse fit may be represented as a set of pixels that define an ellipse around the object. Ellipse data 132 may be a vector or matrix that includes the ellipse fit data for all of the video frames.

[0068] At step 304, the open field analysis component 140 may process the ellipse data 132 to determine morphological data 142. The morphological data 142 may correspond to the body composition (e.g., shape, size, length, weight, etc.) of the subject. The open field analysis component 140 may use the major and minor axis lengths of the ellipse fit to the subject (from the ellipse data 132) to determine an estimated length and an estimated width of the subject. The open field analysis component 140 may determine the major and minor axes of the ellipse fit for each video frame of the video data 104. In some embodiments, the open field analysis component 140 may determine a median, mean, standard deviation, maximum and / or minimum value for the major and / or minor axis lengths of all video frames of the video data 104. The open field analysis component 140 may estimate the length and width of the subject using one or more of the aforementioned calculations. In some embodiments, the morphological data 142 may include an estimated length and width of the subject. The morphological data 142 may additionally or alternatively include the lengths of the major and minor axes for each ellipse fit in each image frame of the image data 104 .

[0069] The visual frailty analysis component 160 may be configured to identify correlations between the morphological data 142 and the frailty of a subject. For example, changes in body composition and fat distribution may be observed as a subject ages.

[0070] In some embodiments, the open field analysis component 140 may determine other data that may be used by the visual frailty analysis component 160. For example, the open field analysis component 140 may use the ellipse data 132 (e.g., center pixel / location of the ellipse) to determine the video frames in which the subject is in the center of the open field arena and may determine the amount of time the subject spends in the center of the open field arena during the duration of the video. As another example, the open field analysis component 140 may use the ellipse data 132 to determine the video frames in which the subject is along the walls of the open field arena and may determine the amount of time the subject spends on the periphery (along the walls) during the duration of the video. As yet another example, the open field analysis component 140 may use the ellipse data 132 to determine the video frames in which the subject is in a corner of the open field arena and may determine the amount of time the subject spends in the corner(s) during the duration of the video. As yet another example, the open field analysis component 140 may use the ellipse data 132 to determine the average distance from the center of the open field arena to the subject's location during the video. As yet another example, the open field analysis component 140 may use the ellipse data 132 to determine the average distance of a subject's location from the perimeter / walls of the open field arena during video. As yet another example, the open field analysis component 140 may use the ellipse data 132 to determine the average distance of a subject's location from the corners of the open field arena during video.

[0071] As shown in FIG. 1, in some embodiments, the system(s) 105 may use a grooming behavior analysis component 150 to determine the behavioral data 152 of a subject. FIG. 4 is a flow chart illustrating a process 400 for determining the behavioral data 152 of a subject using the video data 104 of a subject, according to an embodiment of the present disclosure. At step 402, the grooming behavior analysis component 150 may process the video data 104 to determine the behavioral data 152 of the subject. The grooming behavior analysis component 150 may be configured to identify video frames of the video data 104 in which the subject exhibits grooming behaviors that may include at least one of paw licking, one-sided face washing, bilateral face washing, and flank licking. The grooming behavior analysis component 150 may process the video data 104 using one or more ML models to generate multiple predictions as to whether one or more frames of the video data 104 represent a subject exhibiting a prescribed behavior. These ML models may be constructed using training data that includes video capturing the movement of a subject(s), where the training data includes a label for each video frame that identifies whether the subject is exhibiting grooming behavior. Such ML models may be constructed using large training datasets.

[0072] Each ML model of the grooming behavior analysis component 150 may be configured with different initialization parameters or settings such that the ML models can have variations in terms of certain model parameters (learning rate, weights, batch size, etc.) and therefore result in different predictions (regarding the subject's grooming behavior) when processing the same video frames.

[0073] The grooming behavior analysis component 150 may also process different representations of the video data 104. The grooming behavior analysis component 150 may determine different representations of the video data 104 by changing the orientation of the video. For example, one orientation may be determined by rotating the video 90 degrees to the left, another orientation may be determined by rotating the video 90 degrees to the right, and yet another orientation may be determined by reflecting the video along a horizontal or vertical axis. The grooming behavior analysis component 150 may process the video frames in the originally captured orientation and other different generated orientations. Based on processing the different orientations, the grooming behavior analysis component 150 may generate different predictions regarding the grooming behavior of the subject. The grooming behavior analysis component 150 may use the different predictions determined as described above to make a final determination regarding whether the subject is exhibiting grooming behavior in the video frame(s).

[0074] The final decision may be output in behavioral data 152. Behavioral data 152 may be a vector or a set of values ​​indicating whether the subject exhibited grooming behavior in a particular video frame. For example, behavioral data 152 may include a Boolean value (e.g., 1 or 0, true or false, yes or no, etc.) for each video frame indicating whether the subject exhibited grooming behavior. As another example, behavioral data 152 may alternatively or additionally include a score (e.g., a confidence score, a probability score, etc.) corresponding to whether the subject exhibited grooming behavior in a particular video frame.

[0075] The grooming behavior analysis component 150 may use the video data 104 to determine a plurality of sets of frames, where the different sets (e.g., at least four sets) may represent different orientations of the video data. The first set of frames may be an original orientation of the video data 104 captured by the image capture device 101. The rotated set of frames may be a rotated orientation of the video data 104, e.g., the first set of frames may be rotated 90 degrees to the left to generate a set of rotated frames. The reflected set of frames may be a reflected orientation of the video data 104, e.g., the first set of frames may be reflected across a horizontal axis (or rotated 180 degrees) to generate a set of reflected frames. The other set of rotated frames may be another rotated orientation of the video data 104, e.g., the first set of frames may be rotated 90 degrees to the right to generate another set of rotated frames. In other embodiments, the set of frames may be generated by manipulating the original set of frames in other ways (e.g., reflected across a vertical axis, rotated a different number of degrees, etc.) In other embodiments, more or fewer orientations of the video data 104 may be processed by the grooming behavior analysis component 150.

[0076] The grooming behavior analysis component 150 may employ at least four ML models. As part of the processing of the video data 104, the grooming behavior analysis component 150 may use the same ML model to process different aforementioned sets of frames to generate different predictions. For example, a first ML model may process a first set of frames to generate a first prediction representing the probability or likelihood that the subject exhibits the grooming behavior during the first set of frames. The first ML model may process a rotated set of frames to generate a second prediction representing the probability or likelihood that the subject exhibits the grooming behavior during the rotated set of frames. The first ML model may process a reflected set of frames to generate a third prediction representing the probability or likelihood that the subject exhibits the behavior during the reflected set of frames. The first ML model may process another rotated set of frames to generate a fourth prediction representing the probability or likelihood that the subject exhibits the behavior during the other rotated set of frames. In this way, the same first ML model may process different orientations of the video data 104 and generate different predictions for the motion of the same captured object.

[0077] As part of further processing the video data 104, the grooming behavior analysis component 150 may process a different set of the aforementioned frames using another ML model to generate more predictions. For example, a second ML model may process the first set of frames to generate a fifth prediction representing the probability or likelihood that the subject exhibits the behavior during the first set of frames. The second ML model may process the rotated set of frames to generate a sixth prediction representing the probability or likelihood that the subject exhibits the behavior during the rotated set of frames. The second ML model may process the reflected set of frames to generate a seventh prediction representing the probability or likelihood that the subject exhibits the behavior during the reflected set of frames. The second ML model may process the other rotated set of frames to generate an eighth prediction representing the probability or likelihood that the subject exhibits the behavior during the video represented by the other rotated set of frames. In this manner, separate ML models may process different orientations of the video data 104 to generate additional predictions for the same captured object motion. The probabilities may be values ​​ranging from 0.0 to 1.0, or values ​​ranging from 0 to 100, or another numeric range.

[0078] Each of the different predictions (8 predictions) may be a data vector including multiple probabilities (or scores), each corresponding to a respective frame of the set and indicating the likelihood that the subject will exhibit grooming behavior in the corresponding frame. For example, the predictions may include a first probability corresponding to a first frame of the video data 104, a second probability corresponding to a second frame of the video data 104, etc.

[0079] In some embodiments, the set of video frames may include multiple video frames (e.g., 16 video frames), each video frame being video duration for a period of time (e.g., 30 milliseconds, 30 seconds, etc.). Each of the ML models may be configured to process the set of frames to determine the probability that the subject exhibits grooming behavior in the last frame of the set of frames. For example, if there are 16 frames in the set of frames, the output of the ML model indicates whether the subject exhibits grooming behavior in the 16th frame of the set of frames. The ML model may be configured to make a prediction of the last frame using contextual information from other frames in the set of frames. In other embodiments, the output of the ML model may determine the probability that the subject exhibits grooming behavior in another frame of the set of frames (e.g., an intermediate frame, an eighth frame, a first frame, etc.).

[0080] In some embodiments, the grooming behavior analysis component 150 may generate 32 different predictions corresponding to the frames by processing four different orientations / frame sets using four different ML models.

[0081] The grooming behavior analysis component 150 may include an aggregation component to process different predictions determined by different ML models using different sets of frames to determine a final prediction shown in the behavior data 152. The aggregation component may be configured to merge, aggregate, or otherwise combine the different predictions (e.g., the eight predictions described above) to determine the behavior data 152.

[0082] In some embodiments, the aggregation component may average the probabilities for each frame, and the behavioral data 152 may be a data vector of the average probabilities for each frame in the video data 104. In some embodiments, the grooming behavior analysis component 150 may determine a behavioral label for a frame (or number of frames) based on the corresponding average probability of the frames satisfying a condition (e.g., if the probability exceeds a threshold probability / threshold), and the behavioral label may be a Boolean value indicating whether the subject exhibited grooming behavior.

[0083] In other embodiments, the aggregation component may aggregate the probabilities of each frame, and the behavior data 152 may be a data vector of the summed probabilities for each frame in the video data 104. In some embodiments, the grooming behavior analysis component 150 may determine a behavior label for a frame based on the corresponding summed probability of the frame satisfying a condition (e.g., if the probability exceeds a threshold probability / threshold).

[0084] In some embodiments, the aggregation component may be configured to select, for each frame, the maximum value (e.g., highest probability) from the predictions as the final prediction for the frame, hi other embodiments, the aggregation component may be configured to determine the median value from the predictions as the final prediction for the frame.

[0085] In some embodiments, other components may be configured in a similar manner to the grooming behavior analysis component 150 to detect subjects exhibiting other predetermined behaviors. The other components may use more ML models to process the video data 104. These ML models may be configured to detect specific behaviors using training data including video capturing the movements of the subject(s), where the training data includes labels for each video frame that identify whether the subject is exhibiting a specific behavior. Such ML models may be configured using large training data sets. Based on the configuration of the ML models, they may be configured to detect different behaviors.

[0086] In other embodiments, the grooming behavior analysis component 150 may employ other techniques for determining the behavior data 152 .

[0087] The behavioral data 152 may also include the number / times during the duration of the video in which the subject exhibits grooming, the length of each grooming behavior (consecutive video frames in which the subject is grooming), the average length of the grooming behavior, the number of grooming behaviors during the duration of the video, and other metrics.

[0088] The visual frailty analysis component 160 may be configured to identify correlations between the behavioral data 152 and the frailty of a subject. For example, elderly / frail subjects may groom less (or more) than control subjects.

[0089] FIG. 5 is a flow chart illustrating a process for determining a visual frailty score using one or more of the data determined according to the process of FIGS. 2-4, according to an embodiment of the present disclosure. At step 502, the visual frailty analysis component 160 may process one or more of the determined data 122, 124, 126, 128, 142, 152 using one or more ML models. In some embodiments, the visual frailty analysis component 160 may employ one ML model to process all features / data 122, 124, 126, 128, 142, 152. In other embodiments, the visual frailty analysis component 160 may employ different / separate ML models to process each of the data 122, 124, 126, 128, 142, 152. For example, a first ML model may be used to process the gait measurement data 122 and a second ML model may be used to process the spine measurement data 124. In yet another embodiment, the visual frailty analysis component 160 may employ different / separate ML models to process the data 122, 124, 126, 128, 142, 152 based on how the data is derived and / or which components generate the data. For example, a first ML model may be used to process the gait measurement data 122, vertebral measurement data 124, rearing event data 126, and rear limb data 128 determined by the gait and posture analysis component 120, while a second ML model may be used to process the morphological data 142.

[0090] In some embodiments, the visual frailty analysis component 160 may select different features / data based on the subject's age, sex, breed, and / or other characteristics to determine the visual frailty score 162.

[0091] At step 504, the visual frailty analysis component 160 may determine a visual frailty score 162 for the subject. In some embodiments, the visual frailty analysis component 160 may determine different / multiple initial frailty scores based on processing different types of data input to the visual frailty analysis component 160 and then aggregate the different / multiple frailty scores to determine a final visual frailty score 162. In aggregating the results of processing the different types of data, the visual frailty analysis component 160 may use a weighted sum or weighted average technique, and the different types of data may have different corresponding weights. For example, the results of processing the morphological data 142 may be associated with a first weight, while the results of processing the vertebral measurement data 124 may be associated with another weight.

[0092] The visual frailty score 162 may be a numerical value within a predetermined range. For example, the visual frailty score 162 may be a value of 0 to 1, 0 to 10, 1 to 27, 0 to 100, etc.

[0093] In other embodiments, the visual frailty analysis component 160 may aggregate / combine the results of processing different types of data using separate ML models to determine the visual frailty score 162. In yet other embodiments, the visual frailty analysis component 160 may aggregate / combine the results of processing different types of data using a rule-based engine to determine the visual frailty score 162.

[0094] In yet other embodiments, the visual frailty analysis component 160 may be configured to use the point data 112 and / or the ellipse data 132 and may take into account the reliability of the point tracking component 110 and / or the ellipse generation component 130 in determining the visual frailty score 162.

[0095] In some embodiments, the visual frailty analysis component 160 may determine the visual frailty score 162 based on a comparison / assessment of the data 122, 124, 126, 128, 142, 152 with respect to several stored / control data. The visual frailty analysis component 160 may select the stored / control data based on the age, sex, lineage, and / or characteristics of the subject.

[0096] In some embodiments, the visual frailty analysis component 160 may determine the visual frailty score 162 based on which factors / features / data are visible / evident / detected for the subject. The visual frailty analysis component 160 may use the subject's data 122, 124, 126, 128, 142, and 152 to determine further factors. The visual frailty analysis component 160 may sum the number of factors detected and divide the sum by the number of total factors considered. For example, the visual frailty analysis component 160 may use the gait measurement data 122 to determine that the subject has a gait disorder and the morphological data 142 to determine that the subject has gained weight. The detection of gait disorder and weight gain may be two factors detected for the subject out of 10 potential factors. Based on this, the visual frailty analysis component 160 may determine the visual frailty score 162 to be 0.2 (2 / 10).

[0097] In some embodiments, visual frailty analysis component 160 may employ multiple different types of models / algorithms to process different types of data. For example, visual frailty analysis component 160 may include one or more of a linear regression model, a penalized linear regression model, a random forest, a support vector machine, a gradient boosting model, an extreme gradient boosting model, and a neural network.

[0098] Although FIG. 1 illustrates particular types of data, it should be understood that the visual frailty analysis component 160 may process additional or different types of data to determine the visual frailty score 162.

[0099] 1 may be determined by different components and / or using different data extracted from the video data 104. For example, morphological data may be determined using point data 112. As another example, hindlimb data may be determined using ellipse data 132. As another example, behavioral data may be determined using point data 112.

[0100] In some embodiments, the visual frailty analysis component 160 may be configured / trained using data corresponding to a manually generated frailty score. The manual frailty score may be generated by an observer / scorer by observing multiple different video of the subject. Some factors that an observer considers in generating the manual frailty score are listed in FIG. 15. The video of the subject may be annotated / labeled with the factors detected for the subject in generating the corresponding manual frailty score and / or the manual frailty score. The visual frailty analysis component 160 may be configured using such annotated video. The factors / data considered in generating the manual frailty score may be different from the factors / data used in generating the visual frailty score 162.

[0101] subject Some aspects of the invention include determining a visual frailty score of a subject. As used herein, the term "subject" may refer to a human, a non-human primate, a cow, a horse, a pig, a sheep, a goat, a dog, a cat, a pig, a bird, a rodent, or other suitable vertebrate or invertebrate. In certain embodiments of the invention, the subject is a mammal, and in certain embodiments of the invention, the subject is a human. In some embodiments, the subject used in the methods of the invention is a rodent, including but not limited to mice, rats, gerbils, hamsters, and the like. In some embodiments of the invention, the subject is a normal, healthy subject, and in some embodiments, the subject is known to have a disease or condition associated with frailty, or is at risk of having a disease or condition associated with frailty, or is suspected of having a disease or condition associated with frailty. Diseases associated with frailty may include clinical features / symptoms such as muscle weakness, loss of balance, abnormal muscle fatigue, muscle wasting, and the like. In certain embodiments of the invention, the subject is an animal model for a disease or condition associated with frailty. For example, and not intended to be limiting, in some embodiments of the invention, the subject is a mouse that is an animal model of aging and has characteristics of frailty such as one or more of muscle weakness, loss of balance, abnormal muscle fatigue, muscle wasting, etc.

[0102] As a non-limiting example, the subject evaluated by the methods and systems of the present invention may be a subject that is an animal model for a pathology, such as a model for one or more of aging, frailty, neurodegenerative disease, neuromuscular disease, muscle trauma, ALS, Parkinson's disease, multiple sclerosis, muscular dystrophy, etc. Such pathologies may be referred to herein as activity disorders.

[0103] In some embodiments of the method of the present invention, the subject is a wild-type subject. As used herein, the term "wild-type" refers to the phenotype and / or genotype of a species that occurs in nature in a typical form. In certain embodiments of the present invention, the subject is a non-wild-type subject, e.g., a subject that has one or more genetic modifications compared to the wild-type genotype and / or phenotype of the species of the subject. In some instances, the difference in the subject's genotype / phenotype compared to the wild-type is due to inherited (germline) variants or acquired (somatic) variants. Factors that can result in a subject exhibiting one or more somatic mutations include, but are not limited to, environmental factors, toxins, ultraviolet radiation, spontaneous errors occurring in cell division, radiation, maternal infection, chemicals, and other teratogenic events, but are not limited to these.

[0104] In certain embodiments of the method of the present invention, the subject is a genetically modified organism, also referred to as a genetically modified subject. A genetically modified subject may include preselected and / or deliberate genetic modifications, and therefore exhibit one or more genotypic and / or phenotypic traits that are different from those in non-modified subjects. In some embodiments of the present invention, by using conventional genetic engineering techniques, a genetically modified subject can be produced that exhibits genotypic and / or phenotypic differences compared to non-modified subjects of that kind. As a non-limiting example, a genetically modified mouse in which a functional gene product is absent or present at reduced levels in the mouse, and the method or system of the present invention can be used to evaluate the phenotype of the genetically modified mouse, and the results can be compared to those obtained from a control (control results).

[0105] In some embodiments of the present invention, a subject may be monitored using the visual frailty determination method or system of the present invention to detect the presence or absence of an activity-related impairment or pathology. In certain embodiments of the present invention, a test subject that constitutes an animal model of an activity and / or movement pathology may be used to evaluate the response of the test subject to the pathology. In addition, a test subject that constitutes an animal model of a movement and / or activity pathology may be administered a candidate therapeutic agent or method and monitored using the gait monitoring method and / or system of the present invention, and the results may be used to determine the effectiveness of the candidate therapeutic agent for treating the pathology. The terms "activity" and "behavior" may be used interchangeably herein.

[0106] As described elsewhere herein, the methods and systems of the present invention may be configured to determine a visual frailty score of a subject regardless of the subject's physical characteristics. In some embodiments of the present invention, one or more physical characteristics of the subject may be pre-specified characteristics. For example, and not intended to be limiting, the pre-specified physical characteristics may be one or more of body type, build, coat color, sex, age, and a phenotype of a disease or condition.

[0107] Diseases and Disorders The methods and systems of the present invention can be used to assess frailty, activity and / or behavior of a subject known to have, suspected to have, or at risk of having a disease or condition associated with frailty. It will be appreciated that in some cases, frailty is a condition associated with aging. For example, the subject may be an elderly subject and / or an animal model for an elderly condition. In certain embodiments of the present invention, frailty may be associated with a disease or condition that is not associated with aging but is not considered an elderly condition. For example, muscle weakness may be a feature assessed using the methods of the present invention and may be present in a young subject, a subject that is not an animal model for an elderly condition, a subject that is an animal model for an elderly condition, or an elderly subject. In some embodiments, the disease and / or condition is associated with an abnormally reduced level of activity or behavior, such as movement, muscle use, stamina, etc. In non-limiting examples, the test subject may be a subject with muscle wasting and muscle weakness, or a subject may be an animal model for a condition that exhibits muscle wasting and / or muscle weakness, etc. In either case, the method of the present invention can be used to assess the subject to determine the frailty status of the subject. The results of evaluating the test subject can be compared to a control result of the evaluation, non-limiting examples of the control subject being a subject that does not have the disease or condition of the model, a subject that does not have muscle wasting, a subject that does not have muscle weakness, etc. The control standard may be obtained from a plurality of subjects that do not have the condition, etc. The difference between the test subject results and the control results can be compared. Some embodiments of the methods of the present invention can be used to identify subjects that have a disease or condition associated with frailty.

[0108] The onset, progression, and / or regression of a disease or condition associated with frailty can also be assessed or tracked using embodiments of the methods of the invention. For example, in certain embodiments of the methods of the invention, two, three, four, five, six, seven, or more assessments of a subject are performed using the methods of the invention at different times. Comparison of the results of two or more assessments performed at different times can indicate a difference in the subject's frailty status (e.g., level of frailty). An increase in the determined level and / or characteristics of frailty exhibited by the subject may indicate the onset and / or progression in the subject of a disease or condition associated with frailty. A decrease in the determined activity level or type of activity may indicate regression in the subject of a disease or condition associated with the assessed activity. A determination that the subject has ceased activity may indicate that the disease or condition associated with the assessed activity has ceased in the subject.

[0109] Certain embodiments of the methods of the invention can be used to evaluate the efficacy of a therapy for treating a disease or condition associated with frailty. For example, a test subject may be administered a candidate therapy used to determine whether or not there is a change in frailty in the subject, as well as a method of the invention. A reduction in frailty determined in the subject after administration of the candidate therapy may indicate the efficacy of the candidate therapy against a disease or condition associated with frailty.

[0110] As shown elsewhere herein, the visual frailty analysis method of the present invention may be used to assess disease, pathology, or aging in a subject, and may also be used to assess animal models of disease, pathology, or aging. Many different animal models of disease, pathology, and aging are known in the art, including, but not limited to, many mouse models. The subject to be assessed using the system and / or method of the present invention may be an animal model for a disease or condition, such as, but not limited to, a model for a disease or condition such as a neurodegenerative disorder, a neuromuscular disorder, ALS, depression, hyperactivity disorder, anxiety disorder, muscle wasting disease, muscle injury, developmental disorder, Parkinson's disease, physical injury, etc.Additional models for diseases and disorders that may be evaluated using the methods and / or systems of the present invention are described in, for example, Dawson TM et al., Neuron Jun 10, 66(5):646-61 (2010); Cenci MA & A. Björklund Prog Brain Res. 252:27-59 (2020); Fleming SM et al., NeuroRx Jul;2(3):495-503 (2005); Falcim P. P, & GP Bates Mol. Biol. 1780:97-120 (2018); Nehru RR et al., Mammalian Genome Aug;30(7-8):173-191 (2019); Skov-Rizzo SJ & JN Crawley, Annual Rev. Animal Biosci. Feb 8, 5:371-389 (2017); Transicova A. et al., Prog Mol Biol Transl Sci. 100:419-82 (2011); Russell VA Curr Protoc Neurosci. January; Chapter 9: Unit 9.35 (2011); Leo, D. & RR Ganetdinov Cell Tissue Res. October; 354(1):259-71 (2013); Campos AC et al., Bratz J. Psychiatry 35 Suppl 2:S101-11 (2013); and Szechtman. .J. et al., Neurosci. May Rev; 76(Pt B); 254-279 (2017), are known in the art, the contents of which are incorporated herein by reference in their entireties.

[0111] In addition to testing subjects with known disease or disorder, the method of the present invention may also be used to evaluate new genetic variants, such as genetically engineered organisms.Thus, the method of the present invention can be used to evaluate genetically engineered organisms for one or more characteristics of disease or pathology.In this way, new organism strains, such as new mouse strains, can be evaluated, and the results can be used to determine whether the new strain is an animal model for disease or disorder.

[0112] Exemplary Devices and Systems One or more of the ML models of the automated visual frailty system 100 may take many forms, including a neural network. A neural network may include several layers, ranging from an input layer to an output layer. Each layer is configured to introduce a particular type of data as input and to output another type of data. The output from one layer is introduced as input to the next layer. Although the values ​​of the input / output data at a particular layer are unknown until the neural network actually operates at run time, the data describing the neural network describes the structure, parameters, and operation of the layers of the neural network.

[0113] One or more of the intermediate layers of a neural network may also be known as a hidden layer. Each node of the hidden layer is connected to each node of the input layer and to each node of the output layer. If a neural network includes multiple intermediate networks, each node of the hidden layer will be connected to each node in the next higher layer and in the next lower layer. Each node of the input layer represents a potential input to the neural network, and each node of the output layer represents a potential output from the neural network. Each connection from one node to another node in the next layer may be associated with a weight or score. The neural network may output a single output or a weighted set of possible outputs.

[0114] In one aspect, a neural network may be constructed with recurrent connections such that the output of a hidden layer of the network is fed back to the hidden layer for the next set of inputs. Each node of the input layer connects to each node of the hidden layer. Each node of the hidden layer connects to each node of the output layer. The output of the hidden layer is fed back into the hidden layer to process the next set of inputs. A neural network incorporating recurrent connections may be referred to as a recurrent neural network (RNN).

[0115] In some embodiments, the neural network may be a long short-term memory (LSTM) network. In some embodiments, the LSTM may be a bidirectional LSTM. A bidirectional LSTM implements inputs from two time directions, one from a past state to a future state and one from a future state to a past state, where the past state may correspond to features of the video data for a first time frame and the future state may correspond to features of the video data for a subsequent second time frame.

[0116] The processing by a neural network is determined by the learned weights of each node input and the structure of the network: given a particular input, the neural network determines the output, one layer at a time, until the output layer of the entire network has been calculated.

[0117] The connection weights may be initially learned by the neural network during training, where a given input is associated with a known output. In a set of training data, various training examples are fed into the network. In each example, the weight of the correct connection from the input to the output is typically set to 1, and all connections are given a weight of 0. When the training data examples have been processed by the neural network, the inputs may be sent to the network and compared with the associated outputs to determine how the network performance compares to a target performance. Training techniques such as backpropagation may be used to update the neural network weights to reduce errors made by the neural network when processing the training data.

[0118] Various machine learning techniques may be used to train and operate the model to perform various steps described herein, such as determining point data, determining ellipse data, determining behavioral data, and determining visual frailty scores. The model may be trained and operated according to various machine learning techniques. Such techniques may include, for example, neural networks (deep neural networks and / or recurrent neural networks, etc.), inference engines, trained classifiers, etc. Examples of trained classifiers include support vector machines (SVMs), neural networks, decision trees, AdaBoost (short for "adaptive boost") combined with decision trees, and random forests. Focusing on SVMs as an example, SVMs are supervised learning models with associated learning algorithms that analyze data to recognize patterns in the data, and are commonly used for classification and regression analysis. Given a set of training examples, each marked as belonging to one of two categories, the SVM training algorithm builds a model that assigns new examples to one category or the other, making it a non-probabilistic binary linear classifier. More complex SVM models may be built with a training set that identifies three or more categories, and the SVM determines which category is most similar to the input data. The SVM model may be mapped such that examples of distinct categories are separated by a clear gap. New examples are then mapped into the same space and predicted to belong to a category based on which side of the gap they fall on. The classifier may issue a "score" indicating which category the data most closely matches. The score may provide an index of how closely the data matches the category.

[0119] In order to apply machine learning techniques, the machine learning process itself needs to be trained. In this case, in order to train a machine learning component, such as one of the first model or the second model, a "ground truth" for the training examples needs to be established. In machine learning, the term "ground truth" refers to the accuracy of classification of a training set for supervised learning techniques. Various techniques may be used to train the model, including backpropagation, statistical learning, supervised learning, semi-supervised learning, probability learning, or other known techniques.

[0120] FIG. 6 is a block diagram conceptually illustrating a device 600 that may be used with the system. FIG. 7 is a block diagram conceptually illustrating exemplary components of a remote device, such as the system(s) 105, that may assist in processing video data, identifying subject behavior, and the like. The system(s) 105 may include one or more servers. As used herein, "server" may refer to a traditional server as understood in a server / client computing architecture, but may also refer to a number of different computing components that may assist in the operations described herein. For example, a server may include one or more physical computing components (such as a rack server) that are connected physically and / or via a network to other devices / components and may perform computing operations. A server may also include one or more virtual machines that emulate a computer system and run on one device or across multiple devices. A server may also include other combinations of hardware, software, firmware, or the like, for performing the operations described herein. The server may be configured to operate using one or more of a client-server model, a computer bureau model, grid computing technology, fog computing technology, mainframe technology, utility computing technology, a peer-to-peer model, sandbox technology, or other computing technologies.

[0121] Multiple systems 105 may be included in the overall system of the present disclosure, such as one or more systems 105 for performing point / body part tracking, one or more systems 105 for ellipse fitting / representation determination, one or more systems 105 for behavior classification, one or more systems 105 for determining a visual frailty score 150, etc. In operation, each of these systems may include computer readable and computer executable instructions that reside on the respective device 105, as described further below.

[0122] Each of these devices (600 / 105) may include one or more controller / processors (604 / 704), which may include a central processing unit (CPU) for processing data and computer-readable instructions, and a memory (606 / 706) for storing the respective device's data and instructions. The memory (606 / 706) may individually include volatile random access memory (RAM), non-volatile read-only memory (ROM), non-volatile magnetoresistive memory (MRAM), and / or other types of memory. Each device (600 / 105) may also include a data storage component (608 / 708) for storing data and controller / processor executable instructions. Each data storage component (608 / 708) may individually include one or more non-volatile storage types, such as magnetic storage, optical storage, solid-state storage, etc. Each device (600 / 105) may also be connected to removable or external non-volatile memory and / or storage (removable memory cards, memory key drives, network storage, etc.) via a respective input / output device interface (602 / 702).

[0123] Computer instructions for operating each device (600 / 105) and its various components may be executed by the respective device's controller / processor (604 / 704), using the memory (606 / 706) as temporary "working" storage during execution. The device's computer instructions may be stored in a non-transitory manner in non-volatile memory (606 / 706), in storage (608 / 708), or in an external device. Alternatively, some or all of the executable instructions may be embedded in hardware or firmware on the respective device in addition to or instead of software.

[0124] Each device (600 / 105) includes an input / output device interface (602 / 702). As described further below, various components may be connected via the input / output device interfaces (602 / 702). In addition, each device (600 / 105) may include an address / data bus (624 / 724) for transmitting data between the components of each device. Each component within the device (600 / 105) may also be directly connected to other components in addition to (or instead of) being connected to other components via the bus (624 / 724).

[0125] 6, device 600 may include an input / output device interface 602 for connecting to various components, such as an audio output component, such as a speaker 612, a wired or wireless headset (not shown), or other component that may output audio. Device 600 may additionally include a display 616 for displaying content. Device 600 may further include a camera 618.

[0126] Via antenna 614, input / output device interface 602 may be connected to one or more networks 199 via a wireless local area network (WLAN) (such as WiFi) radio, Bluetooth, and / or a wireless network radio, e.g., a radio capable of communicating with wireless communication networks such as a long-term evolution (LTE) network, a WiMAX network, a 3G network, a 4G network, a 5G network, etc. Wired connections such as Ethernet may also be supported. Via network 199, the system may be distributed across a network environment. I / O device interface (602 / 702) may also include communication components that allow data to be exchanged between devices, such as different physical servers, in a collection of servers or other components.

[0127] The components of device(s) 600 or system(s) 150 may include their own dedicated processors, memory, and / or storage. Alternatively, one or more of the components of device(s) 600 or system(s) 105 may utilize the I / O interfaces (602 / 702), processors (604 / 704), memory (606 / 706), and / or storage (608 / 708) of device(s) 600 or system(s) 105, respectively.

[0128] As mentioned above, multiple devices may be employed within a single system. In such a multi-device system, each of the devices may include different components for performing different aspects of the system's processing. Multiple devices may include overlapping components. The components of device 600 and system(s) 105 described herein are exemplary and may be deployed as stand-alone devices or may be included in whole or in part as components of a larger device or system.

[0129] The concepts disclosed herein may be applied within many different devices and computer systems, including, for example, general purpose computing systems, video / image processing systems, and distributed computing environments.

[0130] The above-described aspects of the present disclosure are meant to be illustrative. They are selected to illustrate the principles and applications of the present disclosure, and are not intended to be exhaustive or to limit the present disclosure. Many modifications and variations of the disclosed aspects may be apparent to those skilled in the art. Those skilled in the art of computer and voice processing will recognize that the components and process steps described herein may be interchangeable with other components or steps, or with combinations of components or steps, and still achieve the benefits and advantages of the present disclosure. Moreover, it will be apparent to those skilled in the art that the present disclosure may be practiced without some or all of the specific details and steps disclosed herein.

[0131] Aspects of the disclosed system may be implemented as a computer method or as an article of manufacture, such as a memory device or a non-transitory computer-readable storage medium. The computer-readable storage medium may be readable by a computer and may contain instructions for causing a computer or other device to perform the processes described in this disclosure. The computer-readable storage medium may be implemented by volatile computer memory, non-volatile computer memory, hard drives, solid-state memory, flash drives, removable disks, and / or other media. In addition, components of the system may be implemented as firmware or hardware.

[0132] Working Example Example 1. Development of an automated visual frailty index method mouse C57BL / 6J mice were obtained from the Nathan Shock Center at The Jackson Laboratory.

[0133] Open field assay and frailty indexing Open field behavior assays were performed as previously described [Kumar V. et al., PNAS 108, 15557-15564 (2011); Goiter B. et al., Communications Biology 2, 124 (2019); Bean G et al., A video-based phenotyping platform for laboratory mice. bioRxiv (2022)]. Mice were shipped from an aging colony from the Nathan Shock Center in a different room within the same animal facility at the Jackson Laboratory. Aged mice were acclimated for 1 week in an animal holding room adjacent to the behavioral testing room. During the day of open field testing, mice were acclimated to the behavioral testing room for 30–45 min before testing began. A 1-h open field test was performed as previously described. After open field testing, mice were returned to the Nathan Shock Center and manual frailty indexing was performed. Manual frailty indexing was performed within 1 week of the open field assay. The frailty indexing procedure was modified from that of Whitehead et al. [Whitehead JC et al., Journal of Gerontology, Biological Sciences and Medical Sciences 69, 621-632 (2014)]. Figure 17 shows the FI test sheet listing all the items of the manual frailty indexing.

[0134] Video, Segmentation, and Tracking The open field arena, video equipment, and tracking and segmentation network were as previously described [Goiter B. et al., Communication Biology 2, 124 (2019); Bean G et al., A video-based phenotyping platform for laboratory mice. bioRxiv(2022)]. The open field arena measured 20.5 inches by 20.5 inches with a Sentec (Omron Sentec, Kanagawa, Japan) camera mounted on a 40" monitor. The camera collected data at 30 frames per second (fps) with a resolution of 640 x 480 pixels (px). A neural network was used that was trained to generate a segmentation mask of the mouse to generate an elliptical fit of the mouse in each frame, as well as MouseTrack.

[0135] Pose Estimation and Walking A 12-point 2D pose estimate was generated using a deep convolutional neural network trained as previously described [(Shepard K. bioRxiv.doi.org / 10.1101 / 2020.12.29.424780 (2020)]. Points captured were nose, left ear, right ear, base of neck, left front paw, right front paw, mid-spine, left hind leg, right hind leg, base of tail, mid-tail, and tip of tail. Each point in each frame had an x-coordinate, y-coordinate, and a confidence score. A minimum confidence score of 0.3 was used to determine which points were included in the analysis.

[0136] Gait metrics were generated as previously described [Shepherd K. et al., bioRxiv.doi.org / 10.1101 / 2020.12.29.424780 (2020)]. Stride periods were defined by beginning and ending with the contact of the left hindlimb, tracked by pose estimation. These strides were then analyzed for several temporal, spatial, and whole-body coordination characteristics to generate gait metrics across the entire footage.

[0137] Open Field Metrics and Feature Engineering Open field metrics were derived from elliptical tracking of mice as previously described [Goiter B. et al., Communication Biology 2, 124 (2019); Goiter BQ et al., Elife 10 (2021); Bean G et al., A Video-Based Phenotyping Platform for Laboratory Mice. bioRxiv (2022)]. Tracking was used to generate features of locomotor activity and anxiety. Grooming was classified using an action detection network, as previously described. All other engineered features (spine mobility, anthropometric measurements, and hindlimb rearing) were derived using posture estimation data. Spine mobility metrics used three points from the posture at the base of the head (A), mid-back (B), and base of the tail (C). For each frame, we measured the distance between A and C (dAC), the distance between point B and the midpoint of the line AC (dB), and the angle formed by points A, B, and C (aABC). The mean, median, maximum, minimum, and standard deviation of dAC, dB, and aABC were determined across all frames and across non-ambulatory frames (when the animal was not ambulatory). For morphological indices, the distance between the two hind limb points in each frame was measured along with the mean, median, and standard deviation of that distance across all frames.

[0138] For rearing, we considered the coordinates of the boundary between the arena floor and the walls (using OpenCV contours) and added a 4-pixel buffer. Every time the mouse's nose point passed through the buffer, we counted this frame as a rearing frame. Each uninterrupted series of frames in which the mouse was rearing (nose crossing the buffer) was counted as a rearing behavior. The total number of behaviors, the average length of the behaviors, the number of behaviors in the first 5 min, and the number of behaviors within 5–10 min were calculated.

[0139] modeling The effect of scorer was investigated using a linear mixed model with scorer as a random effect and showed no significant effect on the variability of manual FI scores (RLRT = 183.85, p < 2:2e -16We found that 42% of the variance could be explained by the scorer (Figure 8C). A restricted likelihood ratio test (RLRT) [Crainiceanu C. M and Rupert D. Journal of the Royal Statistical Society, Series B (2004)] provides strong evidence of a scorer (random) effect with non-zero variance. A cumulative link model (log link) [Agresti A. Categorical Data Analysis (2003)] was fitted to the ordinal response (frailty parameters) with weight, age, and sex as fixed effects and tester as a random effect. The effects are the estimated variance associated with the random tester effect in the model (Y axis) across each FI item.

[0140] Tester effects were removed from FI scores using linear mixed models (LMM) with the lme4R package [Bates D. et al., Journal of Statistical Software Research 67, 1-48 (2015)]. The following model was fitted: y i j = μ i + ε i j , ε i j ~ N(0, σ 2 ), μ i ~ P ≡ N(0, τ 2 ) In the formula, y ij is the jth animal scored by tester i, and μ i is the tester-specific mean, and ε ij are the animal-specific residuals, and σ 2 is the within-tester variance and P was the distribution of tester-specific means. Four testers were used, with different numbers of animals tested by each tester, i.e., i = 1, ... , 4. Tester effects, estimated with the best linear unbiased predictor (BLUP), were subtracted from the animals' FI scores using restricted maximum likelihood estimators [Kenward MG & Roger JH, Biometrics 983-997 (1997)].

number

[0141] Tester-adjusted FI score

number

[0142] For FRIGHT modelling to predict years passed with manual FI items, frailty parameters with single values ​​were removed to avoid unstable model fits, i.e. zero-variance predictors. Ordinal regression models [McCollar, P. Journal of the Royal Statistical Society, Series B (1980)] were fitted without including any regularisation terms and a global likelihood ratio test (p<2.2e -16) was used to determine whether the video features showed any evidence of predicting each frailty parameter separately, i.e., evidence of a predictive signal. Ordinary regression models were then used with elastic net penalties [Zou H and Hastie T, Journal of the Royal Statistical Society, Series B (2005)] to predict frailty parameters using the video features.

[0143] To predict manual FI items, p i Frailty parameters were selected such that p1 < 0.80, where i is the mode of the count distribution of the parameter. For example, i = 1 is the mode of the count distribution of threat responses, and threat responses are excluded because p1 > 0.95.

[0144] Let X be the vector of covariates, C n 100(1 - α)% out-of-bag prediction interval I for the training set α (X, C n ) were obtained via quantile random forests [Meinshausen, N, Journal of Machine Learning Research, 7, 983-999 (2006)] and the grf package [Atti, S. et al., Annals of Statistics 47, 1148-1178 (2019)]. Prediction intervals generated using quantile regression forests are often at nominal levels, i.e.

number

[0145] We selected animals with an inverse relationship between age and FI score, i.e., young animals with higher FI score and old animals with lower FI score. Five test sets containing animals with these criteria were formed, and a random forest (RF) model was trained on the remaining mice. The prediction accuracy was evaluated to predict the FI score of the five test sets, and the results are displayed (Figure 21B). The test sets were age 100-2000 mice, ...L , age U , F.I. L , and F.I. U The FI cutoffs for young and old animals were defined using the following formula: For the five test sets, the parameters were set as follows: ageL = 60, ageU = 90, FI L = 0.20, and F.I. U = 0.15, ageL = 60, ageU = 100, FI L = 0.20, and F.I. U = 0.15 ageL = 50, ageU = 90, FI L = 0.20, and F.I. U =0.20, ageL = 60, ageU = 110, FI L = 0.20, and F.I. U = 0.20, ageL = 70, ageU = 100, FI L = 0.25, and FI U = 0.15

[0146] Data and Code Availability The code and models are available at github.com / KumarLabJax and www.kumarlab.org / data. The markdown files in the Github repository github.com / KumarLabJax / vFI-modeling contain details for reproducing the results in the manuscript and training the model for vFI / Age prediction. Manual FI scores and vFI features for all mice in the dataset can also be found. Code for genetically engineered features can be found at github.com / KumarLabJax / vFI-features.

[0147] result Data collection Figure 8A shows that 451 individual C57BL / 6J mice (256 males, 195 females) were evaluated, and for 117 mice, the test was repeated a second time after 5 months, resulting in a dataset of 568 mice ranging in age from 8 to 148 weeks. Top-down footage of each mouse in the open field for 1 h was collected as described in the methods herein above (Figure 8A) according to previously published protocols [Kumar V. et al., PNAS 108, 15557-15564 (2011); Goiter B. et al., Communications Biology 2, 124 (2019)]. After 1 h in the open field, each mouse underwent standard mouse frailty indexing by trained experts at the Nathan Shock Center on Aging to assign a manual FI score. During the course of data collection, manual FI was performed by four different scorers. The open field footage was processed by tracking and pose estimation networks to generate tracks for each frame, ellipse fits, and a 12-point pose for the mouse [Goiter, B. et al., Communications Biology 2, 124 (2019); Shepherd, K. et al., bioRxiv.doi.org / 10.1101 / 2020.12.29.424780 (2020)]. Using these frame-by-frame measurements, we computed a variety of video-based features (all extracted video-based features are listed and defined in Table 1 along with their source) including traditional open field measures such as anxiety, hyperactivity, etc. [Goiter B. et al., Communication Biology 2,124 (2019)], neural network-based grooming [Goiter B. et al., bioRxiv doi.org / 10.1101 / 2020.10.08.331017 (2020)], and a novel gait measure [Shepherd K. et al., [g19]bioRxiv[ / g19]doi.org / 10.1101 / 2020.12.29.424780 (2020)]. Video-based features for each mouse were then analyzed using penalized linear regression (LR) *) were used as features in an array of machine learning models, including [Zou H. and Hastie T., Journal of the Royal Statistical Society Series B, Statistical Methodology 67, 301-320 (2005)], Random Forest (RF) [Brayman L., Machine Learning 45, 5-32 (2001)], Support Vector Machine (SVM) [Cortez C. and Vapnik V., Machine Learning 20, 273-297 (1995)], and Extreme Gradient Boosting (XGB) [Friedman JH, Statistical Annals 1189-1232 (2001)]. Manual FI scores were used as the response variable for the models. As expected, the mean FI scores increased with increasing age (Figure 8B). Heterogeneity in FI scores (indicated by standard deviation bars) also increased with age. For the data obtained in this study, the submaximal limit of the FI score was found to be slightly below 0.5, which falls within the submaximal limit previously shown in mice [Whitehead JC et al., Journal of Gerontology, Biological and Medical Sciences 69, 621-632 (2014); Lockwood K. et al., Scientific Reports 7, 43068 (2017)]. These results indicated that the FI data obtained in this study are typical of other mouse data and reflect the characteristics of human FI, with increasing mean FI scores and heterogeneity of FI scores with age [Lockwood K. et al., Scientific Reports 7, 43068 (2017)]. Visual inspection of the scorers indicated that there may have been a scorer-dependent effect in the manual FI. For example, scorer 1 and scorer 2 tended to generate high and low frailty scores, respectively. The effect of scorer was investigated using a linear mixed model with scorer as a random effect and was found to account for 35% of the variability in the dataset (RLRT = 66.41, p < 2.2e -16) was found to be explained by the scorer (Figure 8C). A restricted likelihood ratio test (RLRT) [Crainiceanu CM and Rupert D., Journal of the Royal Statistical Society Series B Statistical Methodology 66, 165-185 (2004)] showed strong evidence of scorer (random) effects with non-zero variance, suggesting that scorer variability is an important source of variation in the data and should be adjusted for before modeling.

[0148] The overall approach is illustrated in Figure 8A. The study was conducted with 643 data points (371 males, 272 females) over three rounds of testing with 533 unique mice. The first round of testing (Batch 1) was conducted with 222 mice (141 males, 81 females). The second round of testing (Batch 2) occurred approximately 5 months later and was conducted with 319 mice (173 males, 146 females). Of these mice, 105 were repeated from the first batch. The third round of testing (Batch 3) was conducted approximately 1 year later with 102 mice (57 males, 45 females). Of these mice, 18 were previously tested in the first round and 15 were tested in the second round. Top-down footage of each mouse in a 1-h open-field session was collected according to previously published protocols [Kumar V. et al., PNAS 108, 15557-15564 (2011); Goiter B. et al., Communications Biology 2, 124 (2019)] (see Methods and Figure 8A for examples of young and aged mice). After the open field, each mouse was assigned a manual FI score by trained experts at the Nathan Shock Center for Aging, who scored each mouse using the standard Mouse Frailty Index [Skov-Rizzo SJ et al., Current Protocols in Mouse Biology (2018)] (Figure 19A). Bimodality of the data (Hartigan's test [Hartigan JA and Hartigan PM, Annals of Statistics (1985)], D = 0.07; p < 2.2e -16Despite this, we found that Simpson's paradox [Simpson EH, Journal of the Royal Statistical Society, Series B (1951)] did not appear in any of the top 15 features in the data (Figure 20). Open field footage was processed by a tracking network and a pose estimation network to generate mouse tracks, ellipse fits, and 12-point poses for each frame [Goiter B et al., Communication Biology 2, 124 (2019); Shepherd K. et al., Cell Reports (2022)]. These frame-by-frame measurements were used to calculate a variety of features for each video, including traditional open field indices of anxiety and hyperactivity [Goiter B. et al., Communication Biology 2, 124 (2019)], grooming [Goiter BQ et al., Elife (2021)], gait and posture indices [Shepherd K. et al., Cell Reports (2022)], as well as genetically engineered features. Described herein are features used to train machine learning models to predict chronological and biological age, as well as visual FI (vFI).

[0149] Consistent with previous data, in our data set, the mean FI score increases with age (Figure 8B). Heterogeneity of FI scores (indicated by standard deviation bars) also increases with age. The submaximal limit of FI scores was found to be slightly below 0.5 for data that fell within the submaximal limit shown for mice [Whitehead JC et al., Journal of Gerontology, Series A (2014); Lockwood K. et al., Scientific Reports (2017)]. These results indicate that the FI data are typical of other mouse data and reflect the characteristics of human FI, with an increase in the mean FI score and an increase in FI score heterogeneity with age [Lockwood K. et al., Scientific Reports (2017)]. During the course of data collection, manual FI was performed by four different scorers. Visual inspection of the data showed a scorer effect on the manual FI scores (Figure 8C). For example, scorer 1 and scorer 2 tended to generate high and low frailty scores, respectively (Figure 18B). Modeling showed that 42% of the variance in manual FI scores was attributable to scorer effects (RLRT = 183:85, p < 2.2e -16 ). A closer examination of which FI items were most affected by the scorer revealed that hair ruffles, kyphosis, and vision were the most subjective (Figure 18A). This analysis suggests that scorer effects are an important source of variability in mouse clinical FI.

[0150] Feature Extraction Per-video features were extracted using frame-by-frame segmentation, ellipse fitting, and 12-point pose coordinates. Extracted features, including description and source of measurements, are listed in Table 1. Overall, there was a very high correlation between median and mean video metrics (Figure 9A-B). Only medians were used in modeling for two reasons: medians tend to have a higher correlation with FI scores than means, and medians were more stable to outlier effects than means. Similarly, interquartile ranges were used as model features, rather than standard deviations when available, because interquartile ranges tend to be more stable to outliers than standard deviations. This generated a total of 44 video features (Figure 14, Table 1). Metrics obtained in standard open-field assays, such as total locomotor activity, time spent peripheral vs. central, and grooming behavior (Figure 9A), were considered. Standard open-field metrics showed low correlations with both FI scores and age (Figures 14 and 15).

[0151] In addition to the existing features, we designed a set of features that were hypothesized to correlate with FI. These features included morphometric features capturing the shape and size of the animal, as well as behavioral features related to flexibility and vertical locomotion. Age-related changes in body composition and fat distribution have been observed in humans and rodents [Pappas L. and Nagy T., European Journal of Clinical Nutrition 73 (October 2018)]. We hypothesized that measurements of body composition may indicate some signal of aging and frailty. The major and minor axes of the ellipse fitted to the mouse in each frame were used as the estimated length and width of the mouse, respectively (Figure 10B). The distance between the hindlimb coordinates in each frame was taken as another width measurement closer to the hip. The mean and median of ellipse width, ellipse length, and hindlimb width across all frames were used as metrics per video. Many of these morphological features were highly correlated with FI score and age (FIGS. 14 and 15), e.g., median hindlimb width in particular correlated with r=0.56 and 0.57, respectively (FIG. 10C).

[0152] Changes in gait have been shown to be a hallmark of aging in humans [Zou, Y. et al., Scientific Reports 10, 4426 (2020); Skiadopoulos A. et al., J. Neuroeng Rehab. 17, 41 (2020)] and mice [Tarantini S. et al., Journal of Gerontology, Biological Sciences and Medical Sciences 74, 1417-1421 (2018); Baer W.-N. et al., Journal of Gerontology, Biological Sciences and Medical Sciences 74, 1413-1416 (2019)]. To explore gait changes associated with aging in the current cohort of mice, we conducted analyses similar to the methods used to extract gait metrics from mice moving freely in the field (Figure 10D-E) [Shepherd K. et al., bioRxiv. doi.org / 10.1101 / 2020.12.29.424780 (2020)]. Each stride was analyzed for its spatial, temporal, and whole-body coordination metrics (Figure 10D), resulting in an array of measurements in which the median of all strides for each mouse was taken. Intra-mouse heterogeneity of gait features was also examined using standard deviations and interquartile ranges across all strides for each mouse. Many of these calculated metrics showed high correlations with FI scores and age (Figures 14 and 15), for example, median step width and tip-to-caudal lateral displacement interquartile range (r=0.58 and r=0.63, respectively) (Figure 10E).

[0153] Next, we investigated spine flexion throughout the footage. We hypothesized that older mice would flex their spine less or less frequently due to reduced flexibility or spinal mobility. That change in flexibility could be captured by the postural estimate coordinates of three points on the mouse in each footage frame: the back of the head (A), the middle of the back (B), and the base of the tail (C). At each frame, we calculated the distance between points A and C normalized to the mouse length (dAC), the orthogonal distance of the middle of the back B from a line (dB), and the angle of the three points (aABC) (Figure 10F). For each of the three frame-by-frame metrics (dAC, dB, and aABC), we calculated the mean, median, standard deviation, minimum, and maximum for all frames and non-ambulatory frames (frames in which the mouse was not in a stride state) for each footage. We found a moderately high correlation between spine flexion and FI scores, contradicting our hypothesis (Figures 14 and 15). That is, median dB and median dAB (for non-ambulatory frames) were expected to decrease with age, but instead were found to increase (r=0.51 and 0.35, respectively) (Figure 10G). One possible reason for this result was that very frail mice spent more time grooming. However, neither grooming behavior nor grooming seconds showed any relationship with FI score or median dB. Another possibility was that highly frail mice spent less time walking and more time walking, since there was little relationship between number of steps or distance traveled and FI score or median dB. Highly frail mice may also have had a higher median dB due to body composition, since median dB has a correlation of 0.496 with body weight. It is also important to note that these bending metrics cast a wide net; that is, they are a cheap, general description of all spine activity during the 1-h open field. Thus, these indices may have captured interactions between body composition and behavior.

[0154] Previous spine flexibility indices examined lateral spine flexibility, but vertical flexibility may also have a relationship with frailty. To investigate this, we examined the occurrence of wall-supported rearing (Figure 10H). We hypothesize that more frail mice may have poorer rearing abilities due to reduced lateral spine mobility and / or reduced exploratory activity. We took the edge of the open field and added a 5-pixel buffer as a boundary. Frames in which the mouse's nose coordinates crossed that boundary were used as instances of rearing. From these heuristics, we determined the number of rearings and the average length of each rearing event (Table 1). Several metrics related to rearing behavior provide signals of frailty, particularly the total number of rearings and the first 5 min of rearing (r=0.2 and 0.3, respectively, Figure 10I).

[0155] Interestingly, the correlation with age was generally slightly higher than with FI scores (Figures 14 and 15). This may be due to differences in how mice become frail. Some mice may be more frail but have no impairment in stride width, whereas older mice on average may have greater variation in stride width regardless of their frailty. Also noteworthy is the increased heterogeneity in many of these indices with both age and FI scores (e.g., median stride width, median step width, median dB).

[0156] sex To analyze gender differences in frailty, FI score data were stratified into the four age groups, and box plots were compared between males and females for each age group (Figure 11A). The oldest age group contained 81 males and only 9 females. The range of frailty scores for females in each age group tended to be slightly lower than that of males, except for the oldest age group. The two intermediate age groups showed highly significant differences in distribution between males and females.

[0157] Comparison of the correlations between FI item scores and age for males and females (Figure 11B) showed an overall high correlation (r = 0.85). The average difference in correlations between FI index items and age between males and females was 0.08, but there were several index items that showed significant differences. Alopecia and threat reaction had the highest gender differences in correlation with age (0.29 and 0.21, respectively), with females having a high correlation for alopecia and males having a high correlation for threat reaction (Figure 16).

[0158] Correlations of male and female video features with both FI score and age were also high (r=0.88 and r=0.90, respectively), with mean differences between male and female correlations of video metrics with FI score and age of 0.14 and 0.13, respectively (Figures 14 and 15). For both FI score and age, video features with the highest gender differences were gait metrics related to stride and step length, lateral displacement of the base of the tail, and lateral displacement of the tip of the tail. The highest gender differences were the correlations between median lateral displacement of the base of the tail to age (difference of 0.57) and median lateral displacement of the tip of the tail to age (0.50), with females tending to have higher correlations to both FI score and age. For metrics related to stride and step length (median stride length, difference of 0.33), males were more highly correlated with FI score and age than females. These results indicate that with age, females significantly increase the lateral displacement of the tail base and tail tip during walking, while males show little change in this trait and males show a greater decrease in stride length with age compared to females.

[0159] Prediction of age and frailty index from video data Once the video features described herein were confirmed to correlate with aging and frailty, these features were used as covariates in models predicting age and manual FI scores (Figure 12A, vFRIGHT, and vFI, respectively). Age is an empirical truth and has a strong relationship with frailty. Prediction of age using video features (Figure 12A, model vFRIGHT) is compared to prediction of age using manual FI items, i.e., a method called the FRIGHT age clock [Schultz MB et al., Nature Communications (2020)] (Figure 12A, model FRIGHT). Four models were initially tested: Penalized Linear Regression (LR*) [Zou H and Hastie T, Journal of the Royal Statistical Society, Series B (2005)], Support Vector Machine (SVM) [Cortes C. Vapnik V. Machine Learning (1995)], Random Forest (RF) [Bryman I., Machine Learning (2001)], and Extreme Gradient Boosting (XGB) [Friedman J. H, Annals of Statistics (2001)] (Figure 12B, panel 1). The Random Forest regression model had the lowest mean absolute error (MAE) (p<2.2e) when compared using repeated measures ANOVA. -16 , F 3,147 =190.43), Root Mean Square Error (RMSE)(p<2.2e -16 , F 3,147 =59.53), and the highest R2 (p<2.2e -16 , F 3,147 The vFRIGHT model was chosen to predict age for unseen future data because it performed better than other models with a mean MAE of 1.0 ± 0.99 weeks (p < 4.7e-0.99) compared to the FRIGHT clock with FI items (15.7 ± 4 weeks) (Figure 12B, panel 1; Figure 19C). The vFRIGHT model was able to predict age more accurately and precisely than the FRIGHT clock. vFRIGHT performed better (p < 4.7e-0.99) with a lower MAE (13.1 ± 0.99 weeks) compared to the FRIGHT clock with FI items (15.7 ± 4 weeks). 5 , F 1,49= 19.9, using repeated measures ANOVA) (Figure 12B, panel 2). RMSE (RMSEvFRIGHT = 17.97 ± 1.44, RMSEFRIGHT = 20.62 ± 4.78, p < 6.1e -7 , F 1,49 =32.84) and R2 (RMSEvFRIGHT=0.78±0.04, RMSEFRIGHT=0.76±0.07, p<2.1e -8 , F 1,49 = 44.54) and found a similar significant improvement in age prediction when using video features (Figure 19D). The variance of prediction error was significantly reduced for video-based age prediction (vFRIGHT) compared to manual FI product-based age prediction (FRIGHT) (Figure 12B, panel 2). Predicted vs. actual values ​​were plotted for the training and test sets of the vFRIGHT model (Figure 12G) and the FRIGHT model (Figure 19G). Taken together, these results suggest that the automated videos contain more accurate information about aging than is addressed in the manual FI items. Video features may also provide aging information that overlaps with health disorders scored in the manual FI.

[0160] To address this, we predicted individual FI items using video features (Figure 12A). Of the 27 items, many had scores that were nonzero to near-zero, indicating that in a genetically homogenous dataset, at least the majority of the information in the manual FI comes from a subset of index items (Figure 19F). Only index items with a balanced ratio of scores between 0 and 0.5 and 1 were selected for prediction (Figure 12C). Classifiers were then built for each of the nine index items to predict scores given mouse video features. Scores for individual FI items were predicted using a normal elastic net regression model. For all nine, scores were predicted with accuracy that exceeded what would be expected by guessing randomly (Figure 12C, dotted lines are guess accuracy). Many of these FI items have implicit relationships with video features such as grooming (e.g., coat condition, alopecia), gait / mobility (e.g., gait disorder, kyphosis), and body composition (e.g., abdominal distension, body condition). In the FRIGHT model, gait disturbance, kyphosis, and hair ruffled contributed most to age prediction in the dataset, followed by abdominal swelling and body condition (Figure 19B) - all items for which video features were able to predict the score (Figure 12C). Together, these results indicate that most of the information about aging and frailty comes from a small subset of manual FI items, and that video data can be used to predict the information within this subset. Furthermore, because aging was predicted more accurately and precisely with video data than with manual FI items, video data may also contain additional signals for aging.

[0161] We next address the goal of vFI (Figure 4A, Model vFI): predicting manual FI scores using video data. Similar to vFRIGHT modeling, the random forest regression model achieved the lowest mean absolute error (MAE) (p<2.1e -15 , F 3,147 =30.53), root mean square error (RMSE)(p<8:3e- 14 , F 3,147 =26.62), and the highest R2 (p<4.7e -14 , F 3,147= 27.2) (Figure 12D, Figure 19E). The model was able to predict FI scores within 0.04 ± 0.002 of the actual FI score (FI scores have a possible range of 0 to 1 in the data set where a range of 0.04 to 0.47 was found). This error was such that one item of the FI was incorrectly scored at 1 point, or two items at 0.5 points, indicating the stability of the model. Residuals were plotted for the training and test sets of the model (Figures 12F, 12G, and 19G). The residuals calculated from the training data show that their distribution is symmetrical around zero for both models, with most residuals around the black diagonal line. The residuals for the test set follow a similar pattern. It is concluded that the video-generated features described herein can be successfully used for automated frailty scoring. Age has a higher correlation with manual FI scores of r = 0.81 than any of the video features. Thus, higher prediction accuracy is obtained when using models with only age as a feature (Figure 21A). The model using both video features and age (AllRF) is notably better than the model with age only, indicating that video features provide important information about frailty (Figure 21A). In particular, when looking at mice whose FI scores deviate from the age group, i.e., young mice with high frailty and older mice with low frailty, the vFI model (VideoRF) performs significantly better than the model using age and even better than the model using video features + age (AllRF) (Figure 21B). This indicates that video features provide better information than age for mice that are outliers in the age group.

[0162] Finally, to see how much training data is actually needed for high-performance prediction with vFI and vFRIGHT, we performed simulation studies in which different percentages of the total data were allocated to training (Figure 21E). We found that training sets of <80% of the current dataset achieved similar performance, but decreasing training set size below this showed a general downward performance trend. Since open field testing can sometimes be performed in less than an hour, we next investigated the accuracy drop-off using shorter videos by truncating the videos to the first 5 minutes and the first 20 minutes for vFI prediction (Figure 21D). Features associated with the 60-minute videos yielded the highest accuracy for vFI prediction using LMM with "simulation" as a random effect, the lowest MAE, and F 2,98 =178.39, p<2.2e -16 ,Lowest RMSE,F 2,98 =156.93, p<2.2e -16 ), i.e., the highest R2(p<2.2e -16 , F 2,98 = 297.3). When the length of the open field test was shortened from 60 to 20 min of footage, a significant decrease in performance accuracy was observed (LMM-MAE with post-hoc pairwise comparisons, t 98 =14.82, FDR adjusted p<0:0001, RMSE, t 98 =13.69, FDR adjusted p<0:0001, R 2 , t 98 =-19:22, FDR adjusted p<0:0001). Based on the experiments, it was concluded that 60-minute video-generated features provide the most accurate vFI prediction, but even with 80% of the video there is no substantial loss in prediction accuracy.

[0163] Quantifying uncertainty in frailty index predictions In addition to quantifying the average accuracy, the error was also investigated in more detail within the dataset. The prediction error was quantified by providing a prediction interval (PI) that yields a range of values ​​including FI scores with unknown age and a specified level of confidence based on the same data that gave the random forest point predictions [Zhang H. et al., Am. Statistics. 74, 392-406 (2020)]. One approach to obtain random forest-based prediction intervals was to use generalized random forests to model the conditional distribution of FI given the features, as previously described [Meinshausen N., Journal of Machine Learning Research, 7, 983-999 (2006); Atai S. et al., Annals of Statistics, 47, 1148-1178 (2019)]. For animals in the test set, a quantile-based generalized random forest was used to provide point predictions (age response) and prediction intervals for FI scores. This yielded a range of FI (age-responsive) values ​​that included unknown FI scores (respiration) with 95% confidence. (Figure 12J-I). The mean PI width of predicted FI scores for all tested animals was 5.72 ± 1.49 (80.29 ± 16.8 for response-predicted age) and PI lengths ranged from 2.3 to 8.5 (28 to 114 times for age), highlighting that PI width was animal and age group specific. A smoothed regression fit of PI width against age was plotted, showing that the width increased with the age of the animals (Figure 12G-H). The variability of the 95% PI widths shown in Figure 124G-H (right panels) showed higher variability for animals belonging to the middle-aged group (M, pink). Beyond simple point predictions, prediction intervals of the frailty index (PI) are provided to quantify the uncertainty of the predictions and allow FI scores and ages to be pinpointed for some animals with greater accuracy than others.

[0164] Importance of characteristics in frail and healthy animals A useful visual FI (vFI) should rely on several features that capture the intrinsic frailty of an animal and are simultaneously interpretable. Two approaches were used to identify features that are important for making vFI predictions using a trained random forest model: (1) feature importance, and (2) feature interaction strength. Feature importance is a measure of how frequently the random forest model uses a feature at different depths in the forest. Higher importance values ​​indicate that the feature is at the top of the forest and therefore is important for building a predictive model. In the second approach, a total interaction measure was derived that indicates the extent to which a feature interacted with all other model features.

[0165] Comparison of feature importance between the vFI and vFRIGHT models (Figure 11A) reveals that many of the most important video features for the models are shared, but show some important differences (Figure 21C). For example, step width IQR is much more important for vFI than for vFRIGHT, and apical-caudal lateral displacement (LD) IQR is much more important for vFRIGHT than for vFI. A more complete picture of feature importance was obtained by modeling three different quantiles of the conditional distribution of FI scores. The three quantiles represent three frailty groups: low frail (Q1), medium frail (M), and high frail (Q3) mice. We hypothesized that different sets of features would be important for mice in different frailty groups. Indeed, step length 1IQR was important in mice in both the Q1 and Q3 quantiles (Figure 13A). Additionally, features such as length, hindlimb velocity, dAC / dB (non-gait), and step width were important for less frail mice, while step length dB and rear count were more important for more frail animals. Similarly, step width, tail tip LD, and width were important for mice with FI scores closer to M.

[0166] For the feature interaction strength approach, we used the H-statistic [Friedman JH et al., Annals of Applied Statistics 2, 916-954 (2008)] as an interaction metric that measured the percentage of variability in the prediction explained by feature interactions after considering individual features. For example, 15% of the variability in the prediction function was explained by interactions between tail tip LD and other features after considering the individual contributions by tail tip LD and other features. Approximately 13% and 8% of the variability in the prediction function were explained by interactions between width (respectively step length) and other features. For a deeper analysis, we examined all two-way interactions between tail tip LD and other features (results not shown). We found strong LD interactions between the tail tip and the animal's width, stride length, hindlimb, and dB.

[0167] Both the feature importance and the strength of feature interactions indicated that the trained random forest for vFI was dependent on some features and their interactions. However, they did not show how vFI depends on these features and what the interactions look like. We used the Accumulated Local Effect (ALE) plots [Apley DW and Chu J. Journal of the Royal Statistical Society Series B, Statistical Methodology 82, 1059-1086 (2020)], which described how features affected the vFI predictions of the random forest model on average. For example, an increase in tail tip lateral displacement positively influenced (increased) the predicted FI scores of animals in the intermediate and high frailty groups (Figure 13B). Similarly, an increase in hindlimb measurements positively influenced the predictions. This effect was most seen in animals in the high frailty group. Animals with larger step widths positively influenced the predictions. That is, larger step widths and dBs positively influenced the model predictions. Thus, the ALE plots for the key features provided a clear interpretation consistent with the first hypothesis. ALE quadratic interaction effect plots were explored for the predictors of step length-step width (Figure 13D) and length-width (Figure 13E). This revealed an additive interaction effect of the two features, but not the marginal effect of the key feature. Figure 13D clearly depicts the interaction between step width and step length; that is, larger step width and step length increased the predicted FI score. Similarly, larger width (36-44 cm) and length (52-60 cm) positively influenced the mean FI score prediction.

[0168] In summary, the utility of vFI was established by demonstrating dependence on several features through marginal feature importance and feature interactions. ALE plots were then used to understand the impact of features on the model predictions. This helped to relate the predictions of the black-box model to some of the video-generating features. Opening up the black-box model was an essential final step in the modeling framework.

[0169] Consideration Mouse FI is a valuable tool in the study of biological aging. The work described herein seeks to extend it by generating an automated visual frailty index (vFI) using video-generated features to model the FI score. This vFI provided a reliable, high-throughput method to study aging. One of the largest frailty datasets for mice was generated with associated open-field video data. Computer vision techniques were used to extract behavioral and morphological features, many of which showed strong correlations with aging and frailty. Sex-specific aging in mice was also analyzed. A machine learning classifier was then trained that could accurately predict frailty from video features. Through modeling, insights into the importance of features across age and frailty status were also gained.

[0170] Data were collected at the National Center on Aging in a design similar to that obtained in high-throughput intervention studies that may be conducted over several years. Mice were tested by experienced scorers when available. Four different scorers were used to FI test different batches of mice. In addition, personnel were rotated between batches. These conditions may provide a more realistic example of inter-laboratory conditions that are difficult to discuss and fine-tune. We found that 42% of the variability in the dataset could be explained by the scorer, indicating the presence of a tester effect. This variability affected some items, such as hair ruffles, more than others. Previous studies that have examined tester effects have found good to high inter-tester reliability in most cases, but FI items with low inter-reliability required discussion and fine-tuning for improvement [Kane AE, Ayaz O., Ghimire, A., Feridoni HA and Howlett SE, Canadian Journal of Physiology and Pharmacology (2017)].

[0171] Top-down footage of the mouse in the open field was processed by a previously trained neural network to generate an ellipse fit and segmentation of the mouse, as well as a pose estimate of 12 salient points on the mouse for each frame. These frame-by-frame metrics were used to design features for use in the model. The first category of features were standard open field metrics such as time spent in the periphery versus the center, total distance traveled, and number of grooming behaviors. These standard open field metrics correlated poorly with both FI scores and age. These results suggested that the standard open field assay is insufficient for studying aging.

[0172] In humans, age-related changes in body composition and dimensional indicators such as waist-to-hip ratio are predictors of health status and mortality risk [Mizrahi-Lehrer E., Cepeda-Valery B. and Romero-Corral, Handbook of Anthropometry: Physical Indicators of Human Shape in Health and Disease (2021); Pappas LE and Timm R, N., European Journal of Clinical Nutrition (2019); Gervais M., Metz L., Lingotto E. and Courtex D., Lipids in Health and Disease (2010)]. The effect of aging on body composition in rodent models is less well established, but similar changes in body composition have been observed in humans [Mizrahi-Lehrer E., Cepeda-Valery B. and Romero-Corral, Handbook of Anthropometry, Physical Indicators of Human Shape in Health and Disease (2021); Gervais M., Metz L., Lingotto E. and Courtex D., Lipids in Health and Disease (2010)]. High correlations have been found between morphological features and both FI scores and age, especially medial width and medial hindlimb width.

[0173] The prevalence of gait disorders increases with age [Zou et al., Scientific Reports (2020)]. Elderly patients have been shown to have gait irregularities. For example, older adults have increased step width variability [Tarantini S. et al., Journal of Gerontology: Series A (2018)]. We examined the spatial, temporal, and postural characteristics of each mouse's gait and found many features that have strong correlations with both frailty and age. Similar to human data, we observed a decrease in stride velocity and increased step width variability with age [Tarantini S. et al., Journal of Gerontology: Series A (2018)]. Gait is a compelling area for frailty research because it is thought to have both cognitive and musculoskeletal components.

[0174] Spinal mobility in humans is a predictor of quality of life in elderly populations, and mice are used as a model of the aging human spine. Surprisingly, some spinal flexion metrics showed moderately high correlations with FI scores, but the relationship was the opposite of the initial hypothesis. These metrics are general descriptions of all spinal activity during the experiment, and therefore likely capture a combination of behavior and body composition that led to the observed results. Nevertheless, some of these metrics showed moderately high correlations with FI scores and age, and were considered important features in the model.

[0175] Many of the biochemical and physiological changes associated with age are known to be sex-specific. Understanding sex differences in the presentation and progression of frailty in mice is important to translate preclinical results for clinical use. It is interesting to understand how sex characteristics such as hormones and body fat distribution relate to biological aging. In humans, there is a known “mortality and morbidity paradox” or “sex-frailty paradox” in which females tend to be more frail but paradoxically survive longer. However, in C57BL / 6J mice, males appear to tend to survive slightly longer than females, but there is variability and females do not appear to survive paradoxically longer when frail. In the study described herein, we found that more males survived to older ages than females, and furthermore, females tended to have a slightly lower frailty distribution than males in the same age group. These results suggest that in mice, the “sex-frailty paradox” exhibited in humans may not exist or may be reversed. Correlations of FI index items with age between males and females were compared, and some gender differences were found in the strength of correlations for some index items, mainly those related to visual fur changes. When comparing correlations of video features with age and FI scores between males and females, many orders of magnitude correlations were also found. Both the median lateral displacement of the tail base and the median lateral displacement of the tail tip were much more strongly correlated with age in females than in males. As female mice aged, the lateral displacement of the tail within the stride tended to increase, whereas males showed little change. On the other hand, males showed a large shortening of stride length and a large increase in step length with age, whereas females showed little change. Most of the video features with large differences were gait-related, and some were related to spinal flexion. These age differences in gait were novel insights. Understanding the differences between human frailty and mouse frailty is important to critically evaluate how results from mouse studies can be translated to humans.

[0176] The manual FI assesses a broader range of body systems than the vFI. However, the complex behaviors measured and described herein contain a lot of implicit information about body systems. In the isogenic dataset, most information in the manual FI comes from a limited subset of index items. Of the 27 manual FI items scored, 18 items had little or no variation in score in our dataset (nearly all mice had the same score, i.e., 0), and only 9 items had a balanced score distribution. Video features can accurately predict these 9 FI items. Models using video features also predicted age more accurately (FRIGHT vs. vFRIGHT) with much less variability than models using manual FI items. This suggests that the video features described herein are not only able to predict relevant FI items, but also contain signals about aging beyond traditional manual FI. Additionally, the details of the feature measurements compared to FI items (using actual values ​​rather than simplified scores of 0, 0.5, or 1) may contribute to higher performance.

[0177] Finally, video features were used as input to a random forest model to predict manual FI scores, on average, to within 0.04 ± 0.002 of the actual score. Although not normalized, this error was 1.08 ± 0.05, which is comparable to one FI item being misscored by 1 point, or two FI items being misscored by 0.5 points. Additionally, we determined a simple point prediction by providing a 95% prediction interval. Quantile random forests were applied to low and high quantiles of the conditional distribution of FI scores to reveal how specific features differentially affected frail and healthy animals.

[0178] Ease of use of machine learning models by non-computational laboratories is a significant challenge. Therefore, in addition to the implementation details in the Methods section, an integrated mouse phenotyping platform, i.e., hardware and software solutions, that provide tracking, pose estimation, feature generation, and automated behavioral analysis, is detailed in [Bean G. et al., bioRxiv (2022)]. Although this platform requires specific open field equipment, researchers can use machine learning models if they use their own open field data collection equipment to generate the same features as the models described herein. A setup that allows tracking and pose estimation using available software allows researchers to calculate the features required to use machine learning models.

[0179] vFI can be further improved by adding new features through reanalysis of existing data and future technical improvements to data acquisition [Pereira TD, Schaewitz JW and Marcy M. Nature Neuroscience (2020); Matthias A. Neuron (2020)]. For example, quantification of defecation and urination can provide information on additional systems, while higher camera quality can provide more detailed information on fine motor-based behaviors and appearance-based features such as coat condition. Furthermore, this approach may be used in long-term home cage environments. This would not only further reduce handling and environmental factors, but also potentially integrate features such as social interactions, feeding, drinking and sleep. Furthermore, given the evidence of a strong genetic component to aging [Shin PP, Demitt BA, Nass RD and Brunet A., Cell (2019)], applying this method to other lineages and genetically heterogeneous populations such as diversity outcrosses and collaborative crosses may shed light on how genetic variation influences frailty. Furthermore, video features can be used to study lifespan, as predicting mortality risk is an important feature of frailty. The value of this work extends beyond community adoption and may be directed toward community engagement. Training on data from multiple labs can provide even more stable and accurate models. This may provide uniform FI across multiple studies. Overall, this approach generates new insights into mouse frailty and demonstrates that video data of mouse behavior can be used to quantify abstract concepts such as frailty. An automated frailty index would enable high-throughput and reliable aging research, especially intervention studies, which are a priority for the aging research community.

[0180] Equivalent While several embodiments of the invention have been described and illustrated herein, those skilled in the art can readily envision various other means and / or structures for performing the functions and / or obtaining the results and / or one or more advantages described herein, and each of such variations and / or modifications is deemed to be within the scope of the invention. More generally, those skilled in the art will readily appreciate that all parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and that the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications for which the teachings of the invention are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments of the invention described herein. It will thus be understood that the foregoing embodiments have been presented by way of example only, and that within the scope of the appended claims and their equivalents, the invention may be practiced otherwise than as specifically described and claimed. The invention is directed to each individual feature, system, article, material, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, and / or methods is included within the scope of the present invention, if such features, systems, articles, materials, and / or methods are not mutually inconsistent. It will be understood that all definitions defined and used herein control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings for the defined terms.

[0181] As used herein in the specification and claims, the indefinite articles "A" and "An" will be understood to mean "at least one" unless clearly indicated to the contrary. As used herein in the specification and claims, the term "and / or" will be understood to mean "either or both" of the elements so conjoined, i.e., "either or both" of elements that are conjunctively present in some cases and disjunctively present in other cases. Unless expressly indicated to the contrary, other elements, whether related or unrelated to the specifically identified elements, may optionally be present other than the elements specifically identified by the term "and / or."

[0182] As used herein, conditional language, particularly "can," "could," "might," "may," "eg," and the like, is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not, unless specifically stated otherwise or understood otherwise within the context in which it is used. Thus, such conditional language is not generally intended to imply that the features, elements, and / or steps are required in any manner with respect to one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included in or performed in any particular embodiment, with or without other input or prompting. The terms "comprising," "including," "having," and the like are synonymous and used in an inclusive, open-ended manner and do not exclude additional elements, features, acts, operations, and the like. Also, the term "or" is used in its inclusive sense (and not its exclusive sense), so that, for example, to connect a list of elements, the term "or" means one, some, or all of the elements in the list.

[0183] All references, patents, and patent applications and publications cited or referred to in this application are hereby incorporated by reference in their entirety.

Claims

1. 1. A computer-implemented method comprising: receiving video data representing video capturing motion of a subject; using the imaging data to determine a spinal mobility characteristic of the subject for a duration of the imaging; processing at least the spinal mobility features using at least one machine learning model to determine a visual frailty score for the subject; The method includes:

2. determining the spinal mobility characteristics of the subject during the duration of the video; determining a plurality of spine measurements, each spine measurement of the plurality of spine measurements corresponding to one image frame of the imaging data; determining said spinal mobility characteristic using said plurality of spinal measurements; 2. The computer implemented method of claim 1, comprising:

3. determining the spinal mobility characteristics of the subject during the duration of the video; For each video frame of the video data, determining a first distance between a head of the subject and a tail of the subject; determining a second distance between the subject's mid-back and a midpoint between the head and the tail; determining an angle formed between the head, the tail, and the mid-back of the subject; determining the spinal mobility characteristics of the image frames to include the first distance, the second distance, and the angle; 2. The computer implemented method of claim 1, comprising:

4. determining the spinal mobility characteristics of the subject during the duration of the video; determining, for each image frame of the image data, a distance between a central back of the subject and a midpoint between the head of the subject and the tail of the subject; 2. The computer implemented method of claim 1, comprising:

5. processing the video data using at least an additional machine learning model to determine pose estimation data that tracks at least a position of the subject's head, a position of the subject's tail, and a position of the subject's mid-back during the duration of the video; using the posture estimation data to determine the spinal mobility characteristics. The computer implemented method of claim 1 further comprising:

6. processing the video data to determine pose estimation data that tracks positions of at least twelve body parts of the subject during the duration of the video; determining features of the object using the pose estimation data; and processing the features using the at least one machine learning model to determine the visual frailty score; The computer implemented method of claim 1 further comprising:

7. determining a physical characteristic about the subject, the physical characteristic corresponding to at least one of a length of the subject, a width of the subject, and a distance between the subject's hind legs; processing the physical characteristics using the at least one machine learning model to determine the visual frailty score; The computer implemented method of claim 1 further comprising:

8. determining a number of rearing events occurring during said duration of said video; determining the rear length for each rear event; processing the number of times the rearing events occurred and the rearing length per rearing event using the at least one machine learning model to determine the visual frailty score; The computer implemented method of claim 1 further comprising:

9. processing the video data to determine an ellipse fit for the object during the duration of the video using the at least one machine learning model; determining features of the object using the ellipse fitting data; and processing the features using the at least one machine learning model to determine the visual frailty score; and The computer implemented method of claim 1 further comprising:

10. determining a characteristic of the subject's spinal mobility for a duration of the imaging; determining a first set of video frames representative of locomotion by the subject; determining a first set of spinal mobility characteristics for the first set of image frames; determining a second set of video frames representative of non-ambulatory movement by the subject; determining a second set of spinal mobility characteristics for the second set of image frames; Including, The computer implemented method of claim 1 , wherein the spinal mobility features include a first set of spinal mobility features and a second set of spinal mobility features.

11. 11. The computer implemented method of claim 10, wherein the first set of spinal mobility features corresponds to a distance between the subject's mid-back and a midpoint between the subject's head and tail, and the second set of spinal mobility features corresponds to an angle formed between the head, tail and the mid-back of the subject.

12. using the video data to determine gait measurements of the subject during the duration of the video; processing the gait measurements using the at least one machine learning model to determine the visual frailty score of the subject; The computer implemented method of claim 1 further comprising:

13. processing the image data to determine point data that tracks movement of a set of body parts of the subject during the duration of the image; using the point data to determine a number of stance phases and a number of swing phases represented in the video data; determining a plurality of stride intervals represented in the video data based on the plurality of stance phases and the plurality of swing phases; determining gait measurements using the point data, the gait measurements being based on each stride interval of the plurality of stride intervals; The computer implemented method of claim 12 further comprising:

14. processing the video data to determine point data tracking movement of a set of body parts during the duration of the video, the set of body parts including one or more of the nose, base of the neck, mid-spine, left hind leg, right hind leg, base of the tail, mid-tail, and tip of the tail; determining a feature of the object using the point data; and processing the features using the at least one machine learning model to determine the visual frailty score; and The computer implemented method of claim 1 further comprising:

15. processing the video data using an additional machine learning model to identify a likelihood that the subject exhibits grooming behavior with respect to a plurality of video frames of the video data; determining the visual frailty score using the likelihood that the subject will exhibit the grooming behavior; and The computer implemented method of claim 1 further comprising:

16. processing the video data using an additional machine learning model to identify a likelihood that the subject exhibits a predetermined behavior for a plurality of video frames of the video data; determining the visual frailty score using the likelihood of the subject exhibiting the predetermined behavior; and The computer implemented method of claim 1 further comprising:

17. processing the video data to determine gait measurements for the subject during the duration of the video; processing the video data to determine behavioral data identifying portions of the video in which the subject exhibits a predetermined behavior; processing the spinal mobility characteristics, the gait measurements, and the behavioral data using the at least one machine learning model to determine the visual frailty score; 2. The computer implemented method of claim 1, comprising:

18. 2. The computer-implemented method of claim 1, further comprising using the visual frailty score to determine a physical status of the subject.

19. 20. The computer implemented method of claim 18, wherein the physical condition is frailty.

20. 20. The computer implemented method of claim 18, wherein the physical condition is a pre-frail state.

Images