Determining a visual frailty index using machine learning models

Abstract

The systems and methods described herein provide techniques for determining a visual frailty score by processing video data of a subject. Various features may be used to determine the visual frailty score, including, but not limited to, spinal mobility features, gait measurements, behavioral features, and body composition data. The various features may be extracted from the video data using different techniques. One or more machine learning models may be used to process the various features to determine the visual frailty score.

Description

[Technical Field]

[0001] Related applications This application asserts the interests under 35 § 119(e) of U.S. Patent Act pursuant to U.S. Provisional Application No. 63 / 187892, filed on 12 May 2021, the disclosure thereof is incorporated herein by reference in its entirety.

[0002] In some embodiments, the present invention relates to determining a subject's visual frailty index by processing video data using a machine learning model. Government support

[0003] This invention was made possible with government support under DA041668 and DA048634, awarded by the National Institute on Drug Abuse, and AG38070, awarded by the National Institute on Aging. The government has certain rights to this invention. [Background technology]

[0004] Aging is the final process affecting all biological systems. In contrast to aging over time, biological aging occurs at different rates for different individuals. In humans, aging comes with increased health problems and mortality, but some individuals live long and healthy lives, while others die prematurely from disease and disability. More precisely, heterogeneity in mortality risk and health status has been observed among individuals within age cohorts [Non-Patent Literature 1, Non-Patent Literature 2]. The concept of frailty has been used to quantify this heterogeneity and is defined as a state of increased vulnerability to adverse health outcomes [Non-Patent Literature 3]. Identifying frailty is clinically important because frail individuals have a higher risk of disease and disability, worse health outcomes from the same disease, and even different symptoms of the same disease [Non-Patent Literature 2].

[0005] The Frailty Index (FI) is a widely used approach to quantify frailty [Non-Patent Literature 1] and is superior to other methods [Non-Patent Literature 4]. This method scores individuals for a set of age-related health impairments to generate a cumulative score. Each impairment must have the following characteristics: it must be health-related, its population must increase with age, and its population must not saturate too quickly [Non-Patent Literature 5]. The presence and severity of each health impairment are scored as 0 if absent, 0.5 if partially present, or 1 if present. A compelling finding regarding FI is that while the exact health impairments scored may differ across indices, they still exhibit similar characteristics and usefulness [Non-Patent Literature 5]. That is, even with two sufficiently large FIs, differing in the number and selection of impairments, the average rate of age-related impairment accumulation is similar, as is the possible lower limit of the FI score. More importantly, both FIs reliably predict the individual's risk of adverse health outcomes, hospitalization, and death. The characteristics of this FI are advantageous because they allow researchers to draw data from various large medical databases to support large-scale studies. Furthermore, it suggests that frailty is a legitimate phenomenon, and given the complexity of aging, FI is an effective way to quantify it. Not only do people age at different rates, but they age in different ways as well. For example, one person may have severe mobility problems but a sharp memory, while another may have a healthy heart but a weak immune system. Both may have similar levels of frailty, but this is only evident by sampling a variety of health impairments. Indeed, FI scores outperform other developed indices, such as tracking molecular markers and frailty phenotyping tests, in their efficient prediction of mortality risk and health status [Non-Patent Literature 6-8]. While some FIs have been adapted for use in mice using a variety of behavioral and physiological indicators as index items [Non-Patent Literature 2, 4, 9], there is no suitable method for assessing frailty and predicting mortality risk and health in animal models and humans.

Prior Art Documents

Non-Patent Documents

[0006]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

[0008] According to another aspect of the present invention, a method is provided for evaluating the physical condition of a subject, the method comprising determining a visual frailty score of the subject using a computer implementation method of any one embodiment of the above-described aspects. In some embodiments, the physical condition is frail. In some embodiments, the physical condition is pre-frail. In some embodiments, the physical condition is disease or pathological condition. In some embodiments, the subject is a mammal, and optionally a mouse.

[0009] According to another aspect of the present invention, a method is provided for determining the presence of an effect of a candidate compound on a frail state, the method comprising: obtaining a first visual frailty score relating to a subject, wherein the means for obtaining the score include a computer implementation method according to any one of claims A1 to A39, the subject being a frail state or an animal model exhibiting a frail state; administering the candidate compound to the subject; obtaining a visual frailty score for the subject after administration; and comparing a first gait measurement with the visual frailty score value after administration, wherein the difference between the first visual frailty score and the visual frailty score after administration identifies the effect of the candidate compound on the frail state. In some embodiments, an improvement in the visual frailty score indicating a lower degree of frailty identifies the candidate compound as promoting the regression of the frail state. In some embodiments, a visual frailty score after administration that is statistically equal to the first visual frailty score identifies the candidate compound as inhibiting the progression of the frail state in the subject. In some embodiments, the method also includes additionally testing the effect of the compound on the treatment of the frail state. In some embodiments, the subject is a mammal, optionally a mouse.

[0010] According to another aspect of the present invention, a method is provided for identifying the presence of an effect of a candidate compound on a frail state, the method comprising: administering the candidate compound to a subject having a frail state or to a subject forming an animal model of a frail state; obtaining a visual frailty score for the subject, the means for obtaining the score including embodiments of computer implementation methods of any above-described aspects of the present invention; and comparing the obtained visual frailty score with a control visual frailty score, wherein the difference between the obtained visual frailty score and the control visual frailty score identifies the presence of an effect of the candidate compound on a frail state. In some embodiments, an improvement in the visual frailty score indicating a lower degree of frailty in the subject administered with the candidate compound compared to the control frailty score identifies the candidate compound as promoting the regression of the frail state in the subject. In some embodiments, the visual frailty score obtained in the subject administered with the candidate compound is statistically equivalent to the control frailty score, identifying that the candidate compound inhibits the progression of the frail state in the subject. In some embodiments, the subject is a mammal, optionally a mouse.

[0011] According to another aspect of the present invention, a system is provided which includes at least one processor and at least one memory which includes instructions causing the system to: receive video data representing video capturing the movement of a subject when executed by the at least one processor; use the video data to determine the characteristics of the subject's spinal mobility for the duration of the video; and use at least one machine learning model to process at least one of the spinal mobility characteristics to determine the subject's visual frailty score. In some embodiments, the instructions causing the system to determine the characteristics of the subject's spinal mobility during the duration of the video further cause the system to determine a plurality of spinal measurements, wherein each of the plurality of spinal measurements corresponds to one video frame of the video data; and use the plurality of spinal measurements to determine the characteristics of spinal mobility. In some embodiments, a command to cause the system to determine the spinal mobility features of a subject during the duration of the video causes the system to determine, for each video frame of the video data, a first distance between the subject's head and tail, a second distance between the subject's midpoint and the midpoint between the head and tail, an angle formed between the subject's head, tail and midpoint, and a spinal mobility feature of the video frame including the first distance, the second distance, and the angle. In some embodiments, a command to cause the system to determine the spinal mobility features of a subject during the duration of the video causes the system to determine, for each video frame of the video data, a distance between the subject's midpoint and the midpoint between the subject's head and tail. In some embodiments, at least one memory, when executed by at least one processor, includes a further command to cause the system to process the video data using at least one machine learning model to determine pose estimation data that tracks the positions of at least the subject's head, tail and midpoint of the subject's back during the duration of the video, and to use the pose estimation data to determine the spinal mobility features.In some embodiments, at least one memory also includes further instructions, when executed by at least one processor, to cause the system to process video data to determine pose estimation data that tracks one of at least 12 body parts of a subject during the duration of the video; to determine features of the subject using the pose estimation data; and to process the features using at least one machine learning model to determine a visual frailty score. In some embodiments, at least one memory also includes further instructions, when executed by at least one processor, to cause the system to determine body features of a subject, wherein the body features correspond to at least one of the subject's length, width, and distance between the subject's hind limbs; and to process the body features using at least one machine learning model to determine a visual frailty score. In some embodiments, at least one memory includes further instructions that cause the system to perform, when executed by at least one processor, the following: determine the number of times hind limb standing events occur during the duration of the video; determine the length of each hind limb standing event; and process the number of times hind limb standing events occur and the length of each hind limb standing event using at least one machine learning model to determine a visual frailty score. In some embodiments, at least one memory includes further instructions that cause the system to perform, when executed by at least one processor, the following during the duration of the video: process the video data using at least an additional machine learning model to determine elliptic fitting data for the subject; use the elliptic fitting data to determine features of the subject; and process the features using at least one machine learning model to determine a visual frailty score.In some embodiments, during the duration of the video, instructions to the system to determine the spinal mobility features of a subject include, further, determining a first set of video frames representing walking motion by the subject, determining a first set of spinal mobility features for the first set of video frames, determining a second set of video frames representing non-walking motion by the subject, and determining a second set of spinal mobility features for the second set of video frames, such that the spinal mobility features include a first set of spinal mobility features and a second set of spinal mobility features. In some embodiments, the first set of spinal mobility features corresponds to the distance between the midpoint of the subject's back and the midpoint between the subject's head and tail, and the second set of spinal mobility features corresponds to the angles formed between the subject's head, tail and midpoint of the back. In some embodiments, when at least one memory is also executed by at least one processor, further instructions to the system include, during the duration of the video, using the video data to determine gait measurements of the subject, and processing the gait features using at least one machine learning model to determine the subject's visual frailty score. In some embodiments, the system also includes instructions to cause the system, when executed by at least one processor, to process the video data to determine point data that tracks the movement of a set of multiple body parts of an object during the duration of the video; to use the point data to determine a set of stance phases and a set of swing phases represented in the video data; to determine a set of stride intervals represented in the video data based on the set of stance phases and a set of swing phases; and to use the point data to determine a gait measurement for the object based on each of the set of stride intervals.In some embodiments, at least one memory includes, when executed by at least one processor, further instructions causing the system to: determine a first transition from a first stance phase of multiple stance phases and from a first swing phase of multiple swing phases based on a toe-off event of the left hind limb or the right hind limb of the subject; determine a second transition from a second swing phase of multiple swing phases to a second stance phase of multiple stance phases based on a foot-contact event of the left hind limb or the right hind limb; and determine a gait measurement using the first and second transitions. In some embodiments, at least one memory, when executed by at least one processor, processes video data to determine point data that tracks the movement of a set of multiple body parts of an object during the duration of the video, the set of multiple body parts including a left hind limb and a right hind limb, and determines gait measurements, which includes processing the point data to determine a step length with respect to a stride interval, where the step length represents the distance the right hind limb moves beyond the ground contact position of the anterior left hind limb; determining the stride length used with respect to each stride interval, where the stride length represents the distance the left hind limb moves during each stride interval; and determining the step width for each stride interval, where the step width represents the distance between the left hind limb and the right hind limb. In some embodiments, at least one memory processes video data to determine point data that tracks the movement of a set of multiple body parts of an object during the duration of the video, when executed by at least one processor, the set of multiple body parts including the base of the tail, and an instruction causing the system to determine gait measurements includes an instruction causing the system to process the point data to determine velocity data of the object based on the movement of the base of the tail with respect to stride intervals.In some embodiments, at least one memory processes video data to determine point data that tracks the movement of a set of multiple body parts of an object during the duration of the video, when executed by at least one processor, and the instruction to cause the system to determine a gait measurement includes processing the point data to cause the system to determine a set of multiple velocity data of an object based on the movement of the tail root between a set of multiple frames representing a stride interval among a set of multiple stride intervals, and to determine a stride velocity relative to a stride interval by averaging the set of velocity data. In some embodiments, at least one memory processes video data to determine point data that tracks the movement of a set of multiple body parts of an object during the duration of the video, when executed by at least one processor, the set of multiple body parts includes a right hind limb and a left hind limb, and a command to cause the system to determine a gait measurement includes processing the point data to determine a first stance duration representing the amount of time the right hind limb is in contact with the ground during a stride interval; determining a first load factor based on the first stance duration and the stride interval duration; determining a second stance duration representing the amount of time the left hind limb is in contact with the ground during a stride interval, using the point data; determining a second load factor based on the second stance duration and the stride interval duration; and determining an average load factor for the stride interval based on the first and second load factors.In some embodiments, at least one memory processes video data to determine point data for tracking the movement of a set of body parts of an object during the duration of the video, when executed by at least one processor, the set of body parts includes the base of the tail and the base of the neck, and a command to cause the system to determine a gait measurement includes a command to cause the system to use the point data to determine a set of vectors connecting the base of the tail and the base of the neck between a set of frames representing stride intervals having multiple stride intervals, and a command to cause the system to use the set of vectors to determine the angular velocity of the object with respect to the stride intervals. In some embodiments, at least one memory processes video data to determine point data that tracks the movement of a set of multiple body parts of an object during the duration of the video, when executed by at least one processor, the set of multiple body parts includes the midspine of the object, the stride interval is associated with a set of multiple frames of video data, and the instruction causing the system to determine the gait measurement includes a further instruction causing the system to process the point data to determine a displacement vector for the stride interval, the displacement vector connecting the midspine represented in the first frame of the set of multiple frames to the midspine represented in the last frame of the set of multiple frames. In some embodiments, the set of multiple body parts also includes the nose of the object. The command to the system to determine gait measurements further causes the system to determine metric data which further causes the system to determine a set of lateral displacements of the nose for each stride interval based on the vertical distance of the nose from the displacement vector for each frame of a set of multiple frames using point data. In some embodiments, the lateral displacement of the nose is further based on the body length of the subject. In some embodiments, the command to the system to determine gait measurements further causes the system to generate a smooth curve of the lateral displacement of the nose against the stride interval by performing interpolation using a set of lateral displacements of the nose, determine at what point in the stride interval the maximum displacement of the nose occurred using the smooth curve of lateral displacement of the nose, and determine a percentage stride position which represents the percentage of the stride interval completed when the maximum displacement of the nose occurred. In some embodiments, at least one memory processes video data to determine point data for tracking the movement of a set of multiple body parts of an object during the duration of the video, when executed by at least one processor, wherein the set of multiple body parts of an object further includes the base of the object's tail, and a command to cause the system to determine gait measurements further includes a command to cause the system to use the point data to determine a set of multiple lateral displacements of the tail base with respect to stride intervals, based on the vertical distance of the tail base from a displacement vector for each frame in a set of multiple frames.In some embodiments, a command to the system to determine gait measurements further causes the system to determine the displacement phase offset of the tail base by performing interpolation using a set of lateral displacements of the tail base to generate a smooth curve of the lateral displacement of the tail base with respect to the stride interval, using the smooth curve of the lateral displacement of the tail base to determine at what point in the stride interval the maximum displacement of the tail base occurred, and determining a percentage stride position that represents the percentage of the stride interval completed when the maximum displacement of the tail base occurred. In some embodiments, at least one memory processes video data to determine point data for tracking the movement of a set of multiple body parts of an object during the duration of the video, when executed by at least one processor, wherein the set of multiple body parts of an object includes the tail tip of the object, and an instruction causing the system to determine gait measurements includes a further instruction causing the system to process the point data to determine a set of multiple lateral displacements of the tail tip relative to the stride interval, based on the perpendicular distance of the tail tip from a displacement vector for each frame in a set of multiple frames. In some embodiments, a command to the system to determine gait measurements further causes the system to generate a smooth curve of the lateral displacement of the tailtip against the stride interval by performing interpolation using a set of multiple lateral displacements of the tailtip; to determine, using the smooth curve of the lateral displacement of the tailtip, at what point in the stride interval the maximum displacement of the tailtip occurred; and to determine a percentage stride position representing the percentage of the stride interval completed when the maximum displacement of the tailtip occurred.In some embodiments, at least one memory, when executed by at least one processor, includes instructions to the system to process video data to determine point data that tracks the movement of a set of multiple body parts of an object during the duration of the video, the set of multiple body parts including one or more of the nose, base of the neck, mid-spine, left hind limb, right hind limb, base of the tail, mid-tail, and tip of the tail; to use the point data to determine features of the object; and to process the features using at least one machine learning model to determine a visual frailty score. In some embodiments, at least one memory, when executed by at least one processor, includes instructions to the system to process video data using an additional machine learning model to identify the likelihood that an object exhibits grooming behavior for a set of video frames of the video data; and to use the likelihood that an object exhibits grooming behavior to determine a visual frailty score. In some embodiments, at least one memory includes further instructions that cause the system to process video data using an additional machine learning model to identify the likelihood of an object exhibiting a predetermined behavior for multiple video frames of the video data, and to determine a visual frailty score using the likelihood of the objects exhibiting the predetermined behavior, when executed by at least one processor.In some embodiments, at least one memory includes further instructions to cause the system to perform the following actions when executed by at least one processor: to determine a rotated set of video frames by rotating a first set of video frames of video data; to process the first set of video frames using a first machine learning model configured to identify the likelihood of an object exhibiting a predetermined action; to determine a first probability of an object exhibiting a predetermined action in a first video frame of the first set of video frames, which corresponds to a first duration of video data, based on processing the first set of video frames by the first machine learning model; to process the rotated set of frames using the first machine learning model; to determine a second probability of an object exhibiting a predetermined action in a second video frame of the rotated set of video frames, which corresponds to a first duration of video data, based on processing the rotated set of frames using the first machine learning model; and to use the first and second probabilities to identify a first label for the first video frame, wherein the first label indicates that the object exhibits a predetermined action. In some embodiments, at least one memory includes further instructions causing the system to: process a first set of video frames using a second machine learning model configured to identify the likelihood of an object exhibiting a predetermined action when executed by at least one processor; determine a third probability of an object exhibiting a predetermined action within the first video frames based on the processing of the first set of video frames by the second machine learning model; process a rotated set of video frames using the second machine learning model; determine a fourth probability of an object exhibiting a predetermined action within the second video frames based on the processing of a set of rotated video frames by the second machine learning model; and identify a first label using the first, second, third, and fourth probabilities.In some embodiments, at least one memory includes further instructions to cause the system to: determine a set of reflected video frames by reflecting a first set of video frames when executed by at least one processor; process the set of reflected video frames using a first machine learning model; determine a third probability of an object exhibiting a predetermined behavioral action in a third video frame of the set of reflected frames, which corresponds to a first duration of the first video frame, based on the processing of the set of reflected video frames by the first machine learning model; and identify a first label using the first, second, and third probabilities. In some embodiments, the object is a mouse, and the predetermined behavior includes grooming behaviors, including at least one of licking its paws, washing one side of its face, washing both sides of its face, and licking its flanks. In some embodiments, the first set of video frames represents a portion of video data over a period of time, and the first video frame is the last time frame of that period.In some embodiments, at least one memory includes instructions to cause the system to perform the following actions when executed by at least one processor: identify a second set of video frames from video data; determine a second rotated set of video frames by rotating the second set of video frames; process the second set of video frames using a first machine learning model; determine a third probability that an object exhibits a predetermined action in a third video frame of the second set of video frames based on the processing of the second set of video frames by the first machine learning model; process the second rotated set of video frames using a first machine learning model; determine a fourth probability that an object exhibits a predetermined action in a fourth video frame of the rotated set of frames based on the processing of the second rotated set of video frames by the first machine learning model, wherein the fourth video frame corresponds to the third video frame; and use the third and fourth probabilities to identify a second label of the fourth video frame, where the second label indicates that the object exhibits a predetermined action. In some embodiments, the first machine learning model is a machine learning classifier. In some embodiments, at least one memory includes further instructions to cause the system to process video data to determine gait measurements relating to a subject during the duration of the video, process video data to determine behavioral data identifying portions of the video in which the subject exhibits a predetermined behavior, and process spinal mobility features, gait measurements, and behavioral data using at least one machine learning model to determine a visual frailty score. In some embodiments, the video captures the movement of a subject in an open field arena in a planar view. In some embodiments, at least one memory includes further instructions to cause the system to use the visual frailty score to determine the physical condition of the subject. In some embodiments, the physical condition is frail.In some embodiments, the physical condition is pre-frail. In some embodiments, the subject is a mammal, and optionally a mouse. [Brief explanation of the drawing]

[0012] For a more complete understanding of this disclosure, please refer to the following description in conjunction with the attached drawings. [Figure 1] This is a conceptual diagram of a system for determining the actions of a target according to an embodiment of the present disclosure. [Figure 2] This flowchart shows the process for determining various data about an object using point data derived from video data, according to an embodiment of the present disclosure. [Figure 3] This flowchart shows the process for determining the morphological data of an object using ellipse data derived from video data, according to an embodiment of the present disclosure. [Figure 4] This is a flowchart showing the process for determining target behavioral data using video data according to an embodiment of the present disclosure. [Figure 5] This flowchart shows a process for determining a visual frailty score using one or more data determined according to the process shown in Figures 2 to 4, according to embodiments of the present disclosure. [Figure 6] This is a block diagram conceptually illustrating exemplary components of a device according to an embodiment of the present disclosure. [Figure 7] This is a block diagram conceptually illustrating exemplary components of a server according to the embodiments of this disclosure. [Figure 8A]This is a schematic diagram showing the automated visual frailty index (vFI) pipeline and graphs representing scores from the manual frailty index of mice. The schematic diagram in Figure 8A shows the pipeline for creating the automated visual frailty index (vFI). Top-down open-field video of each mouse was processed by tracking and segmentation networks and pose estimation networks. The resulting frame-by-frame ellipse fit and 12 point pose coordinates were further processed to create video-by-video metrics for the mouse. Each mouse was also manually frailty-indexed to generate an FI score. The video features of each mouse were used to model its FI score. [Figure 8B] This is a schematic diagram showing graphs representing scores from the Automated Visual Frailty Index (vFI) pipeline and the manual frailty index of mice. Figure 8B shows a scatter plot of FI scores by age. The black lines show a partial linear fit to the data piece by piece. Error bars indicate the standard deviation. Males have brighter dots, and females have darker dots. [Figure 8C] This is a schematic diagram showing graphs representing scores from the Automated Visual Frailty Index (vFI) pipeline and the manual frailty index of mice. Figure 8C shows a scatter plot of age-specific FI scores from each score recorder (Score Recorder 1, darkest dot; Score Recorder 2, brighter dot; Score Recorder 3, paler dot; Score Recorder 4, palest dot). [Figure 9A] The graph shows the correlation between video metrics. Close overlap of points around the diagonal indicates a high correlation between the mean and median or IQR, and the standard deviation relative to each exponent. The graph in Figure 9A shows the correlation between the mean / median (x-axis) and median (y-axis) of video walking metrics. The diagonal corresponds to the maximum correlation (i.e., 1). [Figure 9B]The graph shows the correlation between video metrics. Close overlap of points around the diagonal indicates a high correlation between the mean and median or IQR, and the standard deviation relative to each exponent. The graph in Figure 9B shows the correlation between the interquartile range (IQR, x-axis) and the standard deviation (Stdev, y-axis) of video walking metrics. The diagonal corresponds to the maximum correlation (i.e., 1). [Figure 10A] This section shows video images, schematics, and graphs illustrating the morphological and behavioral measurements taken for scoring the Visual Frailty Index. Figures 10A–10I show sample features used in vFI. Figure 10A shows a single frame of top-down open-field video. [Figure 10B] This section presents video images, schematic diagrams, and graphs illustrating the morphological and behavioral measurements taken for scoring the Visual Frailty Index (vFI). Figures 10A–10I show sample features used in vFI. Figure 10B provides morphological features from elliptic fit and hindlimb distance measurements performed on the mouse frame frame by frame. The major and minor axes of the elliptic fit are considered as length and width, respectively. [Figure 10C] Images, schematics, and graphs illustrating the morphological and behavioral measurements taken for scoring the Visual Frailty Index are shown. Figures 10A–10I show the sample features used in vFI. Figure 10C shows that the median ellipse fit width and median hind limb distance taken across all mouse frames correlate highly with the FI score. [Figure 10D] This section shows video images, schematic diagrams, and graphs illustrating the morphological and behavioral measurements taken for scoring the Visual Frailty Index (vFI). Figures 10A–10I show sample characteristics used in vFI. Figure 10D shows the spatial, temporal, and whole-body coordination characteristics of gait used to create the metrics [Shepard K. et al., Cell Report 38, 110231 (January 2022)]. [Figure 10E]Images, schematics, and graphs illustrating the morphological and behavioral measurements taken for scoring the Visual Frailty Index are shown. Figures 10A–10I show the sample features used in vFI. Figure 10E shows the median step width in the interquartile range of apex-to-tail lateral displacement taken across all strides of the mouse, which correlates highly with the FI score. [Figure 10F] This section shows video images, schematic diagrams, and graphs illustrating the morphological and behavioral measurements taken for scoring the Visual Frailty Index. Figures 10A–10I show the sample features used in vFI. Figure 10F shows the spinal mobility measurements taken in each frame. dAC is the distance between points A and C (at the base of the head and the base of the tail, respectively), normalized to body length; dB is the distance from the midpoint of line AC to point B (mid-back); and aABC is the angle formed by points A, B, and C. When the mouse spine is straight, dAC and aABC are their maximum values, while dB is its minimum value. When the mouse spine is curved, dB is at its maximum value, while dAC and aABC are at their minimum values. [Figure 10G] This section shows video images, schematic diagrams, and graphs illustrating the morphological and behavioral measurements taken for scoring the Visual Frailty Index. Figures 10A–10I show the sample features used in vFI. Figure 10G shows the median dB taken across all mouse frames, and the median dB taken across frames in which the mouse is not walking shows a correlation with the FI score. [Figure 10H] This section shows video images, schematics, and graphs illustrating the morphological and behavioral measurements taken for scoring the Visual Frailty Index. Figures 10A–10I show sample features used in vFI. Figure 10H exhibits a wall-hinged event. The outline of an open-field wall is captured, a 5-pixel buffer (edge ​​line) is added, and a threshold is marked. The mouse's nasal point is tracked in each frame. A wall-hinged event is defined by a nasal point that completely exceeds the wall threshold. [Figure 10I]This section shows video images, schematic diagrams, and graphs illustrating the morphological and behavioral measurements taken for scoring the Visual Frailty Index (vFI). Figures 10A–10I show sample features used in vFI. Figure 10I shows the total number of hind-limb standing events, and the number of hind-limb standing events in the first 5 minutes of open-field video shows some correlation with the FI score. [Figure 11A] Graphs showing specific analyses of FI scores by age and sex, as well as graphs showing comparisons of male and female FI metrics, are presented. Figure 11A provides the distribution of FI scores for males and females when the data is divided into four age groups of equal range. Point x represents the midpoint of each age group range. Significant differences in the distribution of scores between males and females for that age group, as determined by the Mann-Whitney U trial. [Figure 11B] Graphs showing specific analyses of FI scores by age and sex, as well as graphs showing comparisons of FI metrics between males and females, are presented. Figure 11B shows the Pearson correlation between age and FI items in males compared to females. [Figure 11CD] Graphs showing specific analyses of FI scores by age and sex, as well as graphs showing comparisons of FI metrics between males and females, are shown. Figure 11C shows the Pearson correlation between video metrics and male FI scores compared to females. Figure 11D shows the Pearson correlation between male video metrics and age compared to females. The keys of open field, walking, and genetic engineering are applied to Figures 11C and 11D. [Figure 12A] This provides an embodiment of age and frailty prediction from video features. Figure 12A provides a graphic diagram showing different fitted models. [Figure 12B] This document provides an embodiment of age and frailty prediction from video features. Figure 12B shows that video features are more accurate than clinical frailty index items for predicting age. The performance of random forest models was compared using the frailty parameter (FRIGHT) and video-generated features (vFRIGHT) for age prediction. [Figure 12C] This provides embodiments for predicting age and frailty from video features. Figure 12A provides a graphical diagram showing different fitted models. Figure 12C shows the performance of an ordinal regression model (classifier) ​​in terms of accuracy (using a model trained on training data to accurately predict the value of the frailty parameter in the test). The black dotted line superimposed on the plot shows the accuracy obtained by whether a certain value was inferred instead of using video features. Video features were found to encode useful information that improves the model's ability to accurately predict frailty parameter values. [Figure 12D] This provides an embodiment of age and frailty prediction from video features. Figure 12D shows a comparison of four models (LR*, SVM, RF, XGB) that predict FI scores from video features in terms of mean absolute error (MAE), where R2 indicates that RF is superior to the other models. [Figure 12E] This provides embodiments for predicting age and frailty from video features. Figure 12A provides a graphical diagram showing different fitted models. Figure 12E shows the uncertainty in predicting age (column 1) and FI score (column 3), plotted as a function of age (weeks). Black curves indicate poor fit. These plots show low uncertainty when predicting age and FI score in very young mice. We plotted the distribution of prediction interval (PI) widths and found that the PI width for predicting age was wider (increased prediction uncertainty) for mice belonging to the middle-aged group (M). Similarly, the PI width for predicting FI score increased with age in the data. [Figure 12F] This provides an embodiment of age and frailty prediction from video features. Figure 12A provides a graphical diagram showing different fitted models. Video feature figure 12F shows residuals versus exponential and predicted FI score versus true for training (columns 1 and 2) and test sets (columns 3 and 4) for the RF model. [Figure 12G]This provides embodiments for predicting age and frailty from video features. Figure 12A provides a graphical diagram showing different fitted models. Figure 12G shows residuals versus exponents and predicted age versus true for the training (columns 1 and 2) and test sets (columns 3 and 4) for the RF model. [Figure 13A] We provide quantile regression modeling of vFI using a generalized random forest. Figure 13A shows variable importance indices for three quantile random forest models (lower tail - Q.025, median - Q.50, upper tail - Q.975). Lower tail and upper tail mice correspond to mice with low and high frailty scores, respectively. [Figure 13B] This provides a quantile regression model of vFI using a generalized random forest. Figure 13B provides a marginal ALE plot showing how much a feature influences the model prediction on average. For example, the average predicted FI score increases with increasing step size, but fits to values ​​greater than 3 for mice belonging to the lower and upper tails. [Figure 13C] This provides quantile regression modeling of vFI using a generalized random forest. Figure 13C provides a plot illustrating how strongly features interact with each other. [Figure 13D] This provides a quantile regression model of vFI using a generalized random forest. Figures 13D–13E show ALE quadratic interaction plots of step width and step length 1 for predicted FI scores (Figure 13E: width and length). When marginal effects from features have already been considered, lighter colors represent predictions above the mean, and darker colors represent predictions below the mean. The plot in Figure 13D (or E) shows a weak (or strong) interaction between step width and step length 1 (or width and length), respectively. Larger step width and step length 1 increase the vFI score. [Figure 13E]This provides a quantile regression model of vFI using a generalized random forest. Figures 13D–13E show ALE quadratic interaction plots of step width and step length 1 for predicted FI scores (Figure 13E: width and length). When marginal effects from features have already been considered, lighter colors represent predictions above the mean, and darker colors represent predictions below the mean. The plot in Figure 13D (or E) shows a weak (or strong) interaction between step width and step length 1 (or width and length), respectively. Larger step width and step length 1 increase the vFI score. [Figure 14-1] This provides a list showing the characteristic correlation with the FI score. [Figure 14-2] This provides a list showing the characteristic correlation with the FI score. [Figure 15-1] This provides a list showing the correlation (Pearson) between vFI features and age. [Figure 15-2] This provides a list showing the correlation (Pearson) between vFI features and age. [Figure 16] Provides a list showing the correlation between manual FI items and age. [Figure 17-1] This document provides an FI test sheet listing all items for manual frailty indexing. The text, shown in a light font, is a modification of Whitehead JC et al., Journal of Gerontology, Series A: Biomedical Science and Medical Science, 69, 621-632 (2014). [Figure 17-2] This document provides an FI test sheet listing all items for manual frailty indexing. The text, shown in a light font, is a modification of Whitehead JC et al., Journal of Gerontology, Series A: Biomedical Science and Medical Science, 69, 621-632 (2014). [Figure 17-3]This document provides an FI test sheet listing all items for manual frailty indexing. The text, shown in a light font, is a modification of Whitehead JC et al., Journal of Gerontology, Series A: Biomedical Science and Medical Science, 69, 621-632 (2014). [Figure 18A] This graph shows the estimated effect of the score recorder on clinical FI items. Figure 18A shows that the effect of the testing device differs depending on the FI item. [Figure 18B] The graph shows the estimated score recorder effect in the clinical FI item. Figure 18B shows the estimated random effect among the four score recorders in the dataset. [Figure 19A] Detailed modeling analysis graphs and plots are provided. Figure 19A shows the age distribution across 643 data points (533 mice). Distribution of manual FIadj scores across 643 data points (533 mice). [Figure 19B] Detailed modeling analysis graphs and plots are provided. Figure 19B shows results related to determining the contribution of frailty parameters to predicting age. The importance of all frailty parameter features was calculated, and gait disturbance, kyphosis, and hair bristles were determined to be the highest contributors. [Figure 19C] Detailed modeling analysis graphs and plots are provided. Figure 19C shows results demonstrating that the random forest regression model performed better than other models with the lowest mean squared error (RMSE) (p<2.2e-16, F3,147=59.53) and highest R2 (p<2.2e-16, F3,147=58.14) when compared using the repeated index ANOVA. [Figure 19D]Detailed graphs and plots of the modeling analysis are provided. Figure 19D shows that the vFRIGHT model performed better than the FRIGHT model with a lower RMSE (RMSEvFRIGHT=17.97±1.44, RMSEFRIGHT=20.62±4.78, p<6.1e-7, F1,49=32.84) and a higher R2 (RMSEvFRIGHT=0.78±0.04, RMSEFRIGHT=0.76±0.07, p<2.1e-8, F1,49=44.54) compared to the FRIGHT model using the repeated index ANOVA. [Figure 19E] Detailed modeling analysis graphs and plots are provided. Figure 19E shows a random forest regression model for predicting FI scores for unseen future data, which performed better than all other models with the lowest root mean square error (RMSE) (p<8.3e-14, F3,147=26.62) and the highest R2 (p<4.7e-14, F3,147=27.2). [Figure 19F] Detailed modeling analysis graphs and plots are provided. The plots in Figure 19F show the count distribution for individual frailty parameters (0th to 1st out of 3, 0.5th to 2nd out of 3, and 1st to 3rd out of 3 in each set of three values) for many parameters such as nasal discharge, rectal prolapse, vaginal uterus, and diarrhea, with the proportion of 0 counts being 1 (p0=1). Similarly, dermatitis, cataracts, eye discharge and swelling, microphthalmia, corneal opacity, tail rigidity, and malocclusion have p0>0.95. [Figure 19G] Detailed modeling analysis graphs and plots are provided. Figure 19G shows the residuals vs. index and predicted residuals vs. true for the training (rows 1 and 2) and test sets (rows 3 and 4) for a model that predicts age using the frailty index item, for both training and test data. [Figure 19H]Detailed modeling analysis graphs and plots are provided. Figures 19H and 19I show the results of the out-of-bag (OOB) error-based 95% prediction interval (PI) (gray line), quantifying the uncertainty of the point estimates / predictions (gray dots). There is one interval per test mouse, and approximately 95% of the PI intervals include the correct age (Figure 19I) and FI score (Figure 19H). The x-axis (test set index) is ordered in ascending order (left to right) by actual age / FI. [Figure 19I] Detailed modeling analysis graphs and plots are provided. Figures 19H and 19I show the results of the out-of-bag (OOB) error-based 95% prediction interval (PI) (gray line), quantifying the uncertainty of the point estimates / predictions (gray dots). There is one interval per test mouse, and approximately 95% of the PI intervals include the correct age (Figure 19I) and FI score (Figure 19H). The x-axis (test set index) is ordered in ascending order (left to right) by actual age / FI. [Figure 20] The results of testing Simpson's paradox are presented. Simpson

[35] showed that statistical relationships observed in a population can be reversed in all of the subgroups that make up that population, leading to erroneous conclusions from population data. To test the manifestation of Simpson's paradox in the data, a bimodal age distribution was split into two distinct unimodal distributions (clusters): less than 70 weeks old (L70) versus over 70 weeks old (U70). The dependent variable (frailty) was then plotted against each of the independent variables / features in the data, and a simple linear regression model was fitted separately to each subgroup, as well as to the aggregated data (data not shown). Correlations were quantified by measuring the slope of the linear fit of the feature (Y) at age (X). The slopes for L70, U70, and the whole (all) were calculated, and these slopes were plotted in descending order against the features relating to their relevance to the model (age was predicted from these features). For each set of three bars, the left bar is L70, the middle bar is U70, and the right bar is All. It was determined that Simpson's paradox does not appear in any of the top 15 features in the data. [Figure 21A]Graphs from further experiments to test the model's performance and parameters are provided. Figure 20A shows the results of comparing the performance of different feature sets—1) age only, 2) video, and 3) age + video—when predicting frailty. Age only was used as a feature for the linear (AgeL) and generalized additive nonlinear models (AgeG). The random forest model with video features (VideoRF) did not show a clear improvement over vFI prediction based on age only, but the model (AllRF) showed a clear improvement in prediction performance. The model (AllRF) includes video features with the lowest MSE (p<2.2e-16, F3,147=213.79, LMM post-hoc pairwise comparison with AgeG, t147=-12.21, FDR adjusted p<0.0001), lowest RMSE (p<2.2e-16, F3,147=172.88, LMM post-hoc pairwise comparison with AgeG, t147=-14.12, FDR adjusted p<0.0001), and highest R2 (p<2.2e-16, F3,147=171.12, LMM post-hoc pairwise comparison with AgeG, t147=14.07, FDR adjusted p<0.0001) plus age. This indicates that video features add important information about frailty that would not be applicable based on age alone. [Figure 21B]Graphs from further experiments to test the model's performance and parameters are provided. Figure 21B shows the results of an embodiment using selected animals with an inverse relationship between age and FI score, i.e., younger animals with higher FI scores and older animals with lower FI scores. Five (5) test sets were formed, including animals with these criteria, and the random forest (RF) model was trained on the remaining mice. The model using only video features (VideoRF) outperforms all other models for these mice with the lowest MSE (p<1.6e-08, F3.12=91.07, compared to AgeG LMM post-hoc pair, t12=13.60, FDR adjustment p<0.0001), lowest RMSE (p<1.6e-08, F3.12=93.88, compared to AgeG LMM post-hoc pair, t12=14.15, FDR adjustment p<0.0001), and highest R2 (p<1.31e-08, F3.12=94.32, compared to AgeG LMM post-hoc pair, t12=14.10, FDR adjustment p<0.0001). [Figure 21C] Graphs from further experiments to test the model's performance and parameters are provided. Figure 21C shows the results of a further investigation into the differences between age predictors and vFI predictors in terms of feature importance. Features lying along the diagonal are important for both age and vFI predictions. [Figure 21D] Graphs from further experiments to test the model's performance and parameters are provided. Figure 21D shows the results of predicting FI scores from video features extracted from shorter duration videos. We investigated the loss of accuracy in predicting age and FI scores using video features generated from shorter duration videos (first 5 minutes and 20 minutes). A random forest model trained on features generated from 60-minute videos was used as a baseline model for comparison. The loss of accuracy was reduced when shorter videos were used. [Figure 21E]Graphs from further experiments to test the model's performance and parameters are provided. Figure 21E shows research results to confirm how much training data is actually needed. Simulation tests were conducted, allocating different proportions of total data to training. As expected, there is a generally downward (upward) trend in MAE and RMSE (R², the increase in the proportion of data allocated to the training set). In fact, the smaller the training set (<80% training), the more similar training performance can be achieved. [Modes for carrying out the invention]

[0013] While renal aging is uniform, biological aging is heterogeneous. Clinically, this heterogeneity manifests in health status and mortality, distinguishing healthy aging from unhealthy aging. The clinical frailty index plays a vital role as an important tool for capturing health status in old age. The frailty index has been adapted for use in mice and is an effective predictor of mortality risk. To advance the understanding of biological aging, a high-throughput approach to preclinical studies is needed. However, currently, frailty indexing in mice is manual and relies on trained / expert manual score recorders, thus limiting the scalability and reliability of frailty index generation.

[0014] This disclosure relates to an automated visual frailty system for processing video data of a subject and generating a visual frailty score for that subject. The automated visual frailty system of this disclosure (e.g., System 100 shown in Figure 1) may use one or more machine learning-based techniques to determine the visual frailty score of a subject and may operate on video data from an open-field assay. The automated visual frailty system may determine the visual frailty score of a subject based on biological aging features extracted from the video data. In some embodiments, the automated visual frailty system may extract morphological features, gait and postural features, behavioral features, and other features from the video data that can be used to determine the visual frailty score. The automated visual frailty system may improve accuracy, reproducibility, scalability, and efficiency in generating a frailty index for a subject.

[0015] The System 100 of this Disclosure can be operated using various components shown in Figure 1. System 100 may include an image capture device 101, a device 102, and one or more systems 105, all connected via one or more networks 199. The image capture device 101 may be part of another device (e.g., device 600), or contained within such a device, or connected to such a device, and may also be a camera, a high-speed video camera, or another type of device capable of capturing images or video. In addition to or instead of the image capture device, device 101 may include motion detection sensors, infrared sensors, temperature sensors, ambient condition detection sensors, and other sensors configured to detect various characteristics / environmental conditions. Device 102 may be a laptop, desktop, tablet, smartphone, or another type of computing device capable of displaying data, and may also include one or more components described below in relation to device 600.

[0016] The image capture device 101 may capture video (or one or more images) of a subject and transmit video data 104 representing the video to the system(s) 105 for processing as described herein. The video may include the movement of a subject in an open field arena. In some cases, the video data 104 may correspond to images (image data) captured by the device 101 at specific time intervals, so that the images capture the movement of the subject over a period of time. The system(s) 105 may include one or more components shown in Figure 1 and may be configured to process the video data 104 to determine the visual frailty score of the subject. The system(s) 105 may generate a visual frailty score 162 corresponding to the subject. The system(s) 105 may transmit the visual frailty score 162 to the device 102 for output to the user to observe the results of processing the video data 104.

[0017] In some embodiments, the video data 104 may include video of two or more subjects, and the system(s) 105 may process the video data 104 to determine the characteristics and visual frailty score of each subject represented in the video data 104.

[0018] The system(s) 105 may be configured to determine various features from video data 104 about a subject. To determine these features and to determine a visual frailty score, the system(s) 105 may include several different components. As shown in Figure 1, the system(s) 105 may include a point tracking component 110, a gait and posture analysis component 120, an ellipse generation component 130, an open field analysis component 140, a grooming behavior analysis component 150, and a visual frailty analysis component 160. The system(s) 105 may include fewer or more components than those shown in Figure 1. In some embodiments, these various components may be located on the same physical system 105. In other embodiments, one or more of the various components may be located on different / separate physical systems 105. Communication between the various components may occur directly or via a network 199. Communication between device 101, system(s) 105, and device 102 may occur directly or via network 199.

[0019] In some embodiments, one or more components shown as part of the system(s) 105 may be located in device 102, or in a computing device (e.g., device 600) connected to the image acquisition device 102.

[0020] At a high level, the system(s) 105 may be configured to process video data 104 to determine point data (which may be referred to as posture estimation data in the following embodiments). Using the point data, the system(s) 105 may determine various features corresponding to the subject's movement in the video, such as gait measurements, spinal measurements, hind limb standing events, and hind limb measurements. Details regarding the determination of point data and various features from point data are described below in relation to Figure 2. The system(s) 105 may also be configured to determine ellipse data (which may be referred to as ellipse fit in the following embodiments). Using the ellipse data, the system(s) 105 may determine morphological data for the subject. Details regarding the determination of ellipse data and morphological data are described below in relation to Figure 3. The system(s) 105 may also be configured to use video data 104 to determine the subject's behavioral features. Details regarding the determination of behavioral features are described below in relation to Figure 4. Using the determined features / data, the system(s) 105 may then determine the subject's visual frailty score 162. Details regarding the determination of the Visual Frailty Score 162 are explained below in relation to Figure 5.

[0021] Figure 2 is a flowchart of a process 200 for determining various data of a subject using point data derived from video data, according to an embodiment of the present disclosure. One or more of the steps 200 may be performed in a different order / sequence than that shown in Figure 2. One or more steps of the process 200 may be performed by the point tracking component 110 and / or the gait and posture analysis component 120.

[0022] In step 202, the point tracking component 110 may receive video data 104 representing the movement of an object. In step 204, the point tracking component 110 may process the video data 104 to determine point data 112 that track the movement of a set of multiple body parts of the object. The point tracking component 110 may be configured to identify different body parts of the object. These body parts may be identified using different point data such that a first point data corresponds to a first body part, a second point data to a second body part, and so on. In some embodiments, the point data may be one or more pixel positions / coordinates (x, y) corresponding to a body part. Thus, the point data 112 may include multiple point data corresponding to multiple body parts. The point tracking component 110 may be configured to identify pixel positions corresponding to specific body parts in one or more video frames of the video data 104. The point tracking component 110 may track the movement of a specific body part over the duration of the video by identifying the corresponding pixel positions in the video. The point data 112 may indicate the position of a specific body part in a particular frame of the video. The point data 112 may include the locations of all body parts identified and tracked by the point tracking component 110 across multiple frames of the video data 104. The point data 112 may also include a confidence score for the location of a particular body part within a particular video frame. The confidence score may indicate how reliable the point tracking component 110 is in determining that particular location. The confidence score may also be the probability / likelihood of a particular body part being at that particular location.

[0023] In some embodiments, when the subject is a mouse, the point tracking component 110 may identify and track body parts such as the nose, left ear, right ear, base of the neck, left forelimb, right forelimb, mid-spine, left hindlimb, right hindlimb, base of the tail, mid-tail, and tip of the tail.

[0024] The point data 112 may be a vector, array, or matrix representing the pixel coordinates of various body parts on multiple video frames. For example, the point data 112 may be [frame1={nose:(x1, y1); right hind limb:(x2, y2)}],[frame2={nose:(x3, y3); right hind limb:(x4, y4)}], etc. For each frame, the point data 112 may, in some embodiments, include the coordinates of at least 12 pixels representing 12 parts / body parts of the object that the point tracking component 110 is configured to track.

[0025] The point tracking component 110 may implement one or more pose estimation techniques. The point tracking component 110 may include one or more machine learning models configured to process the video data 104. In some embodiments, one or more machine learning models may be neural networks such as deep neural networks, deep convolutional neural networks, or iterative neural networks. In other embodiments, one or more machine learning models may be models of a type other than neural networks. The ML model of the point tracking component 110 may be configured for 3D unmarked pose estimation based on transfer learning using a deep neural network.

[0026] The point tracking component 110 may be configured to determine the point data 112 with high accuracy and precision, since the visual frailty score 162 may be sensitive to errors in the point data 112. The point tracking component 110 may implement an architecture that maintains high-resolution features throughout the machine learning model stack, thereby maintaining spatial accuracy. In some embodiments, the architecture of the point tracking component 110 may include one or more transposed convolutions to cause a match between the resolution of the heatmap output and the resolution of the video data 104. The point tracking component 110 may be configured to determine the point data 112 at near real-time speeds and may run on a high-performance GPU. The point tracking component 110 may be configured to allow for easy modification and expansion. In some embodiments, the point tracking component 110 may be configured to generate inferences at a fixed scale rather than processing at multiple scales, thereby saving computational resources and time.

[0027] In some embodiments, the video data 104 may track the movement of a single object, and the point tracking component 110 may be configured not to perform any object detection techniques / algorithms to detect the object within the video frame. In other embodiments, the video data 104 may track the movement of two or more objects, and the point tracking component 110 may be configured to identify one object from another object in the video data 104 by performing object detection techniques.

[0028] In step 206, the gait and posture analysis component 120 may process the point data 112 to determine the gait measurement data 122 of the subject. The gait and posture analysis component 120 may use the point data 112 to determine the distances and / or angles between various body parts of the subject.

[0029] The gait and posture analysis component 120 may determine the distances between various body parts of the subject(s) and generate one or more distance vectors. For each video frame of the video data 104, the gait and posture analysis component 120 may determine a first distance between two body parts (a first pair), a second distance between another two body parts (a second pair), and so on, and the first and second distances may be included in the distance vector. In some embodiments, the gait and posture analysis component 120 may determine a first distance feature vector representing the distance between a first pair of body parts for multiple video frames, a second distance feature vector representing the distance between a second pair of body parts for multiple video frames, and so on. Each value of the first distance vector may represent the distance between a first pair of body parts for different corresponding video frames of the video data 104. In some embodiments, the distance vectors may be included in the gait measurement data 122 used to determine the visual frailty score 162. In other embodiments, the distance vectors may be used by the gait and posture analysis component 120 to determine the data included in the gait measurement data 122.

[0030] The gait and posture analysis component 120 may determine angles between various body parts of the subject(s) and generate one or more angle vectors. The gait and posture analysis component 120 may determine first angle data between three (first trio) body parts, second angle data between another three (second trio) body parts, and so on, for multiple video frames. The gait and posture analysis component 120 may determine first angle vectors representing angles between first trios of body parts across multiple video frames, second angle vectors representing angles between second trios of body parts across multiple video frames, and so on. Each value of the first angle vector may represent angles between first trios of body parts for different corresponding video frames of the video data 104. In some embodiments, the angle vectors may be included in gait measurement data 122 used to determine a visual frailty score 162. In other embodiments, the angle vectors may be used by the gait and posture analysis component 120 to determine data included in the gait measurement data 122.

[0031] In some embodiments, the gait and posture analysis component 120 may determine gait and posture metrics. As used herein, gait metrics may refer to metrics derived from the movement of the subject's feet. Gait metrics may include, but are not limited to, step width, step length, stride length, velocity, angular velocity, and limb load coefficient. As used herein, posture metrics may refer to metrics derived from the movement of the subject's whole body. In some embodiments, posture metrics may be based on the movement of the subject's nose and tail. Posture metrics may include, but are not limited to, lateral displacement of the nose, lateral displacement of the base of the tail, lateral displacement of the tip of the tail, lateral displacement phase offset of the nose, displacement phase offset of the base of the tail, and displacement phase offset of the tip of the tail. One or more of the gait and posture metrics may be included in the gait measurement data 122. In some embodiments, each of the gait and postural metrics may be provided to the visual frailty analysis component 160 as separate inputs rather than as a collective input via the gait measurement data 122.

[0032] The gait and posture analysis component 120 may determine one or more gait and posture metrics for each stride. The gait and posture analysis component 120 may determine the stride interval(s) represented within the video frame of the video data 104. In some embodiments, the stride interval may be based on the stance phase and the swing phase. In an exemplary embodiment, the approach for detecting the stride interval is based on the periodic structure of gait. During a stride cycle, each foot may have a stance phase and a swing phase. During the stance phase, the foot supports the weight of the subject and is in static contact with the ground. During the swing phase, the foot moves forward and does not support the weight of the subject. Hereinafter, the transition from the stance phase to the swing phase is referred to as the toe-off event, and the transition from the swing phase to the stance phase is referred to as the foot-contact event.

[0033] The gait and posture analysis component 120 may determine multiple stance and swing phases represented by the duration of the video data 104. In an exemplary embodiment, the stance and swing phases may be determined with respect to the hind limb of the subject. The gait and posture analysis component 120 may also calculate the velocity of the forelimb and infer that the foot is in the stance phase when the velocity falls below a threshold, and infer that the foot is in the swing phase when the velocity exceeds the threshold. The gait and posture analysis component 120 may determine that a foot contact event occurs in the video frame, resulting in a transition from the swing phase to the stance phase.

[0034] The gait and posture analysis component 120 may also determine the stride intervals represented during that period. The stride intervals may span multiple video frames of the video data 104. The gait and posture analysis component 120 may determine, for example, that a 10-second period has five stride intervals, and that one of the five stride intervals is represented within five consecutive video frames of the video data 104. In an exemplary embodiment, the left hind limb contact event may be defined as an event that separates / distinguishes a stride interval. In another exemplary embodiment, the right hind limb contact event may be defined as an event that separates / distinguishes a stride interval. In yet another exemplary embodiment, a separated stride interval may be defined by using a combination of the left hind limb contact event and the right hind limb contact event. In some other embodiments, the gait and posture analysis component 120 may determine the stance and swing phases with respect to the forelimbs, calculate the foot velocity based on the forelimbs, and further distinguish between stride intervals based on the foot contact events of the right and / or left forelimbs. In some other embodiments, the transition from the stance to the swing phase, i.e., the toe-off event, may be used to separate / distinguish the stride intervals.

[0035] In some embodiments, the inference quality of the point data 112 for the forelimbs (determined by the point tracking component 110) is unreliable in some cases, making it preferable to determine the stride interval based on the hindlimb ground contact events rather than the forelimb ground contact events. This may be a result of the difficulty in accurately determining the position of the forelimbs because they are often more obscured than the hindlimbs in the plan view.

[0036] The gait and posture analysis component 120 may filter the determined stride intervals to determine which stride intervals to use to determine the gait and posture metrics. In some embodiments, such filtering may remove incorrect or unreliable stride intervals. In some embodiments, criteria for removing stride intervals include, but are not limited to, unreliable point data estimates, physiologically unrealistic point data estimates, missing right hind limb contact events, and insufficient whole-body velocity of the subject (e.g., velocity less than 10 cm / sec). In some embodiments, the filtering of stride intervals may be based on the confidence level in determining the point data 112 used to determine the stride intervals. For example, stride intervals determined at a confidence level below a threshold may be removed from a set of stride intervals used to determine the gait and posture metrics. In some embodiments, the first and last strides are removed within a continuous sequence of strides to avoid noise from the start and stop movements in the data to be analyzed. For example, in a sequence of seven strides, a maximum of five strides would be used for analysis.

[0037] After determining the stride interval represented in the video data 104, the gait and posture analysis component 120 may determine the gait and posture metrics included in the gait measurement data 122. The gait and posture analysis component 120 may use the point data 112 to determine the step length of each stride interval. In some embodiments, the point data 112 may relate to the left hind limb, left forelimb, right hind limb, and right forelimb of the subject. In some embodiments, the step length may be the distance between the left forelimb and the right hind limb relative to the stride interval. In some embodiments, the step length may be the distance between the right forelimb and the left hind limb relative to the stride interval. In some embodiments, the step length may be the distance the right hind limb moves beyond the immediate ground contact position of the left hind limb.

[0038] The gait and posture analysis component 120 may use point data 112 to determine the stride length for each stride interval. The gait and posture analysis component 120 may determine the stride length for each stride interval during that period. In some embodiments, the point data 112 may relate to the left hind limb, left forelimb, right hind limb, and right forelimb of the subject. In some embodiments, the stride length may be the distance between the left forelimb and left hind limb for each stride interval. In some embodiments, the stride length may be the distance between the right forelimb and right hind limb. In some embodiments, the stride length may be the total distance the left hind limb travels with respect to the stride from the toe-off event to the foot-contact event.

[0039] The gait and posture analysis component 120 may use point data 112 to determine the step width with respect to the stride interval. The gait and posture analysis component 120 may determine the step width for each stride interval during that period. In some embodiments, the point data 112 may relate to the left hind limb, left forelimb, right hind limb, and right forelimb of the subject. In some embodiments, the step width is the distance between the left forelimb and the right forelimb. In some embodiments, the step width is the distance between the left hind limb and the right hind limb. In some embodiments, the step width is the average of the lateral distances separating the hind limbs. This may be calculated as the length of the shortest line segment connecting the ground contact position of the right hind limb and the straight line connecting the toe-off position and subsequent foot contact position of the left hind limb.

[0040] The gait and posture analysis component 120 may use point data 112 to determine foot utterances related to stride intervals. The gait and posture analysis component 120 may determine foot velocity for each stride interval during that period. In some embodiments, the point data 112 may relate to the subject's left hind limb, right hind limb, left forelimb, and right forelimb. In some embodiments, foot velocity may be the velocity of one foot at the time of stride interval. In some embodiments, foot velocity may be the velocity of the subject, or it may be based on the base of the subject's tail.

[0041] The gait and posture analysis component 120 may use point data 112 to determine the stride velocity with respect to the stride interval. The gait and posture analysis component 120 may determine the stride velocity for each stride interval during that period. In some embodiments, the point data 112 may relate to the base of the tail. In some embodiments, the stride velocity may be determined by determining a set of velocity data about the subject based on the movement of the base of the subject's tail at a set of video frames representing a stride interval. Each velocity data in the set of velocity data may correspond to one video frame in a set of video frames. The stride velocity may be calculated by averaging (or combining in other ways) the sets of velocity data.

[0042] The gait and posture analysis component 120 may use point data 112 to determine a limb load coefficient for each stride interval. The gait and posture analysis component 120 may determine a limb load coefficient for each stride interval over a given period. In some embodiments, the point data 112 may relate to the right and left hind limbs of the subject. In some embodiments, the limb load coefficient for a stride interval may be the average of a first load coefficient and a second load coefficient. The gait and posture analysis component 120 may determine a first stance time representing the amount of time the right hind limb is in contact with the ground during a stride interval, and then determine a first load coefficient based on the first stance time and the length of the stride interval. The gait and posture analysis component 120 may determine a second stance time representing the amount of time the left hind limb is in contact with the ground during a stride interval, and then determine a second load coefficient based on the second stance time and the length of the stride interval. In other embodiments, the limb load coefficient may be based on the stance time and load coefficient of the forelimbs.

[0043] The gait and posture analysis component 120 may determine the angular velocity for each stride interval using point data 112. The gait and posture analysis component 120 may determine the angular velocity for each stride interval over that period. In some embodiments, the point data 112 may relate to the base of the tail and the base of the neck of the subject. The gait and posture analysis component 120 may determine a set of vectors connecting the base of the tail and the base of the neck, in which case each vector in the set corresponds to one frame of a set of frames for the stride interval. The gait and posture analysis component 120 may determine the angular velocity based on a set of vectors. The vectors may represent angles of the subject, and the first derivative of the angle value may be the angular velocity with respect to the frame. In some embodiments, the gait and posture analysis component 120 may determine the stride angular velocity by averaging the angular velocities with respect to the frame for the stride interval.

[0044] The gait and posture analysis component 120 may determine the lateral displacement of the nose, tail tip, and tail base of the subject with respect to each individual stride interval. Based on the lateral displacement of the nose, tail tip, and tail base, the gait and posture analysis component 120 may determine the displacement phase offset of each body part of the subject. To determine the lateral displacement, the gait and posture analysis component 120 may first use point data 112 to determine a displacement vector with respect to the stride interval. The gait and posture analysis component 120 may determine a displacement vector with respect to each stride interval during that period. In some embodiments, the point data 112 may relate to the mid-spine of the subject. The stride interval may span multiple video frames. In some embodiments, the displacement vector may be a vector connecting the mid-spine in the first video frame of the stride interval to the mid-spine in the last video frame of the stride interval.

[0045] The gait and posture analysis component 120 may use point data 112 and displacement vectors to determine the lateral displacement of the subject's nose with respect to the stride interval. The gait and posture analysis component 120 may determine the lateral displacement of the nose with respect to each stride interval during that period. In some embodiments, the point data 112 may relate to the mid-spine and nose of the subject. In some embodiments, the gait and posture analysis component 120 may determine a set of multiple lateral displacements of the nose, in which case each lateral displacement of the nose may correspond to one video frame of the stride interval. The lateral displacement may be the vertical distance of the nose from the displacement vector with respect to the stride interval within each video frame. In some embodiments, the gait and posture analysis component 120 may subtract the minimum distance from the maximum distance and divide it by the subject's body length so that the displacement measured in a larger subject is equivalent to the displacement measured in a smaller subject.

[0046] The gait and posture analysis component 120 may determine the displacement phase offset of the nose using a set of multiple lateral displacements of the nose relative to the stride interval. The gait and posture analysis component 120 may generate a smooth curve of the lateral displacement of the nose relative to the stride interval by performing interpolation using a set of multiple lateral displacements of the nose, and then use the smooth curve of the lateral displacement of the nose to determine at what point in the stride interval the maximum displacement of the nose occurs. The gait and posture analysis component 120 may determine a percentage stride position that represents the percentage of the stride interval completed when the maximum displacement of the nose occurs. In some embodiments, the gait and posture analysis component 120 may perform cubic spline interpolation to generate a smooth curve of displacement, and for cubic interpolation, the maximum displacement may occur at a point between video frames.

[0047] The gait and posture analysis component 120 may use point data 112 and displacement vectors to determine the lateral displacement of the tail base of the subject with respect to the stride interval. The gait and posture analysis component 120 may determine the lateral displacement of the tail base with respect to each stride interval during that period. In some embodiments, the point data 112 may relate to the mid-spine and tail base of the subject. In some embodiments, the gait and posture analysis component 120 may determine a set of multiple lateral displacements of the tail base, in which case each lateral displacement of the tail base may correspond to one video frame of the stride interval. The lateral displacement may be the vertical distance of the tail base from the displacement vector with respect to the stride interval within each video frame. In some embodiments, the gait and posture analysis component 120 may subtract the minimum distance from the maximum distance and divide it by the body length of the subject so that the displacement measured in a larger subject is equivalent to the displacement measured in a smaller subject.

[0048] The gait and posture analysis component 120 may determine the tail base displacement phase offset using a set of lateral displacements of the tail base relative to the stride interval. The gait and posture analysis component 120 may generate a smooth curve of the lateral displacement of the tail base relative to the stride interval by performing interpolation using a set of lateral displacements of the tail base, and then use the smooth curve of the lateral displacement of the tail base to determine at what point in the stride interval the maximum displacement of the nose occurred. The gait and posture analysis component 120 may determine a percentage stride position representing the percentage of the stride interval completed when the maximum displacement of the tail base occurred. In some embodiments, the gait and posture analysis component 120 may perform cubic spline interpolation to generate a smooth curve of displacement, and for cubic interpolation, the maximum displacement may occur at a point between video frames.

[0049] The gait and posture analysis component 120 may use point data 112 and displacement vectors to determine the lateral displacement of the tail tip of the subject with respect to the stride interval. The gait and posture analysis component 120 may determine the lateral displacement of the tail tip with respect to each stride interval during that period. In some embodiments, the point data 112 may relate to the mid-spine and tail tip of the subject. In some embodiments, the gait and posture analysis component 120 may determine a set of multiple lateral displacements of the tail tip, in which case each lateral displacement of the tail tip may correspond to one video frame of the stride interval. The lateral displacement may be the vertical distance of the tail tip from the displacement vector with respect to the stride interval within each video frame. In some embodiments, the gait and posture analysis component 120 may subtract the minimum distance from the maximum distance and divide it by the body length of the subject so that the displacement measured in a larger subject is equivalent to the displacement measured in a smaller subject.

[0050] The gait and posture analysis component 120 may determine the tail root displacement phase offset using a set of multiple lateral displacements of the tail tip relative to the stride interval. The gait and posture analysis component 120 may generate a smooth curve of the lateral displacement of the tail tip relative to the stride interval by performing interpolation using a set of multiple lateral displacements of the tail tip, and then use the smooth curve of the lateral displacement of the tail tip to determine at what point in the stride interval the maximum nose displacement occurred. The gait and posture analysis component 120 may determine a percentage stride position representing the percentage of the stride interval completed when the maximum tail tip displacement occurred. In some embodiments, the gait and posture analysis component 120 may perform cubic spline interpolation to generate a smooth curve of displacement, and for cubic interpolation, the maximum displacement may occur at a point between video frames.

[0051] In some embodiments, the gait and posture analysis component 120 may include a statistical analysis component that may take gait and posture metrics as input for some statistical analysis. The size and speed of the subjects may influence the gait and / or posture metrics of the subjects. For example, subjects that move faster may gait differently than subjects that move slower. As a further example, subjects with larger bodies will gait differently than subjects with smaller bodies. However, in some cases, differences in stride speed (compared to control subjects) may be a definitive feature of gait and posture changes due to aging and frailty. The gait and posture analysis component 120 collects multiple repeated measurements for each subject (via video data 104 and via subjects in open areas), and each subject has a different number of strides, resulting in unbalanced data. Averaging over repeated strides yields one average value per subject, but this can be misleading because it removes variability and introduces a false sense of confidence. At the same time, classical linear models fail to distinguish between stable within-subject variability and between-subject fluctuations, which can lead to bias in statistical analysis. To address these issues, the gait and posture analysis component 120, in some embodiments, employs a linear mixed model (LMM) to separate between-subject variability from genotype-based between-subject variability. In some embodiments, the gait and posture analysis component 120 may capture main effects such as subject size, genotype, and age, and additionally, random effects related to within-subject variability. The technique of the present invention collects multiple repeated measurements at different ages of subjects, resulting in a nested hierarchical data structure. Examples of exemplary statistical models implemented in the gait and posture analysis component 120 are shown below as models M1, M2, and M3. These models follow standard LMM notation, where (genotype, body length, speed, test age) represents fixed effects and (subject ID / test age) (where test age is nested within the subject) represents random effects. M1: Phenotype ~ Genotype + Test Age + Body Length + (1|Mouse ID / Test Age) M2: Phenotype ~ Genotype + Test Age + Speed ​​+ (1|Mouse ID / Test Age) M3: Phenotype ~ Genotype + Test Age + Speed ​​+ Body Length + (1|Mouse ID / Test Age)

[0052] Model M1 takes age and body length as inputs, Model M2 takes age and speed as inputs, and Model M3 takes age, speed, and body length as inputs. In some embodiments, the model of the gait and posture analysis component 120 does not include the subject's sex as an effect because sex may have a high correlation with the subject's body length / size. In other embodiments, the model of the gait and posture analysis component 120 may take the subject's sex as input. Using point data 112 (determined by the point tracking component 110) makes it possible to determine the subject's size and speed with respect to these models. Therefore, no additional measurements are required for these variables with respect to the models.

[0053] One or more of the data included in the gait measurement data 122 may be circular variables (e.g., stride length, angular velocity, etc.), and the gait and posture analysis component 120 may implement a function of linear variables using a circular-linear regression model. Linear variables such as body length and velocity may be included as covariates in the model. In some embodiments, the gait and posture analysis component 120 may implement a multivariate outlier detection algorithm at the individual subject level to identify subjects with injury and developmental effects.

[0054] In some embodiments, the gait measurement data 122 may include, for the subject, one or more of the following: speed, velocity, angular velocity, step count, step length, step width, stride length, lateral displacement, limb load coefficient, time symmetry, number of strides per unit of video duration, and covered distance. Table 1 provides a list of video features and metrics used in one embodiment of the present invention.

[0055] Table 1 lists the characteristics of the video. [Table 1]

[0056] The angular velocity may be the first derivative of the angle of the object, determined by a vector connecting the base of the object's tail to the base of its neck. The lateral displacement may be the difference between the minimum and maximum values ​​of reference points (e.g., nose, base of tail, and tail tip) of the vertical distance from the object's displacement vector for each frame of the stride, normalized by the object's body length. The limb load coefficient may be a time quantity obtained by dividing the total time the feet are in contact with the ground by the total stride time calculated and averaged for each hind limb of the object. The velocity may be determined using the base of the tail. The step length may be the distance the right hind limb travels past the contact point of the opposite limb. The gait measurement data 122 may include two step lengths, one based on the contact of the left hind limb and the other based on the contact of the right hind limb. The step width may be the length of the shortest line segment connecting the contact point of the right hind limb and the toe-off of the left hind limb and the subsequent contact point of the foot. Stride length may be the total distance the left hind limb travels in terms of stride from toe-off to foot contact. Time symmetry may be the difference in ground contact time between the left and right hind limbs divided by the total ground contact time. The stride count may be the sum of all strides represented by the duration of video data 104. The distance covered may be the sum of spontaneous motor activities normalized by the time spent in the open field arena.

[0057] As shown in Figure 1, in some embodiments, the gait and posture analysis component 120 may also generate spinal measurement data 124. Referring to Figure 2, in step 208, the gait and posture analysis component 120 may process point data 112 to determine the spinal measurements of the subject. Spinal mobility measurements(s) may be determined for each video frame of the video data 104. The spinal mobility measurements for a video frame may include several different measurements. Using the point data 112, the gait and posture analysis component 120 may determine a first distance (dAC) between the base of the subject's head (point A) and the base of the subject's tail (point B). In some embodiments, the first distance may be normalized with respect to the subject's body length. Using the point data 112, the gait and posture analysis component 120 may determine a second distance (dB) between the midpoint of the line between the base of the head and the base of the tail (midpoint of line AC). Using point data 112, the gait and posture analysis component 120 may determine the angle (aABC) formed by the base of the head, the base of the tail, and the mid-back (points A, B, and C) of the subject. The spinal mobility measurements for the video frames may include the aforementioned first distance, second distance, and angle. The gait and posture analysis component 120 may determine the spinal mobility measurements for each video frame of the video data 104. The spinal measurement data 124 may be a vector or matrix containing the spinal mobility measurements for each video frame of the video data 104.

[0058] If the subject's spine is straight, the first distance (dAC) and angle (aABC) may be their maximum values, and the second distance (dB) may be their minimum values. If the subject's spine is bent, the second distance (dB) may be its maximum value, while the first distance (dAC) and angle (aABC) may be their minimum values. In determining the visual frailty score 162, the visual frailty analysis component 160 may consider the spinal measurement data 124 throughout the entire duration of the video. The visual frailty analysis component 160 may be configured to identify that older subjects may have less degree or less frequency of spinal flexion due to decreased flexibility or spinal mobility. For each of the three spinal mobility measurements, the visual frailty analysis component 160 may determine the mean, median, standard deviation, minimum, and maximum values ​​for all video frames of the video data 104. In some embodiments, the visual frailty analysis component 160 may identify which video frames are non-walking frames (i.e., frames in which the subject is not striding / walking). For these non-walking frames, the visual frailty analysis component 160 may separately determine the mean, median, standard deviation, minimum, and maximum values. The visual frailty analysis component 160 may be configured to identify the correlation between spinal measurement data 124 and the frailty of the subject. For example, in some cases, the median first distance (dAC) and the median second distance (dB) of the non-walking frames may increase (or decrease) with age.

[0059] As shown in Figure 1, in some embodiments, the gait and posture analysis component 120 may also generate hindlimb standing event data 126. Referring to Figure 2, in step 210, the gait and posture analysis component 120 may process point data 112 to determine the hindlimb standing event data 126 of the subject. In some embodiments, a hindlimb standing event may be defined as when the subject's nose (or other body part) passes a threshold boundary / line on the wall of the open field. The gait and posture analysis component 120 may be configured to identify the threshold boundary on the wall and use point data 112 to identify when the subject's nose passes / is on the threshold boundary (e.g., a video frame of video data 104). In other embodiments, a hindlimb standing event may be defined differently, for example, when the subject's foot passes a threshold boundary on the wall, when the subject spends a threshold amount of time at a corner of the open field, or when the subject spends a threshold amount of time above the threshold boundary on the wall.

[0060] The gait and posture analysis component 120 may be configured to use the coordinates of the boundary between the floor and wall of an open field, using a buffer of some pixels. Each time the point of the subject's nose passes through the buffer, this frame may be identified by the gait and posture analysis component 120 as containing / representing a hindlimb standing event. Each of a series of uninterrupted video frames in which the subject exhibits a hindlimb standing event may be identified by the gait and posture analysis component 120 as a hindlimb standing behavior. In some embodiments, the gait and posture analysis component 120 may determine the total number of hindlimb standing behaviors, the average length of each hindlimb standing behavior, the number of hindlimb standing behaviors in the first few minutes of the video (e.g., 5 minutes), and the number of hindlimb standing behaviors in the next few minutes (e.g., 5-10 minutes). The aforementioned measurements may be included in the hindlimb standing event data 126. The visual frailty analysis component 160 may be configured to identify a correlation between the hindlimb standing event data 126 and the subject's frailty. For example, older / frail subjects may stand less (or more).

[0061] As shown in Figure 1, in some embodiments, the gait and posture analysis component 120 may also generate hindlimb data 128. Referring to Figure 2, in step 212, the gait and posture analysis component 120 may process the point data 112 to determine the hindlimb data 128. For each video frame of the video data 104, the gait and posture analysis component 120 may determine the distance between the coordinates of the hindlimbs (from the point data 112). The hindlimb data 128 may be a vector containing the aforementioned distances for each of the video frames. The visual frailty analysis component 160 may determine the median, mean, standard deviation, maximum and / or minimum values ​​of the hindlimb distances for all video frames. The visual frailty analysis component 160 may be configured to identify a correlation between the hindlimb data 128 and the frailty of the subject. For example, the distance between the hindlimbs of an elderly / frail subject may be smaller (or longer) than that of a control subject.

[0062] As shown in Figure 1, in some embodiments, the system(s) 105 may use elliptic data 132 and open-field analysis components 140 to determine the morphological data 142 of the subject. Figure 3 is a flowchart of a process 300 for determining the morphological data of a subject using elliptic data derived from video data, according to an embodiment of the present disclosure.

[0063] In step 302, the ellipse generation component 130 may process the video data 104 to determine the ellipse data 132. In some embodiments, the ellipse generation component 130 may process the video data 104 to generate a segmentation mask that identifies objects in the video data 104, and then employ techniques to generate ellipse fits / representations for the objects. The ellipse generation component 130 may employ one or more techniques (e.g., one or more ML models) for object tracking in video / image data and may be configured to identify objects (e.g., which pixels represent objects and which pixels represent the background). The segmentation mask generated by the ellipse generator 130 may identify object pixels (sets of pixels) corresponding to objects and background pixels (another set of pixels separate and distinct from the object pixels) corresponding to the background. Using the segmentation mask, the ellipse generation component 130 may determine the ellipse fit. The ellipse fit may be an ellipse drawn around the body of the object. For different types of objects, the system(s) 105 may be configured to determine different shape fits / representations (e.g., circular fits, rectangular fits, square fits, etc.). The ellipse generation component 130 may determine the ellipse fit as a subset of the target pixels. The ellipse data 132 may include this subset of pixels corresponding to the ellipse fit. The ellipse generation component 130 may determine the ellipse fit for each video frame of the video data 104. The ellipse data 132 may be a vector or matrix of ellipse fit pixels for all video frames of the video data 104.

[0064] In some embodiments, the ellipse fit for an object may define several parameters of the object. For example, the ellipse fit may correspond to the position of the object and may include coordinates (e.g., x and y) representing the pixel position of the object (e.g., the center of the ellipse) in the video frame(s) of the video data 104. The ellipse fit may correspond to the length of the major axis and the length of the minor axis of the object. The ellipse fit may include the sine and cosine of the vector angle of the major axis. The angle may be defined with respect to the direction of the principal axis. The major axis may extend from the tip of the head or nose of the object to an end of the object's body, such as the base of the tail. In some embodiments, the ellipse data 132 may include the aforementioned measurements for all video frames of the video data 104.

[0065] In some embodiments, the ellipse generation component 130 may use an encoder-decoder architecture to determine a segmentation mask from the video data 104. In some embodiments, the ellipse generation component 130 may use a neural network model to determine an ellipse fit from the video data 104.

[0066] The ellipse data 132 may also include the confidence score(s) of the ellipse generation component 130 when determining the ellipse fit for the video frame. Alternatively, the ellipse data 132 may include the probability or likelihood of the ellipse fit corresponding to the subject.

[0067] In some embodiments, the ellipse generation component 130 may determine the ellipse fit of an object for each video frame of the video data 104. The ellipse fit may be represented as a set of pixels that define an ellipse around the object. The ellipse data 132 may be a vector or matrix containing the ellipse fit data for all video frames.

[0068] In step 304, the open-field analysis component 140 may process the elliptic data 132 to determine the morphological data 142. The morphological data 142 may correspond to the body composition of the subject (e.g., shape, size, length, weight, etc.). The open-field analysis component 140 may use the lengths of the major and minor axes of the elliptic fit for the subject (from the elliptic data 132) to determine the estimated length and width of the subject. The open-field analysis component 140 may determine the major and minor axes of the elliptic fit for each video frame of the video data 104. In some embodiments, the open-field analysis component 140 may determine the median, mean, standard deviation, maximum, and / or minimum values ​​for the lengths of the major and / or minor axes for all video frames of the video data 104. The open-field analysis component 140 may use one or more of the calculations described above to estimate the length and width of the subject. In some embodiments, the morphological data 142 may include the estimated length and width of the subject. The morphological data 142 may additionally or alternatively include the lengths of the major and minor axes for each ellipse fit in each video frame of the video data 104.

[0069] The visual frailty analysis component 160 may be configured to identify correlations between morphological data 142 and the subject's frailty. For example, changes in body composition and fat distribution may be observed along with the subject's aging.

[0070] In some embodiments, the open-field analysis component 140 may determine other data that can be used by the visual frailty analysis component 160. For example, the open-field analysis component 140 may use ellipse data 132 (e.g., ellipse center pixel / position) to determine video frames in which the subject is at the center of the open-field arena, and to determine the amount of time the subject spends at the center of the open-field arena during the duration of the video. As another example, the open-field analysis component 140 may use ellipse data 132 to determine video frames in which the subject is along the wall of the open-field arena, and to determine the amount of time the subject spends at the periphery (along the wall) during the duration of the video. As yet another example, the open-field analysis component 140 may use ellipse data 132 to determine video frames in which the subject is at a corner of the open-field arena, and to determine the amount of time the subject spends at the corner(s) during the duration of the video. As yet another example, the open-field analysis component 140 may use ellipse data 132 to determine the average distance from the center of the open-field arena to the position of the subject during the video. As yet another example, the open field analysis component 140 may use elliptic data 132 to determine the average distance from the perimeter / walls of the open field arena to the position of the object in the video. As yet another example, the open field analysis component 140 may use elliptic data 132 to determine the average distance from the corners of the open field arena to the position of the object in the video.

[0071] As shown in Figure 1, in some embodiments, the system(s) 105 may use a grooming behavior analysis component 150 to determine the behavior data 152 of a subject. Figure 4 is a flowchart of a process 400 for determining the behavior data 152 of a subject using video data 104 of a subject, according to an embodiment of the present disclosure. In step 402, the grooming behavior analysis component 150 may process the video data 104 to determine the behavior data 152 of a subject. The grooming behavior analysis component 150 may be configured to identify video frames of the video data 104 in which the subject exhibits a grooming behavior that may include at least one of licking its paws, washing one side of its face, washing both sides of its face, and licking its flanks. The grooming behavior analysis component 150 may process the video data 104 using one or more ML models to generate a number of predictions about whether one or more frames of the video data 104 represent a subject exhibiting a defined behavior. These ML models may be constructed using training data that includes video footage capturing the movement of one or more subjects, with each video frame containing a label identifying whether the subject is exhibiting grooming behavior. Such ML models may also be constructed using large training datasets.

[0072] Each ML model in the grooming behavior analysis component 150 may be configured with different initialization parameters or settings so that the ML model can have variations in terms of specific model parameters (learning speed, weights, batch size, etc.), and therefore yield different predictions (regarding the grooming behavior in question) when processing the same video frame.

[0073] The grooming behavior analysis component 150 may also process different representations of the video data 104. The grooming behavior analysis component 150 may determine different representations of the video data 104 by changing the orientation of the video. For example, one orientation may be determined by rotating the video 90 degrees to the left, another orientation may be determined by rotating the video 90 degrees to the right, and yet another orientation may be determined by reflecting the video along the horizontal or vertical axis. The grooming behavior analysis component 150 may process the video frame with the originally captured orientation and other different generated orientations. Based on processing different orientations, the grooming behavior analysis component 150 may generate different predictions about the grooming behavior of the subject. The grooming behavior analysis component 150 may use the different predictions determined as described above to make a final determination as to whether the subject is exhibiting grooming behavior in the video frame(s).

[0074] The final decision may be output in behavioral data 152. Behavioral data 152 may be a vector or a set of values ​​indicating whether the subject exhibits grooming behavior in a particular video frame. For example, behavioral data 152 may include a Boolean value for each video frame indicating whether the subject exhibited grooming behavior (e.g., 1 or 0, true or false, yes or no). Alternatively, behavioral data 152 may also include a score (e.g., confidence score, probability score) corresponding to whether the subject exhibited grooming behavior in a particular video frame.

[0075] The grooming behavior analysis component 150 may use the video data 104 to determine multiple sets of frames, where different sets (e.g., at least four sets) may represent different orientations of the video data. The first set of frames may be the original orientation of the video data 104 captured by the image capture device 101. The rotated set of frames may be the rotated orientation of the video data 104; for example, the first set of frames can be rotated 90 degrees to the left to generate a set of rotated frames. The reflected set of frames may be the reflected orientation of the video data 104; for example, the first set of frames may be reflected across the horizontal axis (or rotated 180 degrees) to generate a set of reflected frames. Another set of rotated frames may be another rotated orientation of the video data 104; for example, the first set of frames can be rotated 90 degrees to the right to generate another set of rotated frames. In other embodiments, a set of frames may be generated by manipulating the original set of frames in a different way (e.g., reflecting across the vertical axis, rotating by a different degree, etc.). In other embodiments, the grooming behavior analysis component 150 may process more or less orientation of the video data 104.

[0076] The grooming behavior analysis component 150 may employ at least four ML models. As part of processing the video data 104, the grooming behavior analysis component 150 may use the same ML models to process different sets of the aforementioned frames and generate different predictions. For example, the first ML model may process a first set(s) of frames to generate a first prediction representing the probability or likelihood that the subject exhibits grooming behavior during the first set of frames. The first ML model may process a rotated set(s) of frames to generate a second prediction representing the probability or likelihood that the subject exhibits grooming behavior during the rotated set(s). The first ML model may process a reflected set(s) of frames to generate a third prediction representing the probability or likelihood that the subject exhibits behavior during the reflected set(s). The first ML model may process other rotated sets(s) of frames to generate a fourth prediction representing the probability or likelihood that the subject exhibits behavior during the other rotated sets(s). In this way, the same first ML model can process different orientations of the video data 104 to generate different predictions for the movement of the same captured object.

[0077] As part of further processing the video data 104, the grooming behavior analysis component 150 may use another ML model to process different sets of the aforementioned frames to generate more predictions. For example, a second ML may process a first set(or more) of frames to generate a fifth prediction representing the probability or likelihood that the subject will exhibit behavior during the first set(or more) of frames. The second ML model may process a set(or more) of rotated frames to generate a sixth prediction representing the probability or likelihood that the subject will exhibit behavior during the rotated set(or more) of frames. The second ML model may process a set(or more) of reflected frames to generate a seventh prediction representing the probability or likelihood that the subject will exhibit behavior during the reflected set(or more) of frames. The second ML model may process other sets(or more) of rotated frames to generate an eighth prediction representing the probability or likelihood that the subject will exhibit behavior in the video represented by other sets(or more) of rotated frames. In this way, another ML model may process different orientations of the video data 104 to generate additional predictions for the movement of the same captured object. The probabilities may be values ​​in the range of 0.0 to 1.0, or values ​​in the range of 0 to 100, or other numerical ranges.

[0078] Each of the eight different predictions may be a data vector containing multiple probabilities (or scores), where each probability corresponds to each frame in the set and indicates the likelihood that the subject exhibits grooming behavior within the corresponding frame. For example, a prediction may include a first probability corresponding to the first frame of video data 104, a second probability corresponding to the second frame of video data 104, and so on.

[0079] In some embodiments, the set of video frames may include multiple video frames (e.g., 16 video frames), where each video frame is the duration of the video over a certain period (e.g., 30 milliseconds, 30 seconds, etc.). Each ML model may be configured to process the set of frames and determine the probability that the subject exhibits grooming behavior within the last frame of the set of frames. For example, if the set of frames has 16 frames, the output of the ML model would indicate whether or not the subject exhibits grooming behavior within the 16th frame of the set of frames. The ML model may be configured to use contextual information from other frames in the set of frames to make a prediction for the last frame. In other embodiments, the output of the ML model may determine the probability that the subject exhibits grooming behavior within another frame of the set of frames (e.g., an intermediate frame, the 8th frame, the first frame, etc.).

[0080] In some embodiments, the grooming behavior analysis component 150 may use four different ML models to process four different sets of orientations / frames, thereby generating 32 different predictions corresponding to frames.

[0081] The grooming behavior analysis component 150 may include an aggregation component for determining the final prediction shown in the behavior data 152 by processing different predictions determined by different ML models using different sets of frames. The aggregation component may be configured to merge, aggregate, or combine different predictions (e.g., the eight predictions described above) to determine the behavior data 152.

[0082] In some embodiments, the aggregation component may average the probabilities of each frame, and the behavior data 152 may be a data vector of the average probabilities of each frame in the video data 104. In some embodiments, the grooming behavior analysis component 150 may determine a behavior label for a frame (or number of frames) based on whether the corresponding average probability of the frame satisfies a condition (e.g., the probability is above a threshold probability / threshold), and the behavior label may be a Boolean value indicating whether the subject exhibited grooming behavior.

[0083] In other embodiments, the aggregation component may aggregate the probabilities of each frame, and the behavior data 152 may be a data vector of the summed probabilities of each frame in the video data 104. In some embodiments, the grooming behavior analysis component 150 may determine the behavior label of a frame based on whether the corresponding summed probabilities of the frames satisfy a condition (e.g., the probability is above a threshold probability / threshold).

[0084] In some embodiments, the aggregation component may be configured to select the maximum value (e.g., the highest probability) from the predictions for each frame as the final prediction for the frame. In other embodiments, the aggregation component may be configured to determine the median from the predictions as the final prediction for the frame.

[0085] In some embodiments, another component may be configured in a similar manner to the grooming behavior analysis component 150 to detect an object exhibiting a different predetermined behavior. This other component may process the video data 104 using more ML models. These ML models may be configured to detect specific behaviors using training data that includes video capturing the movement of an object(s), the training data including per-frame labels that identify whether the object is exhibiting a particular behavior. Such ML models may be configured using a large training dataset. Based on the configuration of the ML models, they may be configured to detect different behaviors.

[0086] In other embodiments, the grooming behavior analysis component 150 may employ other techniques for determining the behavioral data 152.

[0087] Behavioral data 152 may also include the number / number of video frames in which the subject exhibits grooming during the duration of the video, the length of each grooming behavior (consecutive video frames in which the subject is grooming), the average length of grooming behavior, the number of grooming behaviors during the duration of the video, and other metrics.

[0088] The visual frailty analysis component 160 may be configured to identify correlations between behavioral data 152 and the frailty of the subject. For example, older / frail subjects may groom less (or more) than control subjects.

[0089] Figure 5 is a flowchart showing the process for determining a visual frailty score using one or more of the data determined according to the processes of Figures 2 to 4, according to embodiments of the present disclosure. In step 502, the visual frailty analysis component 160 may process one or more of the determined data 122, 124, 126, 128, 142, and 152 using one or more ML models. In some embodiments, the visual frailty analysis component 160 may employ one ML model to process all of the features / data 122, 124, 126, 128, 142, and 152. In other embodiments, the visual frailty analysis component 160 may employ different / separate ML models to process each of the data 122, 124, 126, 128, 142, and 152. For example, the first ML model may be used to process the gait measurement data 122, and the second ML model may be used to process the spinal measurement data 124. In yet another embodiment, the visual frailty analysis component 160 may employ different / separate ML models to process the data 122, 124, 126, 128, 142, 152 based on how the data is derived and / or which components generate the data. For example, a first ML model may be used to process the gait measurement data 122, spine measurement data 124, hind limb standing event data 126, and hind limb data 128 determined by the gait and posture analysis component 120, while a second ML model may be used to process the morphological data 142.

[0090] In some embodiments, the visual frailty analysis component 160 may select different features / data based on the subject's age, sex, lineage, and / or other characteristics in order to determine the visual frailty score 162.

[0091] In step 504, the visual frailty analysis component 160 may determine the target visual frailty score 162. In some embodiments, the visual frailty analysis component 160 may determine different / multiple initial frailty scores based on processing different types of data input to the visual frailty analysis component 160, and then aggregate the different / multiple frailty scores to determine the final visual frailty score 162. When aggregating the results of processing different types of data, the visual frailty analysis component 160 may use weighted sum or weighted average techniques, and different types of data may have different corresponding weights. For example, the results of processing morphological data 142 may be associated with a first weight, while the results of processing spinal measurement data 124 may be associated with another weight.

[0092] The visual frailty score 162 may be a numerical value within a predetermined range. For example, the visual frailty score 162 may be a value such as 0-1, 0-10, 1-27, or 0-100.

[0093] In other embodiments, the visual frailty analysis component 160 may aggregate / combine the results of processing different types of data using a different ML model to determine the visual frailty score 162. In yet another embodiment, the visual frailty analysis component 160 may aggregate / combine the results of processing different types of data using a rule-based engine to determine the visual frailty score 162.

[0094] In yet another embodiment, the visual frailty analysis component 160 may be configured to use point data 112 and / or ellipse data 132, and the reliability of the point tracking component 110 and / or ellipse generation component 130 may be taken into consideration when determining the visual frailty score 162.

[0095] In some embodiments, the visual frailty analysis component 160 may determine the visual frailty score 162 based on a comparison / evaluation of data 122, 124, 126, 128, 142, 152 with respect to some stored / control data. The visual frailty analysis component 160 may select stored / control data based on the subject's age, sex, lineage, and / or characteristics.

[0096] In some embodiments, the visual frailty analysis component 160 may determine a visual frailty score 162 based on which factors / features / data are visible / obvious / detectable for the subject. The visual frailty analysis component 160 may determine further factors using the subject's data 122, 124, 126, 128, 142, and 152. The visual frailty analysis component 160 may sum the number of detected factors and divide the sum by the total number of factors considered. For example, the visual frailty analysis component 160 may determine that the subject has a gait disorder using gait measurement data 122, and that the subject has gained weight using morphological data 142. The detection of gait disorder and weight gain may be two factors detected for the subject out of 10 potential factors. Based on this, the visual frailty analysis component 160 may determine a visual frailty score 162 of 0.2 (2 / 10).

[0097] In some embodiments, the visual frailty analysis component 160 may employ multiple different types of models / algorithms to process different types of data. For example, the visual frailty analysis component 160 may include one or more of the following: linear regression models, penaltyd linear regression models, random forests, support vector machines, gradient boost models, extreme gradient boost models, and neural networks.

[0098] While Figure 1 shows a specific type of data, it should be understood that the visual frailty analysis component 160 may process additional or different types of data to determine the visual frailty score 162.

[0099] In some embodiments, the data shown in Figure 1 may be determined by different components and / or by different data extracted from the video data 104. For example, morphological data may be determined using point data 112. As another example, hindlimb data may be determined using ellipse data 132. As yet another example, behavioral data may be determined using point data 112.

[0100] In some embodiments, the visual frailty analysis component 160 may be configured / trained using data corresponding to manually generated frailty scores. Manual frailty scores may be generated by an observer / score recorder by observing videos of multiple different subjects. Several factors that the observer considers in generating manual frailty scores are listed in Figure 15. Videos of subjects may be annotated / labeled with the corresponding manual frailty scores and / or factors detected for the subjects when generating the manual frailty scores. The visual frailty analysis component 160 may be configured using such annotated videos. The factors / data considered in generating manual frailty scores may differ from the factors / data used to generate the visual frailty score 162.

[0101] subject Some aspects of the present invention involve determining a visual frailty score for a subject. As used herein, the term “subject” may refer to a human, a non-human primate, a cattle, a horse, a pig, a sheep, a goat, a dog, a cat, a bird, a rodent, or other suitable vertebrate or invertebrate. In certain embodiments of the present invention, the subject is a mammal, and in certain embodiments of the present invention, the subject is a human. In some embodiments, the subject used in the method of the present invention is a rodent, including but not limited to a mouse, a rat, a gerbil, a hamster, etc. In some embodiments of the present invention, the subject is a normal, healthy subject, and in some embodiments, the subject is known to have a disease or condition associated with frailty, is at risk of having a disease or condition associated with frailty, or is suspected of having a disease or condition associated with frailty. Diseases associated with frailty may include clinical features / symptoms such as muscle weakness, loss of balance, abnormal muscle fatigue, and muscle wasting. In certain embodiments of the present invention, the subject is an animal model of a disease or condition associated with frailty. For example, but not intended to be limiting, in some embodiments of the present invention, the subject is a mouse, which is an animal model of aging, having one or more features of frailty such as muscle weakness, loss of balance, abnormal muscle fatigue, muscle wasting, etc.

[0102] As a non-limiting example, the subjects evaluated by the methods and systems of the present invention may be animal models of pathological conditions, such as models for one or more of the following: aging, frailty, neurodegenerative diseases, neuromuscular diseases, muscle injuries, ALS, Parkinson's disease, multiple sclerosis, and muscular dystrophy. Such pathological conditions may be referred to herein as activity disorders.

[0103] In some embodiments of the methods of the present invention, the subject is a wild-type subject. As used herein, the term “wild-type” means the phenotype and / or genotype of a species in a typical form as it occurs in nature. In certain embodiments of the present invention, the subject is a non-wild-type subject, for example, a subject having one or more genetic modifications compared to the wild-type genotype and / or phenotype of the subject species. In some examples, the difference in the subject's genotype / phenotype compared to the wild-type is due to hereditary (germline) or acquired (somatic) mutations. Factors that may result in a subject exhibiting one or more somatic mutations include, but are not limited to, environmental factors, toxins, ultraviolet radiation, spontaneous errors in cell division, radiation, maternal infection, and chemicals, as well as teratogenic events.

[0104] In certain embodiments of the methods of the present invention, the subject is a genetically modified organism, also referred to as a genetically engineered subject. The genetically engineered subject may include pre-selected and / or intentional genetic modifications, and thus exhibit one or more genotypes and / or phenotypic traits that differ from those in the unmanipulated subject. In some embodiments of the present invention, by using conventional genetic engineering techniques, it is possible to create a genetically engineered subject that exhibits genotype and / or phenotypic differences compared to an unmanipulated subject of the same species. As a non-limiting example, a genetically engineered mouse may have functional gene products present in the mouse at deficient or reduced levels, and the phenotype of the genetically engineered mouse may be evaluated using the methods or systems of the present invention, and the results may be compared with results obtained from a control (control result).

[0105] In some embodiments of the present invention, subjects may be monitored using the visual frailty determination method or system of the present invention to detect the presence or absence of activity impairment or pathology. In certain embodiments of the present invention, the response of a test subject to an activity and / or movement pathology may be evaluated by using a test subject that constitutes an animal model of the pathology of the movement and / or movement. In addition, a test subject that constitutes an animal model of the pathology of movement and / or activity may be administered a candidate therapeutic agent or method and monitored using the gait monitoring method and / or system of the present invention, and the results may be used to determine the effectiveness of a candidate therapeutic agent for treating the pathology. The terms “activity” and “behavior” may be used interchangeably herein.

[0106] As described elsewhere in this specification, the methods and systems of the present invention may be configured to determine a visual frailty score for an object regardless of the object's physical characteristics. In some embodiments of the present invention, one or more of the object's physical characteristics may be pre-identified characteristics. For example, without being intended to limit, pre-identified physical characteristics may be one or more of body type, build, coat color, sex, age, and disease or pathological phenotype.

[0107] Diseases and Disabilities The methods and systems of the present invention can be used to assess frailty, activity, and / or behavior in subjects known to have, suspected to have, or at risk of having, a disease or condition associated with frailty. In some cases, frailty may be understood to be a condition associated with aging. For example, subjects may be elderly subjects and / or animal models of aging conditions. In certain embodiments of the present invention, frailty may be associated with a disease or condition that is not associated with aging but is not considered an aging condition. For example, muscle weakness may be a feature assessed using the methods of the present invention and may appear in young subjects, subjects that are not animal models of aging conditions, subjects that are animal models of aging conditions, or elderly subjects. In some embodiments, the disease and / or condition is associated with a level of significant reduction in activity or behavior, such as movement, muscle use, or stamina. In non-limiting examples, a test subject may be a subject with muscle wasting and muscle weakness, or a subject may be an animal model of a condition exhibiting muscle wasting and / or muscle weakness, etc. In any case, the methods of the present invention can be used to assess the subject and determine the subject's frailty status. The results of evaluating test subjects can be compared with the results of the control group. Non-limiting examples of control groups include subjects without the model disease or condition, subjects without muscle wasting, subjects without muscle weakness, etc. The control criteria may be obtained from multiple subjects without the condition, etc. Differences can be compared between the results of the test subjects and the control group. Some embodiments of the method of the present invention can be used to identify subjects with diseases or conditions associated with frailty.

[0108] The onset, progression, and / or regression of diseases or conditions associated with frailty can also be evaluated and tracked using embodiments of the method of the present invention. For example, in certain embodiments of the method of the present invention, two, three, four, five, six, seven, or more evaluations of a subject are performed at different times using the method of the present invention. Comparison of the results of two or more evaluations performed at different times can show differences in the frailty status (e.g., level of frailty) of the subject. An increase in the determined level and / or characteristics of frailty exhibited by the subject may indicate the onset and / or progression of a disease or condition associated with frailty in the subject. A decrease in the determined level or type of activity may indicate regression in the subject of a disease or condition associated with the evaluated activity. A determination that activity has ceased in a subject may indicate that the disease or condition associated with the evaluated activity has ceased within the subject.

[0109] Certain embodiments of the method of the present invention can be used to evaluate the effectiveness of a therapy for treating a disease or condition associated with frailty. For example, a test subject may be subjected to a candidate therapy and the method of the present invention, which are used to determine whether or not there is a change in the degree of frailty within the subject. A reduction in frailty determined in the subject after administration of the candidate therapy may indicate the effectiveness of the candidate therapy for a disease or condition associated with frailty.

[0110] As described elsewhere in this specification, the visual frailty analysis method of the present invention may be used to assess disease, pathology, or aging in a subject, and may also be used to assess animal models of disease, pathology, or aging. Numerous different animal models relating to disease, pathology, and aging are known in the art, including, but not limited to, numerous mouse models. Subjects to be assessed using the system and / or method of the present invention may be, but are not limited to, animal models relating to disease or pathology, such as models relating to neurodegenerative disorders, neuromuscular disorders, ALS, depression, attention deficit hyperactivity disorder, anxiety disorders, muscle wasting diseases, muscle injuries, developmental disorders, Parkinson's disease, physical injuries, etc.Additional models of diseases and disorders that can be evaluated using the methods and / or systems of the present invention include, for example, Dawson TM et al., Neuron 6 / 10, 66(5):646-61 (2010); Cenci MA & A. Björkland Progre Brain Res. 252:27-59 (2020); Fleming SM et al., NeuroRx 7 / 2(3):495-503 (2005); Farshim P. P, & GP Bates Method Mol. Biol. 1780:97-120 (2018); Nehle RR et al., Mammalian Genome Aug;30(7-8):173-191 (2019); Skoff Rizzo SJ & JN Crowley, Annual Revision, Animal Bioscience 2 / 8, 5:371-389 (2017); Transicova A. et al., Prog Mol Biol Transl Sci. 100:419-82 (2011); Russell VA, *Curr Protoc Neurosci.*, January; Chapter 9: Unit 9.35 (2011); Leo, D. & RR, *Ganetdinov*, *Res. Cells and Tissues*, October; 354(1):259-71 (2013); Campos AC et al., *Psychiatry*, J., 35 Suppl 2:S101-11 (2013); and Szechtman J et al., *Neuroscience*, May revised; 76(Pt B); 254-279 (2017) are known in the art, and their contents are incorporated herein by reference in their entirety.

[0111] In addition to testing subjects with known diseases or disorders, the method of the present invention may also be used to evaluate novel gene variants, such as genetically modified organisms. Thus, the method of the present invention can be used to evaluate genetically modified organisms with respect to one or more characteristics of a disease or pathological condition. In this way, new biological strains, such as new mouse strains, can be evaluated, and the results can be used to determine whether the new strain is an animal model for the disease or disorder.

[0112] Exemplary devices and systems One or more ML models of the automated visual frailty system 100 can take many forms, including neural networks. A neural network may contain several layers, from an input layer to an output layer. Each layer is configured to take a specific type of data as input and to output a different type of data. The output from one layer is introduced as input to the next layer. While the input / output data values ​​in a particular layer are unknown until the neural network actually operates at runtime, the data describing the neural network describes the structure, parameters, and operation of the neural network's layers.

[0113] One or more of the intermediate layers of a neural network may also be known as hidden layers. Each node in a hidden layer is connected to each node in the input layer and to each node in the output layer. If the neural network contains multiple intermediate layers, each node in a hidden layer will be connected to each node in the next higher layer and the next lower layer. Each node in the input layer represents a potential input to the neural network, and each node in the output layer represents a potential output from the neural network. Each connection from one node to another in the next layer may be associated with a weight or score. The neural network may output a single output or a weighted set of possible outputs.

[0114] In one embodiment, a neural network may be constructed with iterative connections such that the output of the network's hidden layers is fed back into the hidden layers for the next set of inputs. Each node in the input layer is connected to each node in the hidden layer. Each node in the hidden layer is connected to each node in the output layer. The output of the hidden layers is fed back into the hidden layers to process the next set of inputs. A neural network incorporating iterative connections may be called an iterative neural network (RNN).

[0115] In some embodiments, the neural network may be a long-short-term memory (LSTM) network. In some embodiments, the LSTM may be a bidirectional LSTM. A bidirectional LSTM performs input from two time directions, one from past states to future states and the other from future states to past states, where past states may correspond to features of video data for a first time frame, and future states may correspond to features of video data for a subsequent second time frame.

[0116] The processing performed by a neural network is determined by the learned weights of each node's input and the network's structure. Given a specific input, the neural network determines the output layer by layer until the output layer of the entire network is computed.

[0117] Connection weights can be initially learned by the neural network during training, associating a given input with a known output. The training data set provides various training examples into the network. In each example, the weights of the correct connections from input to output are typically set to 1, and all connections are weighted to 0. When the training data examples are processed by the neural network, the input may be sent to the network and compared with the associated output to determine how to compare the network's performance to the target performance. Training techniques such as backpropagation may be used to update the neural network's weights and reduce errors that occur when the neural network processes the training data.

[0118] Various machine learning techniques may be used to train and operate a model to perform various steps described herein, such as determining point data, determining elliptic data, determining behavioral data, and determining visual frailty scores. The model may be trained and operated according to various machine learning techniques. Such techniques may include, for example, neural networks (deep neural networks and / or iterative neural networks, etc.), inference engines, trained classifiers, etc. Examples of trained classifiers include support vector machines (SVMs), neural networks, decision trees, AdaBoost ("Adaptive Boost") in combination with decision trees, and random forests. Focusing on SVM as an example, an SVM is a supervised learning model that has an association learning algorithm to analyze data and recognize patterns in the data, and this supervised learning model is commonly used for classification and regression analysis. Given a set of training examples, each marked as belonging to one of two categories, the SVM training algorithm constructs a model that assigns new examples to one category or the other, making it a non-stochastic binary linear classifier. A more complex SVM model may be constructed with a training set that identifies three or more categories, and the SVM determines which category is most similar to the input data. The SVM model may map examples of distinct categories so that they are separated by distinct gaps. New examples are then mapped into the same space and predicted to belong to a category based on which side of the gap they fall on. The classifier may issue a “score” indicating which category the data best matches. The score may provide an index of how closely the data matches a category.

[0119] To apply machine learning techniques, the machine learning process itself needs to be trained. In this case, to train machine learning components such as either the first or second model, it is necessary to establish a "ground truth" regarding the training examples. In machine learning, the term "ground truth" refers to the accuracy of classification of the training set for supervised learning techniques. Various techniques, including backpropagation, statistical learning, supervised learning, semi-supervised learning, stochastic learning, or other known techniques, may be used to train the model.

[0120] Figure 6 is a block diagram conceptually illustrating a device 600 that may be used with the system. Figure 7 is a block diagram conceptually illustrating exemplary components of a remote device such as system(s) 105 that may assist in processing video data, identifying the actions of an object, etc. System(s) 105 may include one or more servers. As used herein, “server” may refer to a conventional server as understood in a server / client computing structure, but may also refer to a number of different computing components that can assist in the operations described herein. For example, a server may include one or more physical computing components (such as a rack server) that are physically and / or via a network connected to other devices / components and capable of performing computing operations. A server may also include one or more virtual machines that emulate a computer system and run on one device or across multiple devices. A server may also include other combinations of hardware, software, firmware, or similar for performing the operations described herein. The server may be configured to operate using one or more of the following computing technologies: client-server model, computer bureau model, grid computing technology, fog computing technology, mainframe technology, utility computing technology, peer-to-peer model, sandbox technology, or other computing technologies.

[0121] Multiple systems 105 may be included in the overall system of this disclosure, such as one or more systems 105 for performing point / body part tracking, one or more systems 105 for ellipse fitting / representation determination, one or more systems 105 for behavioral classification, and one or more systems 150 for determining a visual frailty score. When in operation, each of these systems may include computer-readable instructions and computer-executable instructions present on each device 105, as further described below.

[0122] Each of these devices (600 / 105) may include one or more controllers / processors (604 / 704) which may include a central processing unit (CPU) for processing data and computer-readable instructions, and memory (606 / 706) for storing data and instructions for each device. The memory (606 / 706) may individually include volatile random access memory (RAM), non-volatile read-only memory (ROM), non-volatile magnetoresistive memory (MRAM), and / or other types of memory. Each device (600 / 105) may also include data storage components (608 / 708) for storing data and controller / processor executable instructions. Each data storage component (608 / 708) may individually include one or more non-volatile storage types, such as magnetic storage, optical storage, and solid-state storage. Each device (600 / 105) may also be connected to removable or external non-volatile memory and / or storage (such as removable memory cards, memory key drives, or network storage) via its respective input / output device interface (602 / 702).

[0123] Computer instructions for operating each device (600 / 105) and its various components may be executed by the controller / processor (604 / 704) of each device, using memory (606 / 706) as temporary "working" storage at runtime. Computer instructions for devices may be stored non-temporarily in non-volatile memory (606 / 706), in storage (608 / 708), or in external devices. Alternatively, some or all executable instructions may be embedded in the hardware or firmware on each device, in addition to or instead of software.

[0124] Each device (600 / 105) includes an input / output device interface (602 / 702). Various components may be connected via the input / output device interface (602 / 702), as will be further described below. In addition, each device (600 / 105) may include an address / data bus (624 / 724) for transmitting data between the components of each device. Each component within a device (600 / 105) may also be directly connected to other components, in addition to (or instead of) being connected to other components via the bus (624 / 724).

[0125] Referring to Figure 6, device 600 may include an input / output device interface 602 for connecting to various components, such as an audio output component, including a speaker 612, a wired or wireless headset (not shown), or other components capable of outputting audio. Device 600 may also include a display 616 for displaying content. Device 600 may further include a camera 618.

[0126] Through antenna 614, input / output device interface 602 connects to wireless local area network (WLAN) (WiFi, etc.) radio, Bluetooth, etc. (Registered trademark)The system may be connected to one or more networks 199 via, and / or via a wireless network radio, such as a radio capable of communicating with wireless communication networks, including Long-Term Evolution (LTE) networks, WiMAX networks, 3G networks, 4G networks, and 5G networks. Wired connections, such as Ethernet, may also be supported. The system may be distributed across the network environment via network 199. The I / O device interfaces (602 / 702) may also include communication components that enable the exchange of data between different physical servers or other devices in a set of servers or other components.

[0127] The components of device(s) 600 or system(s) 150 may include their own dedicated processor, memory, and / or storage. Alternatively, one or more components of device(s) 600 or system(s) 105 may utilize the I / O interface(s) (602 / 702), processor(s) (604 / 704), memory(s) (606 / 706), and / or storage(s) (608 / 708) of device(s) 600 or system(s) 105, respectively.

[0128] As described above, multiple devices may be employed within a single system. In such a multi-device system, each device may contain different components for performing different aspects of the system's processing. Multiple devices may contain overlapping components. The components of device 600 and system(s) 105 described herein are illustrative and may be configured as standalone devices or may be included as a whole or in part as components of a larger device or system.

[0129] The concepts disclosed herein may be applied in a number of different devices and computer systems, including, for example, general-purpose computing systems, video / image processing systems, and distributed computing environments.

[0130] The above-described aspects of this disclosure are illustrative. They have been selected to illustrate the principles and uses of this disclosure and are not intended to be exhaustive or limit the disclosure. Many modifications and variations of the disclosed aspects may be apparent to those skilled in the art. Those skilled in the art in the computer and speech processing fields will recognize that the components and process steps described herein may be interchangeable with other components or steps, or with combinations of components or steps, and that the advantages and benefits of this disclosure may still be achieved. Furthermore, it will be apparent to those skilled in the art that this disclosure may be carried out without some or all of the specific details and steps disclosed herein.

[0131] The disclosed system configuration may be implemented as a computer method or as a manufactured article such as a memory device or a non-temporary computer-readable storage medium. The computer-readable storage medium may be computer-readable and may contain instructions for causing a computer or other device to perform the processes described herein. The computer-readable storage medium may be implemented by volatile computer memory, non-volatile computer memory, hard drives, solid-state memory, flash drives, removable disks, and / or other media. In addition, the system components may be implemented as firmware or hardware.

[0132] Examples Example 1. Development of an automated visual frailty indexing system. method mouse We obtained C57BL / 6J mice from the Nathan Shock Center at Jackson Laboratory.

[0133] Open-field assay and frailty indexing The open-field behavioral assay was performed as described above [Kumar V. et al., PNAS 108, 15557-15564 (2011); Goiter B. et al., Communication Biology 2, 124 (2019); Bean G et al., Video-based phenotypic analysis platform for laboratory mice. bioRxiv (2022)]. Mice were shipped from aging colonies from the Nathan Shock Center, located in different rooms within the same animal facility at Jackson Laboratory. Aged mice were acclimated for one week in an animal holding room adjacent to the behavioral testing laboratory. During the daytime of the open-field test, mice were acclimated to the behavioral testing laboratory for 30-45 minutes before the start of the test. A one-hour open-field test was performed as described above. After the open-field test, mice were returned to the Nathan Shock Center and manually frailty indexed. Manual frailty indexing was performed within one week of the open-field assay. The frailty indexing procedure was modified from that of Whitehead et al. [Whitehead JC et al., Journal of Gerontology, Biological Science and Medical Science 69, 621-632 (2014)]. Figure 17 shows the FI test sheet listing all items for manual frailty indexing.

[0134] Video, segmentation, and tracking The open field arena, video equipment, and tracking and segmentation networks were as previously described [Goiter B. et al., Communication Biology 2, 124 (2019); Bean G. et al., Video-Based Phenotypic Analysis Platform for Laboratory Mice. bioRxiv (2022)]. The open field arena was measured at 20.5 inches x 20.5 inches using Sentec (Omron Sentec, Kanagawa, Japan Camera mounted on a 40-inch screen). The camera collected data at 30 frames per second (fps) with a resolution of 640 x 480 pixels (px). A neural network trained to generate elliptic fit of the mouse in each frame, along with mouse tracking, was used to generate a segmentation mask for the mouse.

[0135] Posture estimation and walking As previously mentioned, 12-point 2D pose estimation was generated using a trained deep convolutional neural network [(Shepard K.bioRxiv.doi.org / 10.1101 / 2020.12.29.424780(2020)]. The captured points were the nose, left ear, right ear, base of the neck, left foreleg, right foreleg, mid-spine, left hindlimb, right hindlimb, base of the tail, mid-tail, and tip of the tail. Each point in each frame had an x-coordinate, y-coordinate, and a confidence score. A minimum confidence score of 0.3 was used to determine which points were included in the analysis.

[0136] Gait metrics were generated as described above [Shepard K. et al., bioRxiv.doi.org / 10.1101 / 2020.12.29.424780 (2020)]. The stride period was defined by beginning and ending with left hind limb contact, tracked by posture estimation. These strides were then analyzed for several temporal, spatial, and whole-body coordination characteristics to generate gait metrics across the entire video.

[0137] Open field metrics and feature engineering The Open Field Index was derived from mouse elliptic tracking, as previously described [Goiter B. et al., Communication Biology 2, 124 (2019); Goiter BQ et al., Elife 10 (2021); Bean G et al., Image-Based Phenotypic Analysis Platform for Laboratory Mice. bioRxiv (2022)]. Tracking was used to generate features for spontaneous motor activity and anxiety. Grooming was classified using an action detection network, as previously mentioned. All other genetically engineered features (spinal mobility, anthropometrics, and hindlimb stance) were derived using posture estimation data. Spinal mobility metrics used three points from the posture of the head base (A), the middle of the back (B), and the tail base (C). For each frame, the distance between A and C (dAC), the distance between point B and the midpoint AC of the line (dB), and the angle formed by points A, B, and C (aABC) were measured. The mean, median, maximum, minimum, and standard deviation of dAC, dB, and aABC were calculated across all frames and across non-walking frames (when the animal was not walking). For morphological indicators, the distance between two hind limb points in each frame was measured, along with the mean, median, and standard deviation of that distance across all frames.

[0138] For hind-limb standing, a 4-pixel buffer was added, taking into account the coordinates of the boundary between the arena floor and walls (using OpenCV contours). Each time the mouse's nose point crossed the buffer, this frame was counted as a hind-limb standing frame. Each uninterrupted sequence of frames in which the mouse was hind-limb standing (nose crossing the buffer) was counted as a hind-limb standing behavior. The total number of behaviors, the average length of behaviors, the number of behaviors in the first 5 minutes, and the number of behaviors within 5-10 minutes were calculated.

[0139] modeling The effect of scorekeepers was investigated using a linear mixed model with scorekeepers as a random effect, and the variability of manual FI scores (RLRT=183.85, p<2:2e) was examined. -16) It was found that 42% of could be explained by the score recorder (Fig. 8C). In the restricted likelihood ratio test (RLRT) [Crainiceanu C. M and Ruppert D., Journal of the Royal Statistical Society, Series B (2004)], strong evidence of a score recorder (random) effect with non-zero dispersion is provided. The cumulative link model (log link) [Agresti A., Categorical Data Analysis (2003)] was fitted to the ordinal response (frailty parameter) using body weight, age, and gender as fixed effects and the tester as a random effect. The effect is the estimated variance associated with the random tester effect in the model (Y-axis) across each FI item.

[0140] The tester effect was removed from the FI scores using a linear mixed model (LMM) with the lme4R package [Bates D. et al., Journal of Statistical Software 67, 1 - 48 (2015)]. The following model was fitted. y i j = μ i + ε i j , ε i j ~ N(0, σ 2 ), μ i ~ P ≡ N(0, τ 2 ) where y ij is the jth animal scored by tester i, μ i is the tester-specific mean, ε ij is the animal-specific residual, σ 2 is the within-tester variance, and P is the distribution of tester-specific means. Four testers were used, with different numbers of animals tested by each tester, i.e., i = 1,..., 4. The tester effect estimated by the best linear unbiased predictor (BLUP) using restricted maximum likelihood estimates [Kenward M. G. & Roger J. H., Biometrics 983 - 997 (1997)] was subtracted from the FI scores of the animals.

Number

[0141] Tester adjustment FI score

number

[0142] For FRIGHT modeling to predict elapsed years using manual FI items, single-value Frail parameters were removed to avoid unstable model fits, i.e., zero-variance predictors. An ordinal regression model [McCullough P. Journal of the Royal Statistical Society, Series B (1980)] was fitted without any regularization terms, and a global likelihood ratio test (p<2.2e) was performed. -16We used ) to determine whether the video features provided any evidence that predicted each frailty parameter separately, i.e., evidence of a predictive signal. Next, we used a standard regression model, along with an elastic net penalty [Zou H and Hasty T, Journal of the Royal Statistical Society, Series B (2005)], to predict the frailty parameters using the video features.

[0143] To predict manual FI items, p i The frailty parameters were selected such that <0.80. In the formula, i is the mode of the parameter count distribution. For example, i=1 is the mode of the intimidation response count distribution, and since p1>0.95, the intimidation response is excluded.

[0144] Let X be the covariate vector, C n The training set is 100(1 - α)% out-of-bag prediction interval I α (X, C n The following were obtained via quantile random forests [Meinshausen, N, Journal of Machine Learning Research, 7, 983-999 (2006)] and by the grf package [Atti, S. et al., Analyses of Statistics 47, 1148-1178 (2019)]. The prediction intervals generated using quantile regression forests are often at the nominal level, i.e.,

number

[0145] We selected animals with an inverse relationship between age and FI score, i.e., younger animals with higher FI scores and older animals with lower FI scores. Five test sets were formed, including animals with these criteria, and a random forest (RF) model was trained on the remaining mice. Predictive accuracy was evaluated to predict the FI scores of the five test sets, and the results are shown (Figure 21B). The test sets were ageL age U FI L , and FI U Defined using the following parameters, which represent the FI cutoffs for young and old animals, respectively. For the five test sets, the parameters were set as follows: • ageL = 60, ageU = 90, FI L = 0.20, and FI U = 0.15, • ageL = 60, ageU = 100, FI L = 0.20, and FI U = 0.15 • ageL = 50, ageU = 90, FI L = 0.20, and FI U =0.20, • ageL = 60, ageU = 110, FI L = 0.20, and FI U = 0.20, • ageL = 70, ageU = 100, FI L = 0.25, and FI U = 0.15

[0146] Data and code availability The code and models are available from github.com / KumarLabJax and www.kumarlab.org / data. The Markdown files in the GitHub repository github.com / KumarLabJax / vFI-modeling contain details for reproducing the manuscript results and training the model for vFI / Age prediction. Manual FI scores and vFI features for all mice in the dataset can also be found. Code for genetically engineered features is available at github.com / KumarLabJax / vFI-features.

[0147] result Data collection Figure 8A shows the evaluation of 451 individual C57BL / 6J mice (256 males and 195 females), with 117 mice being repeated in a second test 5 months later, resulting in a dataset of 568 mice ranging in age from 8 to 148 weeks. Top-down video of each mouse in a 1-hour open field was collected according to previously published protocols [Kumar V. et al., PNAS 108, 15557-15564 (2011); Goiter B. et al., Communication Biology 2, 124 (2019)] as described in the methods described above (Figure 8A). After the 1-hour open field, each mouse was frail-indexed to a standard mouse by a skilled specialist at the Nathan Shock Center for Aging in order to assign a manual FI score. Manual FI was performed by four different score recorders during the data collection process. Open-field video footage was processed by tracking and pose estimation networks to generate tracks, elliptic fit, and 12-point poses of the mouse for each frame [Goiter B. et al., Communication Biology 2, 124 (2019); Shepard K. et al., bioRxiv.doi.org / 10.1101 / 2020.12.29.424780 (2020)]. Using these frame-by-frame measurements, we calculated various video-specific features, including conventional open-field indices such as anxiety and hyperactivity (all extracted video-specific features are listed and defined in Table 1 along with their measurement sources) [Goiter B. et al., Communication Biology 2,124 (2019)], neural network-based grooming [Goiter B. et al., bioRxiv doi.org / 10.1101 / 2020.10.08.331017 (2020)], and a novel gait index [Shepard K. et al., [g19]bioRxiv[ / g19]doi.org / 10.1101 / 2020.12.29.424780 (2020)]. For each mouse, we calculated video-specific features using penaltyized linear regression (LR). *The following machine learning models were used as array features, including [Zou H. and Hasti T., Journal of the Royal Statistical Society Series B, Statistical Methodology 67, 301-320 (2005)], Random Forest (RF) [Brayman L., Machine Learning 45, 5-32 (2001)], Support Vector Machine (SVM) [Cortez C. and Vapnik V., Machine Learning 20, 273-297 (1995)], and Extreme Gradient Boost (XGB) [Friedman JH, Statistical Yearbook 1189-1232 (2001)]. Manual FI scores were used as response variables for the models. As expected, mean FI scores increased with increasing age (Figure 8B). Heterogeneity of FI scores (indicated by standard deviation bars) also increased with age. The data obtained in this study showed that the maximum less than limit of the FI score was slightly below 0.5, which falls within the range of the maximum less than limit previously shown in mice [Whitehead J.C. et al., Journal of Gerontology, Biological Science and Medical Science 69, 621-632 (2014); Lockwood K. et al., Scientific Reports 7, 43068 (2017)]. These results indicate that the FI data obtained in this study are typical of other mouse data and reflect the characteristics of human FI, with increasing mean FI scores and heterogeneity of FI scores with age [Lockwood K. et al., Scientific Reports 7, 43068 (2017)]. Visual inspection by score recorders suggested that there may have been a score recorder-dependent effect on manual FI. For example, score recorder 1 and score recorder 2 tended to produce higher and lower frailty scores, respectively. The effect of score recorders was investigated using a linear mixed model with score recorders as a random effect, and the 35% variability of the dataset (RLRT=66.41, p<2.2e) was used. -16) was found to be explainable by the score recorder (Figure 8C). The restricted likelihood ratio test (RLRT) [Crayniceane C.M. and Rupert D., Journal of the Royal Statistical Society, Series B Statistical Methodology 66, 165-185 (2004)] provides strong evidence for a score recorder (random) effect with non-zero variance, suggesting that variability among score recorders is a significant source of variation in the data and should be adjusted for before modeling.

[0148] The overall approach is illustrated in Figure 8A. The study was conducted over three rounds using 533 unique mice, resulting in 643 data points (371 males and 272 females). The first round of the study (Batch 1) involved 222 mice (141 males and 81 females). The second round of the study (Batch 2) occurred approximately five months later and involved 319 mice (173 males and 146 females). Of these mice, 105 were repeated from the first batch. The third round of the study (Batch 3) occurred approximately one year later and involved 102 mice (57 males and 45 females). Of these mice, 18 had been previously tested in the first round and 15 had been tested in the second round. Top-down video of each mouse during a one-hour open-field session was collected according to a previously published protocol [Kumar V. et al., PNAS 108, 15557-15564 (2011); Goiter B. et al., Communication Biology 2, 124 (2019)] (see Methods and Figure 8A for examples of young and aged mice). After the open-field session, each mouse was assigned a manual FI score by a skilled expert at the Nathan Shock Center on aging, using a standard mouse frailty index [Skov Rizzo SJ et al., Modern Protocols in Mouse Biology (2018)] (Figure 19A). Bimodality of data (Hartigans Test [Hartigan J. and Hartigan PM, Annals of Statistics (1985)], D=0.07; p<2.2e) -16Despite this, Simpson's paradox [Simpson E.H., Journal of the Royal Statistical Society, Series B (1951)] was found not to appear in any of the top 15 features in the data (Figure 20). Open-field images were processed by tracking and posture estimation networks to generate mouse tracking, elliptic fit, and 12-point posture for each frame [Goiter B. et al., Communication Biology 2, 124 (2019); Shepard K. et al., Cell Reports (2022)]. Using these frame-by-frame measurements, a variety of features for each image were calculated, including conventional open-field indices for anxiety and hyperactivity [Goiter B. et al., Communication Biology 2, 124 (2019)], grooming [Goiter B.Q. et al., Elife (2021)], gait and posture indices [Shepard K. et al., Cell Reports (2022)], and genetically engineered features. Features used to train machine learning models to predict chronological age, biological age, and visual FI (vFI) are described herein.

[0149] Consistent with previous data, the mean FI score increased with age in the dataset (Figure 8B). Heterogeneity of FI scores (indicated by standard deviation bars) also increased with age. The submaximal limit of FI scores was found to be slightly below 0.5 for data that fell within the range of the submaximal limit shown in mice [Whitehead J.C. et al., Journal of Gerontology, Series A (2014); Lockwood K. et al., Scientific Reports (2017)]. These results indicate that the FI data are typical of other mouse data, reflect the characteristics of human FI, and that the increase in mean FI score and heterogeneity of FI scores increases with age [Lockwood K. et al., Scientific Reports (2017)]. During the data collection process, four different scorekeepers performed manual FI. Visual inspection of the data revealed scorekeeper effects on manual FI scores (Figure 8C). For example, scorekeeper 1 and score 2 tended to produce high and low frailty scores, respectively (Figure 18B). The modeling showed that 42% of the variability in manual FI scores was attributable to the scorer effect (RLRT=183:85, p<2.2e). -16 Further examination revealed that hair bristles, kyphosis, and vision were the most subjective factors influencing which FI items were most affected by the scorer (Figure 18A). This analysis suggests that the scorer effect is a significant source of variation in mouse clinical FI.

[0150] Feature extraction Features for each video were extracted using frame-by-frame segmentation, elliptic fitting, and 12-point pose coordinates. The extracted features, including descriptions and sources of measurements, are listed in Table 1. Overall, there was a very high correlation between the median and mean video metrics (Figures 9A-B). Only the median was used in the modeling for two reasons: the median tends to correlate more strongly with FI scores than the mean, and the median is more stable than the mean in terms of outlier effects. Similarly, because the interquartile range tends to be more stable with outliers than the standard deviation, the interquartile range was used as a feature of the model, rather than the standard deviation, where available. This generated a total of 44 video features (Figure 14, Table 1). Metrics obtained from standard open-field assays, such as total spontaneous motor activity, time spent in peripheral paired centers, and grooming behavior (Figure 9A), were considered. Standard open-field indices showed low correlations with both FI scores and age (Figures 14 and 15).

[0151] In addition to existing features, a set of features hypothesized to correlate with FI was designed. These features included morphometric features that capture the shape and size of the animal, as well as behavioral features related to flexibility and vertical movement. Age-related changes in body composition and fat distribution have been observed in humans and rodents [Papas L. and Nagy T., European Journal of Clinical Nutrition 73 (October 2018)]. It was hypothesized that body composition measurements could indicate some signal of aging and frailty. The major and minor axes of an ellipse fitted to the mouse in each frame were used as the estimated length and width of the mouse, respectively (Figure 10B). The distance between the hindlimb coordinates in each frame was taken as another width measurement closer to the hip. The mean and median of the ellipse width, ellipse length, and hindlimb width across all frames were used as per-frame metrics. Many of these morphological features showed high correlations with FI scores and age (Figures 14 and 15), for example, the median hind limb width had correlations of r = 0.56 and 0.57, respectively (Figure 10C).

[0152] Changes in gait have been shown to be a characteristic of aging in humans [Zou, Y. et al., Scientific Reports 10, 4426 (2020); Skiadopulos A. et al., J. Neuroeng Rehabil. 17, 41 (2020)] and mice [Tarantini S. et al., Journal of Gerontology, Biological Science and Medical Science 74, 1417-1421 (2018); Baer W.-N. et al., Journal of Gerontology, Biological Science and Medical Science 74, 1413-1416 (2019)]. To explore age-related gait changes in our current mouse cohort, we performed an analysis similar to the method used to extract gait indices from mice moving freely in the field (Figures 10D-E) [Shepard K. et al., bioRxiv. doi.org / 10.1101 / 2020.12.29.424780 (2020)]. Each stride was analyzed for its spatial, temporal, and whole-body coordination indices (Figure 10D), resulting in an array of measurements from which the median of all strides for each mouse was taken. Intra-mouse heterogeneity of gait characteristics was also examined using standard deviation and interquartile ranges across all strides for each mouse. Many of these calculated metrics showed high correlation with FI scores and age (Figures 14 and 15), for example, the median step width and the interquartile range of aptero-tail lateral displacement (r=0.58 and r=0.63, respectively) (Figure 10E).

[0153] Next, we investigated spinal flexion throughout the entire video. We hypothesized that older mice would flex their spines less, or less frequently, due to decreased flexibility or spinal mobility. This change in flexibility could be captured by the estimated pose coordinates of three points on the mouse in each video frame: posterior to the head (A), mid-back (B), and base of the tail (C). For each frame, we calculated the distance between points A and C normalized to mouse length (dAC), the orthogonal distance (dB) from the line to the midpoint of posterior B, and the angle between the three points (aABC) (Figure 10F). For each of the three frame-by-frame indices (dAC, dB, and aABC), we calculated the mean, median, standard deviation, minimum, and maximum for each video, for all frames and non-walking frames (frames in which the mouse was not in a striding state). A moderately high correlation was observed between spinal flexion and the FI score, contradicting our hypothesis (Figures 14 and 15). In other words, while median dB and median dBA (for non-walking frames) were expected to decrease with age, they were found to increase instead (r=0.51 and 0.35, respectively) (Figure 10G). One possible reason for this result is that very frail mice spent more time grooming. However, neither grooming behavior nor grooming time in seconds showed any correlation with FI score or median dB. Another possibility is that highly frail mice spent less time walking and more time walking, as there was little correlation between step count or distance traveled and FI score or median dB. Since median dB has a correlation of 0.496 with body weight, highly frail mice may also have had higher median dB due to their body composition. It is also important to note that these bending metrics cast a broad net; that is, they provide a cheap and general explanation for all spinal activity during one hour in open field. Therefore, these indices may have captured the interaction between body composition and behavior.

[0154] Previous spinal flexibility indices examined lateral spinal flexibility, but vertical flexibility may also be related to frailty. To investigate this, we examined the occurrence of wall-supported hindlimb standing (Figure 10H). We hypothesized that more frail mice might have lower feeding capacity due to decreased lateral spinal mobility and / or reduced exploratory activity. We took the edges of an open field and added a 5-pixel buffer as a boundary. The frame in which the coordinates of the mouse's nose intersected this boundary was used as an instance of hindlimb standing. From these heuristics, we determined the number of hindlimb standings and the average length of each hindlimb standing behavior (Table 1). Several metrics related to hindlimb standing behavior indicate signals of frailty, particularly the total number of hindlimb standings and hindlimb standings during the first 5 minutes (r=0.2 and 0.3, respectively, Figure 10I).

[0155] Interestingly, the correlation with age was generally slightly higher than that with the FI score (Figures 14 and 15). This may be due to the different ways in which mice become frail. Some mice may be highly frail but not functionally impaired in stride length, while older mice, on average, may show greater changes in stride length regardless of their degree of frailty. Even more noteworthy is the increased heterogeneity in many of these indices, which have both age scores and FI scores (e.g., median stride length, median step length, median dB).

[0156] sex To analyze sex differences in frailty, FI score data were stratified into four age groups, and box plots were compared between males and females for each age group (Figure 11A). The oldest group contained only 9 females compared to 81 males. The range of frailty scores for females in each age group tended to be slightly lower than that of males, except in the oldest group. The two moderate groups showed very significant differences in distribution between males and females.

[0157] A comparison of the correlation between male and female FI item scores and age (Figure 11B) showed generally high correlations (r=0.85). The mean difference in correlation between males and females between FI index items and age was 0.08, but there were several index items that showed significant differences. Alopecia and intimidation responses showed the highest sex differences in correlation with age (0.29 and 0.21, respectively), with females showing a high correlation to alopecia and males showing a high correlation to intimidation responses (Figure 16).

[0158] High correlations were observed between male and female video features with both FI score and age (r=0.88 and r=0.90, respectively), and there was a mean difference of 0.14 and 0.13 between the correlation of male and female video metrics with FI score and age, respectively (Figures 14 and 15). For both FI score and age, the video features with the highest sex differences were gait indices related to stride and step length, lateral displacement of the tail base, and lateral displacement of the tail tip. The highest sex difference was the correlation between median lateral displacement of the tail base relative to age (difference of 0.57) and median lateral displacement of the tail tip relative to age (0.50), with females tending to have higher correlations with both FI score and age. For metrics related to stride length and step length (median stride length, difference of 0.33), males had higher correlations with FI score and age than females. These results indicate that, with age, females significantly increase the lateral displacement of the tail base and tail tip during walking, while males show little change in this feature, suggesting that males experience a greater decrease in stride length with age compared to females.

[0159] Prediction of age and frailty index from video data Once the video features described herein were confirmed to correlate with aging and frailty, these features were used as covariates in models vFRIGHT and vFI, respectively, to predict age and manual FI score (Figure 12A). Age is a truth based on empirical rules and is strongly associated with frailty. The age prediction using video features (Figure 12A, model vFRIGHT) was compared to the age prediction using a manual FI item, i.e., a method called the FRIGHT age clock [Schultz MB et al., Nature Communications (2020)] (Figure 12A, model FRIGHT). Four models were initially tested: Penalized Linear Regression (LR*) [Zou H. and Hasti T., Journal of the Royal Statistical Society, Series B (2005)], Support Vector Machine (SVM) [Cortez C., Vapnik V., Machine Learning (1995)], Random Forest (RF) [Bryman I., Machine Learning (2001)], and Extreme Gradient Boost (XGB) [Friedman J. H., Analyses of Statistics (2001)] (Figure 12B, Panel 1). The Random Forest regression model showed the lowest mean absolute error (MAE) (p<2.2e) when compared using the iterative index ANOVA. -16 F 3,147 =190.43), Root Mean Square Error (RMSE)(p<2.2e -16 F 3,147 =59.53), and the highest R² (p<2.2e -16 F 3,147 Because it exhibits superior performance compared to other models with an FI item of 58.14, it was selected to predict age for unseen future data (Figure 12B, Panel 1, Figure 19C). The vFRIGHT model was able to predict age more accurately and precisely than the FRIGHT clock. vFRIGHT performed better with a lower MAE (13.1 ± 0.99 weeks) compared to the FRIGHT clock using an FI item (15.7 ± 4 weeks) (p<4.7e- 5 F 1,49The RMSE (RMSEvFRIGHT = 19.9, using the repeated index ANOVA) was present (Figure 12B, Panel 2). RMSEvFRIGHT = 17.97±1.44, RMSEFRIGHT = 20.62±4.78, p<6.1e -7 F 1,49 =32.84) and R2(RMSEvFRIGHT=0.78±0.04, RMSEFRIGHT=0.76±0.07, p<2.1e -8 F 1,49 Comparing the two (=44.54), we found a similar significant improvement in age prediction when using video features (Figure 19D). The variance of prediction error was significantly reduced for video-based age prediction (vFRIGHT) compared to manual FI item-based age prediction (FRIGHT) (Figure 12B, Panel 2). Predicted vs. actual values ​​were plotted for the training and test sets of the vFRIGHT model (Figure 12G) and the FRIGHT model (Figure 19G). Taken together, these results indicate that automated video contains more accurate information about aging than is handled by manual FI items. Video features also sometimes provide aging information that overlaps with health impairments scored by manual FI.

[0160] To address this, we predicted individual FI items that used video features (Figure 12A). Of the 27 items, many had scores ranging from zero to near zero, indicating that, in a genetically homogeneous dataset, at least the majority of the information in manual FI comes from a subset of the exponential items (Figure 19F). For prediction, only exponential items with balanced ratios of scores between 0 and 0.5 and 1 were selected (Figure 12C). We then built a classifier for each of the nine exponential items to predict the scores given video features of the mouse. The scores for individual FI items were predicted using a standard elastic network regression model. For all nine, the scores predicted with accuracy exceeding the values ​​expected by random estimation (Figure 12C, dotted line represents estimation accuracy). Many of these FI items have implicit relationships with video features such as grooming (e.g., coat condition, alopecia), gait / mobility (e.g., gait disturbance, kyphosis), and body composition (e.g., abdominal distension, body condition). In the FRIGHT model, gait disturbance, kyphosis, and hair bristles contributed most to age prediction in the dataset, followed by abdominal swelling and physical condition (Figure 19B)—all items for which video features could predict the score (Figure 12C). Together, these results indicate that most of the information about aging and frailty can be obtained from a small subset of manual FI items, and that information within this subset can be predicted using video data. Furthermore, since aging was predicted more accurately and precisely with video data than with manual FI items, video data may also contain additional signals for aging.

[0161] Next, the goal of vFI (Figure 4A, Model vFI): to predict manual FI scores using video data. Similar to vFRIGHT modeling, the random forest regression model aims for the lowest mean absolute error (MAE) (p<2.1e -15 F 3,147 =30.53), root mean square error (RMSE)(p<8:3e- 14 F 3,147 =26.62), and the highest R² (p<4.7e -14 F 3,147The model predicted FI scores for unseen future data better than all other models with a score of 27.2 (Figures 12D, 19E). The model was able to predict FI scores within 0.04 ± 0.002 of the actual FI scores (for datasets where a range of 0.04 to 0.47 was found, the FI scores had a possible range of 0 to 1). This error is such that one item of the FI was incorrectly scored at 1 point, or two items at 0.5 points, demonstrating the stability of the model. Residuals were plotted for the training and test sets of the model (Figures 12F, 12G, and 19G). The residuals calculated from the training data show that their distribution is nearly symmetrical to zero for both models, with most residuals around the black diagonal. The residuals for the test set follow a similar pattern. It was concluded that the video-generating features described herein can be successfully used for automated frailty scoring. Age correlates more highly with manual FI scores (r = 0.81) than any other video feature. Therefore, using a model that includes age only as a feature yields higher predictive accuracy (Figure 21A). Models that use both video features and age (AllRF) perform particularly well than models that use age only, indicating that video features provide important information about frailty (Figure 21A). In particular, when considering mice whose FI scores deviate from the age group, i.e., young mice with high frailty and older mice with low frailty, the vFI model (VideoRF) performs significantly better than models that use age only, and even better than models that use video features + age (AllRF) (Figure 21B). This indicates that for mice that are outliers in the age group, video features provide better information than age.

[0162] Finally, to see how much training data is actually needed for high-performance predictions using vFI and vFRIGHT, we conducted simulation tests with different percentages of the total data allocated to training (Figure 21E). We found that training sets of <80% of the current dataset achieved similar performance, but reductions in training set size below this level generally showed a downward trend in performance. Since open-field tests are sometimes conducted in less than an hour, we then investigated the decrease in accuracy for vFI predictions using shorter videos by truncating the video to the first 5 minutes and the first 20 minutes (Figure 21D). The features associated with the 60-minute video were: LMM with random effects for best accuracy "simulation" for vFI predictions, MAE for worst accuracy, and F 2,98 = 178.39, p < 2.2e -16 , lowest RMSE, F 2,98 = 156.93, p < 2.2e -16 ), that is, the maximum R² (p < 2.2e -16 F 2,98 The results showed a score of 297.3. A significant decrease in performance accuracy was observed when the length of the open-field trial was shortened from 60 minutes to 20 minutes of video (LMM-MAE with post-hoc pairwise comparisons, t 98 =14.82, FDR adjusted p<0:0001, RMSE, t 98 =13.69, FDR adjusted p<0:0001, R 2 , t 98 =-19:22, FDR adjustment p<0:0001). Based on the experiment, it was concluded that 60 minutes of video generation features provide the most accurate vFI prediction, but even with 80% of the video, there is no substantial loss of prediction accuracy.

[0163] Quantifying the uncertainty in frailty index predictions. In addition to quantifying average accuracy, errors were also investigated in more detail within the dataset. Prediction errors were quantified by providing prediction intervals (PIs) that yield a range of values, including FI scores with unknown ages and specified levels of confidence, based on the same data that provided the random forest point predictions [Zang H. et al., Am. Statistics. 74, 392-406 (2020)]. One approach to obtaining random forest-based prediction intervals was to use a generalized random forest to model the conditional distribution of the featured FI, as previously mentioned [Meinshausen N., Journal of Machine Learning Research, 7, 983-999 (2006); Attai S. et al., Analyses of Statistics, 47, 1148-1178 (2019)]. For the animals in the test set, a quantile-based generalized random forest was used to provide point predictions (age response) and prediction intervals for the FI scores. This yielded FI (age response) values ​​within a range that included an unknown FI score (respiration) with 95% confidence (Figure 12J-I). The mean PI width of the predicted FI score for all test animals was 5.72 ± 1.49 (80.29 ± 16.8 for predicted response age), and the PI length ranged from 2.3 to 8.5 (28 to 114 times for age), highlighting that the PI width was animal and age-group specific. Plotting regression fits with the PI width smoothed against age showed that the width increased with the age of the animals (Figure 12G-H). The variability of the 95% PI width shown in Figures 124G-124H (right panel) indicated that animals belonging to the middle-aged group showed higher variability (M, pink). Beyond simple point predictions, providing prediction intervals (PIs) for the frailty index quantifies the uncertainty of predictions, allowing FI scores and ages to be pinpointed with greater precision for some animals than others.

[0164] Importance of characteristics of frail and healthy animals A useful visual flail (vFI) should rely on several features that capture the animal's intrinsic flailness and are simultaneously interpretable. Two approaches were used to identify features important for making vFI predictions using a trained random forest model, (1) feature importance, and (2) feature interaction strength. Feature importance is a measure of how often the random forest model uses a feature at different depths of the forest. A higher importance value indicates that the feature is at the top of the forest and therefore important for building a predictive model. In the second approach, a total interaction measure was derived, indicating how much a feature interacted with all other model features.

[0165] Comparing the importance of features in the vFI and vFRIGHT models (Figure 11A), we find that many of the most important image features for each model are common, but there are some significant differences (Figure 21C). For example, step width IQR is far more important for vFI than for vFRIGHT, and apterocaudal lateral displacement (LD) IQR is far more important for vFRIGHT than for vFI. By modeling three different quantiles of the conditional distribution of FI scores, we were able to gain a more complete understanding of feature importance. The three quantile angles represent three frailty groups: low-frail (Q1), moderate-frail (M), and high-frail (Q3) mice. We hypothesized that different sets of features are important for mice belonging to different frailty groups. Indeed, step length 1IQR was important for mice belonging to both the Q1 and Q3 quantiles (Figure 13A). Furthermore, features such as length, hindlimb velocity, dAC / dB (excluding walking), and step width were important for mice with a lower degree of frailty, while step length dB and posterior number were more important for animals with a higher degree of frailty. Similarly, step width, tail tip LD, and width were important for mice with an FI score close to M.

[0166] For the feature interaction strength approach, H-statistics [Friedman JH et al., Annals of Applied Statistics 2,916-954 (2008)] was used as an interaction metric, measuring the rate of variation in predictions explained by feature interactions after considering individual features. For example, 15% of the variability in predictive function was explained by interactions between the tailtip LD and other features after considering the individual contributions of the tailtip LD and other features. Approximately 13% and 8% of the variability in predictive function were explained by interactions between width (each step length) and other features. For a deeper analysis, all bidirectional interactions between the tailtip LD and other features were examined (results not shown). Strong LD interactions were found between animal width, stride length, hind limb, and dB and the tailtip.

[0167] Both the importance of features and the strength of feature interactions indicated that trained random forests for vFI depended on several features and their interactions. However, they did not show how vFI depended on these features or how the interactions appeared. We used accumulated local effects (ALE) plots [Applie DW and Tue J. Journal of the Royal Statistical Society Series B, Statistical Methodology 82, 1059-1086 (2020)] to describe how features affected the random forest model's vFI predictions on average. For example, increased lateral displacement of the tail tip had a positive impact (increase) on the predicted FI scores of animals in the intermediate and high-frail groups (Figure 13B). Similarly, increased hindlimb measurements had a positive impact on predictions. This impact was most evident in animals in the high-frail group. Animals with larger widths had a positive impact on predictions; that is, larger step widths and dB values ​​had a positive impact on model predictions. Therefore, the ALE plots for key features provided a clear interpretation consistent with the initial hypothesis. ALE secondary interaction effect plots were explored for the predictors of step length-step width (Figure 13D) and length-width (Figure 13E). This revealed additional interaction effects between the two features, excluding the minor effects of the major features. Figure 13D clearly shows the interaction between step width and step length; that is, larger step width and step length increased the predicted FI score. Similarly, larger width (36–44 cm) and length (52–60 cm) had a positive impact on the mean FI score prediction.

[0168] In summary, the usefulness of vFI was established by demonstrating dependence on several features through the importance and interaction of a few features. Next, ALE plots were used to understand the influence of features on model predictions. This helped to relate the predictions of the black-box model to some of the image-generating features. Opening up the black-box model was an essential final step in the modeling framework.

[0169] Consideration Mouse frailty index (FI) is a valuable tool in the study of biological aging. The study described herein attempts to extend this by generating an automated visual frailty index (vFI) using video-generated features to model the FI score. This vFI provided a reliable, high-throughput method for studying aging. One of the largest frailty datasets for mice was generated from associated open-field video data. Using computer vision techniques, behavioral and morphological features were extracted, many of which showed a strong correlation with aging and frailty. Sex-specific aging in mice was also analyzed. A machine learning classifier capable of accurately predicting frailty from video features was then trained. Through modeling, insights into the importance of features across age and frailty status were also gained.

[0170] Data were collected at the National Center for Aging, using a design similar to that of high-throughput intervention studies that could be conducted over several years. Mice were tested by available, skilled scorekeepers. Four different scorekeepers were used to test different batches of mice for FI testing. Furthermore, personnel were rotated between batches. These conditions may provide a more realistic example of inter-laboratory conditions that are difficult to discuss and fine-tune. It was found that 42% of the variability in the dataset could be explained by the scorekeeper, indicating the presence of a tester effect. This variability affected some items, such as hair bristles, more than others. Previous studies examining the tester effect have generally found good to high inter-tester mutual reliability, although FI items with low mutual reliability required discussion and fine-tuning for improvement [Kane AE, Ayaz O., Gimille, A., Feridoni HA and Howlett SE, Canadian Journal of Physiology and Physiology (2017)].

[0171] Top-down video of mice in an open field was processed by a previously trained neural network to generate elliptic fit and segmentation of the mice, as well as pose estimates of 12 prominent points on the mouse for each frame. These frame-by-frame metrics were used to design features for use in the model. The first category of features consisted of standard open field metrics such as time spent periphery versus center, total distance traveled, and the number of grooming behaviors. These standard open field metrics showed low correlation with both FI scores and age. These results suggest that standard open field assays are insufficient for studying aging.

[0172] In humans, age-related changes in body composition and indicators of human body dimensions such as the waist-to-hip ratio are predictors of health status and mortality risk [Mizrahi Lehrer E., Cepeda Valery B., and Romero Corral, Handbook of Anthropometry, Physical Indicators of Human Form in Health and Disease (2021); Papas L. and Tim R., N., European Journal of Clinical Nutrition (2019); Gervais M., Metz L., Lingott E., and Courtex D., Lipids in Health and Disease (2010)]. The effects of aging on body composition in rodent models are not well established, but changes in body composition similar to those in humans have been observed [Mizrahi-Lehrer E., Cepeda-Valery B., and Romero-Coral, Handbook of Anthropometry, Physical Indicators of Human Form in Health and Disease (2021); Gervais M., Metz L., Lingott E., and Courtex D., Lipids in Health and Disease (2010)]. High correlations have been observed between morphological features and both FI score and age, particularly medial width and medial hindlimb width.

[0173] The prevalence of gait disorders increases with age [Zou et al., Scientific Reports (2020)]. Elderly patients have been shown to have gait irregularities. For example, older adults showed increased variability in step length [Tarantini S. et al., Journal of Gerontology: Series A (2018)]. Spatial, temporal, and postural features of gait in each mouse were examined, and many features were found to be strongly correlated with both frailty and age. Similar to human data, a decrease in stride velocity and increased variability in step length were observed with age [Tarantini S. et al., Journal of Gerontology: Series A (2018)]. Because gait is thought to have both cognitive and musculoskeletal components, it is a compelling area for frailty research.

[0174] Spinal mobility in humans is a predictor of quality of life in aging populations, and mice are used as a model of the aging human spine. Surprisingly, some spinal flexion metrics showed a moderately high correlation with FI scores, but the relationship was the opposite of what was initially hypothesized. These metrics are general descriptions of all spinal activity during the experiment and therefore likely capture a combination of behavior and body composition that led to the observed results. Nevertheless, some of these metrics showed a moderately high correlation with FI scores and age and were considered important features in the model.

[0175] Many age-related biochemical and physiological changes are known to be sex-specific. Understanding sex differences in the presentation and progression of frailty in mice is important for translating preclinical outcomes for clinical use. It is interesting to understand how sexual characteristics, such as hormone and body fat distribution, relate to biological aging. In humans, there is a known “mortality and morbidity paradox” or “sex and frailty paradox,” in which females tend to be more frail but paradoxically survive longer. However, in C57BL / 6J mice, males appear to tend to survive slightly longer than females, but with variability, and females do not appear to survive paradoxically longer at the time of frailty. In the study described herein, more males survived to older ages than females, and furthermore, females tended to have a slightly lower distribution of frailty than males of the same age group. These results suggest that the “sex and frailty paradox” shown in humans may not exist or may be reversed in mice. Comparing the correlation of FI index items with age between males and females, some sex differences were observed in the strength of correlation for some index items, mainly those related to visual fur changes. Comparing the correlation of visual features with age and FI scores between males and females also revealed many orders-of-magnitude correlations. The median lateral displacement of the tail base and median lateral displacement of the tail tip both correlated much more strongly with age in females than in males. Lateral tail displacement within the stride tended to increase as female mice aged, while males showed almost no change. Conversely, males showed a significant decrease in stride length and a significant increase in step length with age, while females showed little change. Most of the visual features with large differences were gait-related, and some were related to spinal flexion. These age differences in gait were novel insights. Understanding the differences between human and mouse frailty is important for critically evaluating how results from mouse studies can be translated to humans.

[0176] Manual FI assesses a broader range of bodily systems than vFI. However, the complex behaviors measured and described herein contain a great deal of implicit information about bodily systems. In the isogenic dataset, most of the information from manual FI is derived from a limited subset of the exponential items. Of the 27 manual FI items scored, 18 items showed little to no variation in score in our dataset (almost all mice had the same score, i.e., 0), and only 9 items had a balanced score distribution. Video features can accurately predict these 9 FI items. Models using video features also predicted age with much less variability and greater accuracy than models using manual FI items (FRIGHT vs vFRIGHT). This suggests that the video features described herein not only can predict the relevant FI items but also contain signals about aging beyond conventional manual FI. Furthermore, more detailed feature measurements compared to FI items (using actual values ​​rather than simplified scores of 0, 0.5, or 1) may contribute to higher performance.

[0177] Finally, using the video features as input to a random forest model, we predicted manual FI scores to within 0.04 ± 0.002 of the actual scores on average. Although not normalized, this error was 1.08 ± 0.05, which is comparable to one FI item being mis-scored by only 1 point, or two FI items being mis-scored by only 0.5 points. Furthermore, we determined simple point predictions by providing a 95% prediction interval. Applying quantile random forests to the low and high quantiles of the conditional distribution of FI scores revealed how specific features affected frail and healthy animals differently.

[0178] The ease of use of machine learning models in non-computerized laboratories is a critical challenge. Therefore, in addition to the implementation details in the Methods section, an integrated mouse phenotypic analysis platform—that is, a hardware and software solution—providing tracking, pose estimation, feature generation, and automated behavioral analysis is detailed in [Bean G. et al., bioRxiv (2022)]. While this platform requires specific open-field equipment, researchers can use machine learning models if they generate the same features as the models described herein using their own open-field data acquisition equipment. With a setup that enables tracking and pose estimation using available software, researchers can compute the features necessary to use the machine learning models.

[0179] vFI can be further improved by adding new features through reanalysis of existing data and future technological improvements to data acquisition [Pereira TD, Schewitz JW, and Marcy M. Nature Neuroscience (2020); Matthias A. Neuron (2020)]. For example, quantification of defecation and urination can provide information on additional systems, while higher camera quality can provide more detailed information on subtle motor-based behaviors and appearance-based features such as coat condition. Furthermore, this approach may be used in long-term domestic cage environments. This not only further reduces handling and environmental factors but may also integrate features such as social interaction, feeding, drinking, and sleep. In addition, given the strong evidence of a genetic component to aging [Sin PP, Demitt BA, Nas RD, and Brune A., Cell (2019)], applying this method to other strains and genetically heterogeneous populations such as diversity outcross and collaborative cross may reveal how genetic variation affects frailty. Furthermore, since predicting mortality risk is a key function of frailty, lifespan can be studied using video features. The value of this research may extend beyond community adoption to community engagement. Training with data from multiple labs can provide a more stable and accurate model. This could provide a consistent FI across multiple studies. Overall, this approach yields new insights into mouse frailty and demonstrates that abstract concepts such as frailty can be quantified using video data of mouse behavior. Automated frailty indices enable high-throughput and reliable aging research, particularly intervention studies, which are a priority for the aging research community.

[0180] Equivalents While several embodiments of the present invention are described and illustrated herein, those skilled in the art will readily conceive of various other means and / or structures for carrying out the function and / or obtaining the results and / or one or more advantages described herein, and each of such variations and / or modifications will be considered within the scope of the present invention. More generally, those skilled in the art will readily understand that all parameters, dimensions, materials and configurations described herein are illustrative, and that the actual parameters, dimensions, materials and / or configurations will depend on the specific one or more applications in which the teachings of the present invention are used. Those skilled in the art will understand many equivalents to the specific embodiments of the present invention described herein, or will be able to verify such equivalents using only routine experiments. Therefore, it will be understood that the embodiments described herein are presented only illustratively, and that the present invention may be carried out in ways other than those specifically described and described in the claims, within the scope of the appended claims and its equivalents. The present invention covers the individual features, systems, articles, materials and / or methods described herein. In addition, any combination of two or more such features, systems, articles, materials, and / or methods is included within the scope of the invention, provided that they are not mutually inconsistent. It will be understood that all definitions defined and used herein govern dictionary definitions, definitions in documents referenced by reference, and / or the common meanings of the defined terms.

[0181] As used herein in this specification and in the claims, the indefinite articles "A" and "An" will be understood to mean "at least one" unless explicitly stated to the contrary. As used herein in this specification and in the claims, the phrase "and / or" will be understood to mean "either or both" of the elements thus joined, that is, "either or both" of elements that exist as a combination in some cases and separately in other cases. Unless explicitly stated otherwise, other elements may exist, at their discretion, in addition to the elements specifically identified by the phrase "and / or," whether related to or unrelated to the elements specifically identified.

[0182] Conditional language used herein, in particular “can,” “could,” “might,” “may,” “eg,” and similar terms, is generally intended to convey that a particular embodiment includes certain features, elements, and / or steps, while other embodiments do not, unless otherwise specifically stated or understood in the context in which they are used. Therefore, such conditional language is not generally intended to mean that features, elements, and / or steps are required in any way with respect to one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps are included in or performed within any particular embodiment, with or without other inputs or prompts. “comprising,” “including,” “having,” and similar terms are synonymous, used comprehensively and in an open-ended manner, and do not exclude additional elements, features, actions, operations, and similar items. Furthermore, the term "or" is used in its inclusive (and not exclusive) sense; for example, to connect a list of elements, the term "or" can mean one, some, or all of the elements in the list.

[0183] All references, patents, patent applications, and publications cited or referred to in this application are incorporated herein by reference in their entirety.

Claims

1. A method by which a computer performs an action. Receiving video data representing the movement of the subject, Using the aforementioned video data, the characteristics of the spinal mobility of the subject during the duration of the video are determined. To determine the visual frailty score of the subject, at least one machine learning model is used to process at least the spinal mobility features. A method that includes this.

2. During the duration of the aforementioned video, the characteristics of the spinal mobility of the subject are determined. Determining multiple spinal measurement values, wherein each of the multiple spinal measurement values ​​corresponds to one video frame of the video data. Using the aforementioned multiple spinal measurements, the characteristics of spinal mobility are determined. A method performed by the computer according to claim 1, including the following:

3. During the duration of the aforementioned video, the characteristics of the spinal mobility of the subject are determined. For each video frame of the aforementioned video data, Determining a first distance between the head and tail of the object, To determine a second distance between the central part of the back of the subject and the midpoint between the head and the tail, Determining the angle formed between the head, tail, and central part of the back of the subject, The characteristics of the spinal mobility of the video frame are determined to include the first distance, the second distance, and the angle. A method performed by the computer according to claim 1, including the following:

4. During the duration of the aforementioned video, the characteristics of the spinal mobility of the subject are determined. For each video frame of the aforementioned video data, the distance between the central part of the target's back and the midpoint between the target's head and its tail is determined. A method performed by the computer according to claim 1, including the following:

5. During the duration of the video, the video data is processed using at least an additional machine learning model to determine pose estimation data that tracks the position of the object's head, the position of the object's tail, and the position of the object's central back. To determine the characteristics of the spinal mobility, the posture estimation data is used. A method performed by a computer according to claim 1, further comprising:

6. The video data is processed to determine pose estimation data that tracks the positions of at least 12 body parts of the subject during the duration of the video. Using the aforementioned posture estimation data, the characteristics of the object are determined, To determine the visual frailty score, the features are processed using the at least one machine learning model. A method performed by a computer according to claim 1, further comprising:

7. Determining the physical characteristics of the subject, wherein the physical characteristics correspond to at least one of the length of the subject, the width of the subject, and the distance between the hind limbs of the subject. To determine the visual frailty score, the physical characteristics are processed using the at least one machine learning model. A method performed by a computer according to claim 1, further comprising:

8. To determine the number of times the hind limb standing event occurs during the duration of the aforementioned video, To determine the length of hind leg standing for each hind leg standing event, In order to determine the visual frailty score, the number of times the hind limb standing event occurred and the length of the hind limb standing for each hind limb standing event are processed using the at least one machine learning model. A method performed by a computer according to claim 1, further comprising:

9. Using at least one machine learning model, the video data is processed to determine elliptic fitting data for the object during the duration of the video. Using the aforementioned elliptic fitting data, the characteristics of the target are determined, To determine the visual frailty score, the features are processed using the at least one machine learning model. A method performed by a computer according to claim 1, further comprising:

10. During the duration of the aforementioned video, the characteristics of the spinal mobility of the subject are to be determined. To determine a first set of video frames representing the walking motion of the aforementioned object, To determine a first set of spinal mobility features for a first set of the aforementioned video frames, To determine a second set of video frames representing non-walking motion by the aforementioned object, To determine a second set of spinal mobility features for a second set of the aforementioned video frames. Includes, The computer-based method according to claim 1, wherein the spinal mobility features include a first set of spinal mobility features and a second set of spinal mobility features.

11. The computer-operated method according to claim 10, wherein a first set of spinal mobility features corresponds to the distance between the central part of the dorsal portion of the object and the midpoint between the head and tail of the object, and a second set of spinal mobility features corresponds to the angles formed between the head, tail and central part of the dorsal portion of the object.

12. Using the aforementioned video data, determine the gait measurement value of the subject during the duration of the video, In order to determine the visual frailty score of the subject, the gait measurements are processed using the at least one machine learning model. A method performed by a computer according to claim 1, further comprising:

13. Processing the video data to determine point data that tracked the movement of a set of body parts of the subject during the duration of the video, Using the aforementioned point data, determine the multiple stance phases and multiple swing phases represented in the video data. Based on the aforementioned multiple stance phases and multiple swing phases, the multiple stride intervals represented in the video data are determined. The method involves determining a gait measurement using the aforementioned point data, wherein the gait measurement is determined based on each of the multiple stride intervals. A method performed by a computer according to claim 12, further comprising:

14. Processing the video data to determine point data for tracking the movement of a set of body parts during the duration of the video, wherein the set of body parts includes one or more of the nose, base of the neck, middle of the spine, left hind limb, right hind limb, base of the tail, middle of the tail, and tip of the tail. Using the aforementioned point data, the characteristics of the target are determined, To determine the visual frailty score, the features are processed using the at least one machine learning model. A method performed by a computer according to claim 1, further comprising:

15. To identify the likelihood that the subject exhibits grooming behavior toward multiple video frames of the video data, the video data is processed using an additional machine learning model, The visual frailty score is determined using the likelihood that the subject exhibits the grooming behavior. A method performed by a computer according to claim 1, further comprising:

16. To identify the likelihood that the subject will exhibit a predetermined action for multiple video frames of the video data, the video data is processed using an additional machine learning model, The likelihood of the subject exhibiting the predetermined behavior is used to determine the visual frailty score. A method performed by a computer according to claim 1, further comprising:

17. During the duration of the aforementioned video, the video data is processed to determine the walking measurement value for the subject, The process involves processing the video data to determine behavioral data that identifies a portion of the video in which the subject exhibits a predetermined behavior, To determine the visual frailty score, the at least one machine learning model is used to process the spinal mobility features, the gait measurements, and the behavioral data. A method performed by the computer according to claim 1, including the following:

18. The computer-based method according to claim 1, further comprising determining the physical condition of the subject using the visual frailty score.

19. The method performed by the computer according to claim 18, wherein the physical condition is frail.

20. The method performed by the computer according to claim 18, wherein the physical condition is a pre-frail state.

Images