Analysis of gait and posture

Abstract

The systems and methods described herein provide techniques for analyzing a subject's gait and posture against control data. The systems and methods, in some embodiments, process video data, identify key points representing body parts, determine metric data at the stride level, and compare the metric data to the control data.

Description

[Technical Field]

[0001] Related applications This application claims priority under U.S. Provisional Application No. 63 / 144,052, filed on 1 February 2021, and U.S. Provisional Application No. 63 / 131,498, filed on 29 December 2020, pursuant to Section 119(e) of the U.S. Patent Act, the entire contents of each document are incorporated herein by reference.

[0002] In some aspects, the present invention relates to automatically analyzing the gait and posture of a subject by processing video data.

[0003] Government support This invention was made with government support granted by the National Institutes of Health under R21DA048634 and UM1OD023222. The government has certain rights to this invention. [Background technology]

[0004] In humans, gait and posture can be quantified with high accuracy and sensitivity, and have been shown to be useful in assessing the proper functioning of many nervous and muscular systems. Many mental illnesses, neurodegenerative diseases, and neuromuscular diseases, including autism spectrum disorder, schizophrenia, bipolar disorder, and Alzheimer's disease, are associated with changes in gait and posture. This is because proper gait, balance, and posture are under the control of multiple nervous system processes, including key sensory centers that process visual, vestibular, auditory, proprioceptive, and visceral input. In the brain, regions that directly control movement, such as the cerebellum, motor cortex, and brainstem, respond to cognitive and emotional cues. Therefore, the integrity of gait and posture reflects the proper neural functioning of many nervous systems in humans. In rodent models of human psychiatric conditions, the usefulness of gait and posture indicators as in humans has not been demonstrated. This may be due to the lack of readily implementable techniques to detect differences in gait and posture between different mouse strains with sufficient accuracy. [Overview of the Initiative]

[0005] According to one aspect of the present invention, a computer implementation method is provided, which includes receiving video data representing a video capturing the movement of an object; processing the video data to identify point data tracking the movement of a set of multiple body parts of the object over a period of time; using the point data to determine a set of stance phases and corresponding set of swing phases represented in the video data at the time of the period; determining a set of stride intervals represented in the video data at the time of the period based on the set of stance phases and set of swing phases; using the point data to determine index data relating to the object, wherein the index data is determined based on each of the set of stride intervals; comparing the index data relating to the object with control index data; and determining the difference between the index data relating to the object and the control index data based on the comparison. In certain embodiments, a set of multiple body parts includes a nose, a base of the neck, a mid-spine, a left hind limb, a right hind limb, a base of the tail, a mid-tail, and a tail tip, and the multiple stance phases and multiple swing phases are determined based on changes in the locomotion speed of the left and right hind limbs. In certain embodiments, the method also includes determining the transition from the first stance phase of the multiple stance phases and from the first swing phase of the multiple swing phases based on a toe-off event of the left or right hind limb, and determining the transition from the second swing phase of the multiple swing phases to the second stance phase of the multiple stance phases based on a foot-contact event of the left or right hind limb. In some embodiments, the index data corresponds to the gait measurement of the subject at each stride interval.In some embodiments, a set of multiple body parts includes a left hind limb and a right hind limb, and determining the index data includes using point data to determine the step length for each stride interval, where the step length represents the distance the right hind limb moves beyond the immediate left hind limb's contact with the ground; using point data to determine the stride length used for each stride interval, where the stride length represents the distance the left hind limb moves at each stride interval between the left forelimb and the left hind limb for each stride interval from a toe-off event to a foot-contact event; and using point data to determine the step width for each stride interval, where the step width represents the distance between the left hind limb and the right hind limb. In some embodiments, a set of multiple body parts includes the tail base, and determining the index data includes using point data to determine the velocity data of the object based on the movement of the tail base for each stride interval. In certain embodiments, a set of multiple body parts includes the tail root, and determining index data includes using point data to determine a set of multiple velocity data for the object based on the movement of the tail root at a set of multiple frames representing a set of stride intervals, and averaging the set of multiple velocity data to determine the stride velocity with respect to the stride interval. In some embodiments, a set of multiple body parts includes a right hind limb and a left hind limb, and determining index data includes using point data to determine a first stance duration representing the amount of time the right hind limb is in contact with the ground at a stride interval, and determining a first load coefficient based on the first stance duration and the stride interval duration, using point data to determine a second stance duration representing the amount of time the left hind limb is in contact with the ground at a stride interval, and determining a second load coefficient based on the second stance duration and the stride interval duration, and determining an average load coefficient with respect to the stride interval based on the first load coefficient and the second load coefficient.In some embodiments, a set of multiple body parts includes the tail base and the neck base, and determining index data involves using point data to determine a set of multiple vectors connecting the tail base and the neck base at a set of multiple frames representing stride intervals with multiple stride intervals, and using the set of multiple vectors to determine the angular velocity of the object with respect to the stride interval. In certain embodiments, the index data corresponds to a posture measurement of the object at each stride interval. In some embodiments, a set of multiple body parts includes the mid-spine of the object, and stride intervals with multiple stride intervals are associated with a set of multiple frames of video data, and determining index data involves using point data to determine a displacement vector with respect to the stride interval, the displacement vector connecting the mid-spine represented in the first frame of the set of multiple frames and the mid-spine represented in the last frame of the set of multiple frames. In some embodiments, a set of multiple body parts further includes the nose of the subject, and determining the index data includes using point data to determine a set of multiple lateral displacements of the nose relative to a stride interval based on the perpendicular distance of the nose from the displacement vector for each frame of a set of multiple frames. In certain embodiments, the lateral displacement of the nose is further based on the body length of the subject. In some embodiments, determining the index data further includes generating a smooth curve of the lateral displacement of the nose relative to a stride interval by performing interpolation using the set of multiple lateral displacements of the nose, determining at what point in the stride interval the maximum displacement of the nose occurred, and determining a percentage stride position representing the percentage of the stride interval completed when the maximum displacement of the nose occurred.In some embodiments, a set of multiple body parts further includes the tail root of the subject, and determining the index data includes using point data to determine a set of multiple lateral displacements of the tail root with respect to a stride interval, based on the vertical distance of the tail root from the displacement vector for each frame in a set of multiple frames. In some embodiments, determining the index data further includes generating a smooth curve of the lateral displacements of the tail root with respect to a stride interval by performing interpolation using the set of multiple lateral displacements of the tail root, determining at what point in the stride interval the maximum displacement of the tail root occurred, and determining a percentage stride position representing the percentage of the stride interval completed when the maximum displacement of the tail root occurred. In certain embodiments, a set of multiple body parts also includes the tail tip of the subject, and determining index data includes using point data to determine a set of multiple lateral displacements of the tail tip relative to the stride interval based on the vertical distance of the tail tip from the displacement vector for each frame in a set of multiple frames. In some embodiments, determining index data also includes generating a smooth curve of the tail tip's lateral displacement relative to the stride interval by performing interpolation using the set of multiple lateral displacements of the tail tip, determining at what point in the stride interval the maximum displacement of the tail tip occurred, and determining a percentage stride position representing the percentage of the stride interval completed when the maximum displacement of the tail tip occurred, thereby determining the tail tip's displacement phase offset. In some embodiments, processing video data includes processing video data using a machine learning model. In certain embodiments, processing video data includes processing video data using a neural network model. In certain embodiments, the video is... From aboveThe system captures the target determination movement of an object within an open arena within the field of view. In some embodiments, control index data is obtained from one or more control organisms. In some embodiments, the subject is an organism, and the control organism and the subject organism are of the same species. In certain embodiments, the species is a member of the order Rodentia, and is optionally a rat or a mouse. In certain embodiments, the control organism is a laboratory strain of that species. In some embodiments, the laboratory strains are those listed in Figure 14E. In some embodiments, statistically significant differences in the subject's index data compared to the control index data indicate differences in the subject's phenotype compared to the control organism's phenotype. In some embodiments, phenotypic differences indicate the presence of disease or pathology in the subject. In certain embodiments, phenotypic differences indicate differences between the genetic background of the subject and the genetic background of the control organism. In some embodiments, statistically significant differences between the subject's index data and the control index data indicate differences in the subject's genotype compared to the control organism's genotype. In certain embodiments, genotypic differences indicate differences in strains between the subject and the control organism. In certain embodiments, differences in genotype indicate the presence of a disease or condition in the subject. In some embodiments, the disease or condition is Rett syndrome, Down syndrome, amyotrophic lateral sclerosis (ALS), autism spectrum disorder (ASD), schizophrenia, bipolar disorder, neurodegenerative disorder, dementia, or brain injury. In some embodiments, the control organism and the subject organism are of the same sex. In certain embodiments, the control organism and the subject organism are not of the same sex. In some embodiments, the control index data corresponds to elements including the control's stride length, control's step length, and control's step width, while the subject index data includes elements including the subject's stride length, subject's step length, and subject's step width at a given time, and differences between one or more elements of the control data and one or more elements of the index data indicate phenotypic differences between the subject and the control.

[0006] Another aspect of the present invention provides a method for evaluating one or more activities and one or more behaviors in relation to a subject known to have a disease or condition, a subject suspected to have a disease or condition, or a subject at risk of having a disease or condition, the method comprising: obtaining index data relating to the subject, wherein the means for obtaining the data include a computer-generated method according to any embodiment of the above-described method or system of the present invention; and determining the presence or absence of a disease or condition at least in part on the obtained index data. In some embodiments, the method also comprises selecting a treatment regimen relating to the subject at least in part on the presence of the determined disease or condition. In some embodiments, the method also comprises applying the selected treatment regimen to the subject. In some embodiments, the method also comprises obtaining index data relating to the subject at a point in time after the application of the treatment regimen; and optionally comparing the initially obtained index data with the subsequently obtained index data to determine the effectiveness of the applied treatment regimen. In some embodiments, the method also includes repeating, increasing, or decreasing the application of a selected treatment regimen to a subject, at least in part, based on a comparison of the initially obtained index data with the subsequently obtained index data for the subject. In some embodiments, the method also includes comparing the obtained index data with control index data. In some embodiments, the disease or condition is a neurodegenerative disorder, neuromuscular disorder, neuropsychiatric disorder, ALS, autism, Down syndrome, Rett syndrome, bipolar disorder, dementia, depression, attention deficit hyperactivity disorder, anxiety disorder, developmental disorder, sleep disorder, Alzheimer's disease, Parkinson's disease, physical injury, etc.Additional diseases and disorders, as well as animal models, that can be evaluated using the methods and / or systems of the present invention are known in the art; see, for example, Barrot M. Neuroscience 2012;211:39-50, Graham, DM, Lab Anim (NY) 2016;45:99-101, Sewell, RDE, Ann Transl Med 2018;6:S42.2019 / 01 / 08, and Jourdan, D., et al, Pharmacol Res 2001;43:103-110.

[0007] Another aspect of the present invention provides a method for identifying a subject as an animal model of a disease or condition, the method comprising: obtaining index data relating to the subject, wherein the means for obtaining the data include a computer-generated method according to any embodiment of the method or system described above of the present invention; and determining one or more features relating to a disease or condition in the subject, at least in part, based on the obtained index data, thereby identifying the subject as an animal model of a disease or condition due to the presence of one or more features relating to a disease or condition in the subject. In some embodiments, the method also includes an additional evaluation of the subject. In some embodiments, the disease or condition is a neurodegenerative disorder, neuromuscular disorder, neuropsychiatric disorder, ALS, autism, Down syndrome, Rett syndrome, bipolar disorder, dementia, depression, attention deficit hyperactivity disorder, anxiety disorder, developmental disorder, sleep disorder, Alzheimer's disease, Parkinson's disease, physical injury, etc. In some embodiments, the method also includes comparing the obtained index data with control index data and identifying one or more similarities or differences between the obtained index data and the control index data, the identified similarities or differences assisting in identifying the subject as an animal model of a disease or pathology.

[0008] According to another aspect of the present invention, a method is provided for determining the presence or absence of an effect of a candidate compound on a disease or condition, the method comprising obtaining first index data relating to a subject, the means for obtaining this data comprising a computer-generated method according to any embodiment of the computer-generated aspects of the method described above, the subject being a person having a disease or condition, or a person forming an animal model relating to a disease or condition, administering the candidate compound to the subject, obtaining post-administration index data relating to the organism, and comparing the first index data with the post-administration index data, wherein the difference between the first index data and the post-administration index data identifies, compares, the effect of the candidate compound on the disease or condition. In some embodiments, the method also comprises additionally testing the effect of the compound on the treatment of the disease or condition.

[0009] According to another aspect of the present invention, a method is provided for identifying the presence of an effect of a candidate compound on a disease or condition, the method comprising: administering a candidate compound to a subject having a disease or condition, or to a subject constituting an animal model of a disease or condition; obtaining index data relating to the subject, wherein the means for obtaining the data include a computer generation method according to any embodiment of the computer generation aspect of the method described above; and comparing the obtained index data with control index data, wherein the difference between the obtained index data and the control index data identifies the presence of an effect of the candidate compound on the disease or condition.

[0010] According to another aspect of the present invention, a system is provided which includes at least one processor and at least one memory which includes instructions causing the system, when executed by the at least one processor, to receive video data representing a video capturing the movement of an object; to process the video data to identify point data tracking the movement of a set of multiple body parts of the object over a period of time; to use the point data to determine a set of stance phases and a corresponding set of swing phases represented in the video data at a given time; to determine a set of stride intervals represented in the video data at a given time based on the set of stance phases and swing phases; and to use the point data to determine index data relating to the object, wherein the index data is determined based on each of the set of stride intervals; to compare the index data relating to the object with reference index data; and to determine the difference between the index data relating to the object and the reference index data based on the comparison. In some embodiments, a set of body parts includes a nose, a base of the neck, a mid-spine, a left hind limb, a right hind limb, a base of the tail, a mid-tail, and a tail tip, and the multiple stance phases and multiple swing phases are determined based on changes in the movement speed of the left and right hind limbs. In certain embodiments, at least one memory also includes instructions, when executed by at least one processor, to cause the system to further determine transitions from the first stance phase of the multiple stance phases and from the first swing phase of the multiple swing phases based on toe-off events of the left or right hind limb, and to determine transitions from the second swing phase of the multiple swing phases to the second stance phase of the multiple stance phases based on foot-contact events of the left or right hind limb. In certain embodiments, index data corresponds to gait measurements of the subject at each stride interval.In some embodiments, a pair of body parts includes a left hind limb and a right hind limb, and a command to the system to determine index data further causes the system to determine, using point data, the step length for each stride interval, wherein the step length represents the distance the right hind limb moves beyond the immediate left hind limb's contact with the ground; to determine, using point data, the stride length to be used for each stride interval, wherein the stride length represents the distance the left hind limb moves at each stride interval; and to determine, using point data, the step width for each stride interval, wherein the step width represents the distance between the left hind limb and the right hind limb. In some embodiments, a pair of body parts includes a tail base, and a command to the system to determine index data further causes the system to determine, using point data, velocity data for the object based on the movement of the tail base for each stride interval. In a particular embodiment, a set of multiple body parts includes the tail root, and a command to the system to determine index data further causes the system to determine a set of multiple velocity data about the object based on the movement of the tail root at a set of multiple frames representing stride intervals with multiple stride intervals, using point data, and to determine stride velocity with respect to stride intervals by averaging the set of multiple velocity data. In a particular embodiment, a set of multiple body parts includes a right hind limb and a left hind limb, and a command to the system to determine index data further causes the system to: use point data to determine a first stance duration representing the amount of time the right hind limb is in contact with the ground during a stride interval with multiple stride intervals; determine a first load coefficient based on the first stance duration and the stride interval duration; use point data to determine a second stance duration representing the amount of time the left hind limb is in contact with the ground during a stride interval; determine a second load coefficient based on the second stance duration and the stride interval duration; and determine an average load coefficient with respect to the stride interval based on the first load coefficient and the second load coefficient.In some embodiments, a set of multiple body parts includes the tail base and the neck base, and a command to the system to determine index data further causes the system to use point data to determine a set of multiple vectors connecting the tail base and the neck base at a set of multiple frames representing a stride interval with multiple stride intervals, and to use the set of multiple vectors to determine the angular velocity of the object with respect to the stride interval. In some embodiments, the index data corresponds to a posture measurement of the object at each stride interval. In some embodiments, a set of multiple body parts includes the mid-spine of the object, and a stride interval with multiple stride intervals is associated with a set of multiple frames of video data, and a command to the system to determine index data further causes the system to use point data to determine a displacement vector with respect to the stride interval, the displacement vector connecting the mid-spine represented in the first frame of the set of multiple frames to the mid-spine represented in the last frame of the set of multiple frames. In certain embodiments, a set of multiple body parts also includes the nose of the subject, and a command to the system to determine index data further causes the system to determine a set of multiple lateral displacements of the nose relative to the stride interval, based on the perpendicular distance of the nose from the displacement vector for each frame of the set of multiple frames, using point data. In some embodiments, the lateral displacement of the nose is further based on the body length of the subject. In some embodiments, a command to the system to determine index data further causes the system to generate a smooth curve of the lateral displacement of the nose relative to the stride interval by performing interpolation using the set of multiple lateral displacements of the nose, to determine at what point in the stride interval the maximum displacement of the nose occurred, and to determine a percentage stride position representing the percentage of the stride interval completed when the maximum displacement of the nose occurred.In certain embodiments, a set of multiple body parts also includes the tail root of the subject, and a command to the system to determine index data further causes the system to determine a set of multiple lateral displacements of the tail root with respect to the stride interval, based on the perpendicular distance of the tail root from the displacement vector for each frame in a set of multiple frames, using point data. In some embodiments, a command to the system to determine index data further causes the system to determine the displacement phase offset of the tail root by performing interpolation using the set of multiple lateral displacements of the tail root to generate a smooth curve of the lateral displacement of the tail root with respect to the stride interval, using the smooth curve of the lateral displacement of the tail root to determine at what point in the stride interval the maximum displacement of the tail root occurred, and determining a percentage stride position representing the percentage of the stride interval completed when the maximum displacement of the tail root occurred. In certain embodiments, a set of multiple body parts also includes the tail tip of the subject, and a command to the system to determine index data further causes the system to determine a set of multiple lateral displacements of the tail tip relative to the stride interval, based on the perpendicular distance of the tail tip from the displacement vector for each frame in a set of multiple frames, using point data. In some embodiments, a command to the system to determine index data further causes the system to generate a smooth curve of the lateral displacement of the tail tip relative to the stride interval by performing interpolation using the set of multiple lateral displacements of the tail tip, to determine at what point in the stride interval the maximum displacement of the tail tip occurred, and to determine a percentage stride position representing the percentage of the stride interval completed when the maximum displacement of the tail tip occurred. In certain embodiments, a command to the system to process video data further causes the system to process the video data using a machine learning model.In some embodiments, a command to the system to process video data further causes the system to process the video data using a neural network model. In some embodiments, the video is... From aboveThe system captures the target determination movement of an object within an open arena within the field of view. In certain embodiments, control index data is obtained from one or more control organisms. In some embodiments, the subject is an organism, and the control organism and the subject organism are of the same species. In some embodiments, the species is a member of the order Rodentia, and is optionally a rat or a mouse. In certain embodiments, the control organism is a laboratory strain of that species. In certain embodiments, the laboratory strains are those listed in Figure 14E. In some embodiments, statistically significant differences in the subject's index data compared to the control index data indicate differences in the subject's phenotype compared to the control organism's phenotype. In some embodiments, phenotypic differences indicate the presence of disease or pathology in the subject. In certain embodiments, phenotypic differences indicate differences between the genetic background of the subject and the genetic background of the control organism. In some embodiments, statistically significant differences between the subject's index data and the control index data indicate differences in the subject's genotype compared to the control organism's genotype. In some embodiments, genotypic differences indicate differences in strains between the subject and the control organism. In some embodiments, differences in genotype indicate the presence of a disease or condition in the subject. In certain embodiments, the disease or condition is Rett syndrome, Down syndrome, amyotrophic lateral sclerosis (ALS), autism spectrum disorder (ASD), schizophrenia, bipolar disorder, neurodegenerative disorder, dementia, or brain injury. In certain embodiments, the control organism and the subject organism are of the same sex. In some embodiments, the control organism and the subject organism are not of the same sex. In some embodiments, the control index data corresponds to elements including the control's stride length, control's step length, and control's step width, while the subject index data includes elements including the subject's stride length, subject's step length, and subject's step width at a given time, and differences between one or more elements of the control data and one or more elements of the index data indicate phenotypic differences between the subject and the control. [Brief explanation of the drawing]

[0011] For a more complete understanding of this disclosure, please refer to the following description in conjunction with the attached drawings.

[0012] [Figure 1] Figure 1 is a conceptual diagram of an exemplary system for determining indicators related to a subject's gait and posture, according to an embodiment of the present disclosure. [Figure 2] Figure 2 is a flowchart illustrating an exemplary process that may be performed by the system shown in Figure 1 for analyzing video data of a subject to determine an index relating to the subject's gait and posture, according to an embodiment of the present disclosure. [Figure 3] Figure 3 is a flowchart illustrating an exemplary process that may be performed by the point tracking component shown in Figure 1 to track a target body part in video data, according to embodiments of the present disclosure. [Figure 4] Figure 4 is a flowchart illustrating an exemplary process that may be performed by the system shown in Figure 1 to determine the stride interval according to an embodiment of the present disclosure. [Figure 5] Figure 5 is a flowchart illustrating an exemplary process that may be performed by the gait analysis components shown in Figure 1 to determine an index relating to the gait of a subject, according to an embodiment of the present disclosure. [Figure 6] Figure 6 is a flowchart illustrating an exemplary process that may be performed by the posture analysis components shown in Figure 1 to determine an index relating to the posture of a subject, according to an embodiment of the present disclosure. [Figure 7A] Figures 7A-7C show schematic diagrams and graphs illustrating deep convolutional neural networks for pose estimation. Figure 7A shows the HRNet-W32 neural network architecture for performing pose estimation. [Figure 7B] Figure 7B shows the inference pipeline, which sends out video, frames it into HRNet, and generates a heatmap consisting of 12 keypoints as output. [Figure 7C]Figure 7C presents the training loss curve, indicating that the network converges without overfitting. [Figure 8A] Figures 8A - J show schematic diagrams and graphs illustrating the derivation of walking phenotypes from video pose estimation. Figures 8A - B show the spatial and temporal characteristics of walking (based on figures from Green et al., Dev Med Child Neurol (2009) 51:311). Figure 8A is a diagram showing how the step length, stride interval, and stride length, which are three spatial stride indicators, are derived from the hindlimb foot - contact positions. [Figure 8B] Figure 8B is a Hildebrand plot, and all the indicators shown within this plot have percent - stride time as the unit. This shows the relationship between the foot - contact event and the toe - off event with respect to the stance and swing phases of the stride. [Figure 8C] Figure 8C shows a single frame of the input video plotting the hindlimb trajectory over the past 50 frames and the future 50 frames. The positions of the hindlimb contact events are indicated by black circles. Of the three lines, the outermost line is the right hindlimb, the middle line is the base of the tail, and the innermost line is the left hindlimb. [Figure 8D] Figures 8D - F show three plots indicating different aspects of the mouse movement over the same 100 frames. The central vertical line indicates the current frame (shown in Figure 8C). Figure 8D shows three lines indicating the velocities regarding the left hindlimb, right hindlimb, and base of the tail. The thick vertical lines within the plot indicate the estimated start frames of each stride. <000009o> [Figure 8E] The same as above. [Figure 8F] The same as above. [Figure 8G] Figure 8G shows the distribution of confidence values for each of the 12 estimated points. [Figure 8H] Figure 8H presents an aggregated plot for the Hildebrand plot regarding the hindlimb binned according to angular velocity. [Figure 8I] Figure 8I shows the same results as Figure 8H, except that the data was binned by velocity. [Figure 8J] Figure 8J shows that the load coefficients of the limbs change as a function of velocity. [Figure 9] Figures 9A-I present schematic diagrams and graphs illustrating the extraction of periodic whole-body posture indices during the gait cycle. Several indices are related to periodic lateral displacements observed at pose keypoints. Lateral displacement measurements were defined as orthogonal offsets from the associated stride displacement vector. The displacement vector was defined as a line connecting the midpoint of the mouse's spine in the first frame of the stride to the midpoint of the mouse's spine in the last frame of the stride. This offset was calculated for each frame of the stride, and then cubic interpolation was performed to generate a smooth displacement curve. The phase offset of the displacement was defined as the percentage stride position where the maximum displacement occurred on this smoothed curve. The lateral displacement index assigned to a stride was defined as the difference between the maximum and minimum displacement values ​​observed during the stride. Lateral displacement was measured with respect to the tail tip (Figure 9A) and the nose (Figure 9B). Furthermore, by averaging displacements across many strides within a cohort, consensus diagrams such as (Figure 9D) C57BL / 6J vs. (Figure 9E) NOR / LtJ could be formed, or by averaging many strides within an individual, consensus diagrams such as (Figure 9F) C57BL / 6J vs. (Figure 9G) NOR / LtJ could be formed. Figures 9H and 9I show the diversity of lateral displacements between sets of lines selected from the lineage survey. The thin (translucent) bands on these two plots represent the 95% confidence interval for the mean values ​​for each lineage. [Figure 10A] Figures 10A-E illustrate the results of the genetic validation of the walking mutant. Figure 10A shows the q-value (left) and effect size (right) obtained from a Reiner mixed-effects model and a circular linear model adjusted for body length and age. [Figure 10B]In Figure 10B, the difference in stride velocity between the control and the mutant was tested by comparing the kernel density estimate and the cumulative distribution function for the velocity distribution. [Figure 10CD] In Figure 10C, total distance traveled and speed were compared between controls and mutants using linear and linear mixed-effects models adjusted for body length and age. Figure 10D shows the results for gait indices adjusted for body length, which revealed differences with respect to the linear mixed-effects model. [Figure 10E] Figure 10E shows the results for lateral displacement of the nose and tail tip for the Ts65Dn lineage. The solid line represents the mean displacement of the stride, and the light (translucent) band represents the 95% confidence interval for the mean value. [Figure 11A] Figures 11A-F present tables and graphs illustrating the genetic validation of autism variants. Figure 11A shows the q-values ​​(left) and effect sizes (right) obtained from model M1 for the linear phenotype and from the circular-linear model for the circular phenotype. [Figure 11B] Figure 11B shows the q-value (left) and effect size (right) obtained from model M3 for the linear phenotype, and from the circular-linear model for the circular phenotype. [Figure 11CD] In Figure 11C, total distance traveled and speed were compared between controls and mutants using linear and linear mixed-effects models adjusted for body length and age. In each pair shown, the data on the left are from the control, and the data on the right are from the mutant. Figure 11D shows the gait indices adjusted for body length, which were found to differ with respect to the linear mixed-effects model. [Figure 11E] Figure 11E illustrates the use of the first two principal components to construct a 2D representation of the multidimensional space in which the control and mutant are best separated. [Figure 11F]Figure 11F shows the cumulative distribution of velocity in the ASD model. The upper curve represents the control, and the lower curve represents the mutants. Cntnap2, Fmr1, and Del4Aam have smaller stride velocities, while Shank3 has a larger stride velocity. [Figure 12A] Figures 12A-E show the results for the tested strains. In Figure 12A, each box plot corresponds to a strain, and the vertical position indicates the residual stride length adjusted for body length. The strains are sorted by the median residual stride length value. [Figure 12B] Figure 12B shows the z-scores for length-adjusted gait indices for all strains, color-coded by cluster membership (see Figure 12C). [Figure 12C] Figure 12C shows that the K-means algorithm was used to construct a 2D representation of the multidimensional space in which the lineages are best separated, using the first two principal components. The upper right region is cluster 1, the lower region is cluster 2, and the upper left region is cluster 3. [Figure 12D] Figure 12D presents a consensus diagram of lateral displacement of the nose and tail tip across clusters. The solid line represents the mean stride displacement, and the translucent band indicates the 95% confidence interval for the mean. [Figure 12E] Figure 12E is a plot after clustering, summarizing the residual gait index across different clusters. In each of the three groups, the left side is cluster 1, the center is cluster 2, and the right side is cluster 3. [Figure 13A] Figures 13A-D present the results of the GWAS for walking phenotypes. Figure 13A shows the estimated heritability of the mean (left) and variance (right) for each phenotype. [Figure 13B]Figures 13B–D present Manhattan plots for all mean phenotypes (Figure 13B), for the variance phenotype (Figure 13C), and for the combined phenotype (Figure 13D), with the colors corresponding to the phenotype with the smallest p-value for single nucleotide polymorphisms (SNPs). [Figure 13C] Same as above. [Figure 13D] Same as above. [Figure 14A] Figures 14A-D present a list of animal strains used in specific implementations of the present invention. Figure 14A shows the control strain and the official identifier for the walking mutant. [Figure 14B] Figure 14B shows the control line and the official identifiers for the autistic variant. [Figure 14C] Figure 14C shows a table summarizing the body length and weight of the animals in the experiment. [Figure 14D] Figure 14D presents a table summarizing the animals used in the phylogenetic study. [Figure 14E] [Figure 15A] Figures 15A-E present heatmaps, curves, and plots. Figure 15A is a heatmap summarizing the effect size and q-value obtained from model M3, where phenotype ~ genotype + test age + velocity + body length + (1|mouse ID / test age). [Figure 15B] Figure 15B shows the kernel density curve (left) and cumulative density curve (right) with respect to velocity across all systems. [Figure 15C] Figure 15C is a plot showing the positive association between body length and sex across different walking mutant lines. In each pair of results, the control is on the left side of the pair, and the mutant is on the right side. [Figure 15D] Figure 15D shows the residual-adjusted limb load coefficients and step lengths for the Mecp2 gait mutant, along with body length (M1), velocity (M2), and body length and velocity (M3). [Figure 15E]Figure 15E shows the residual-adjusted body length (M1), velocity (M2), and body length and velocity (M3) for the Mecp2 walking mutant, along with step width and stride length. [Figure 16A] Figures 16A-E present heatmaps, curves, and plots. Figure 16A is a heatmap summarizing the effect size and q-value obtained from model M2, where phenotype ~ genotype + test age + velocity + (1|mouse ID / test age). [Figure 16B] Figure 16B shows the kernel density curves with respect to velocity across all systems. [Figure 16C] Figure 16C is a plot showing the positive association between body length and sex across different walking mutant lines. In each pair of results, the control is on the left side of the pair, and the mutant is on the right side. [Figure 16D] Figure 16D shows the residual-adjusted body length (M1), velocity (M2), and body length and velocity (M3) for the Shank3 autistic variant, along with step length and stride length. [Figure 16E] Figure 16E shows the residual-adjusted body length (M1), velocity (M2), and body length and velocity (M3) for the Del4Aam autism variant, along with step length and stride length. In each pair of results shown, the control is on the left side of the pair, and the variant is on the right side. [Figure 17A] Figures 17A–F show the results for length-adjusted phenotypes compared across 62 strains in the phylogenetic study. The box plots are displayed from left to right in ascending order of median. Each panel (Figures 17A–F) corresponds to a different gait phenotype. [Figure 17B] Same as above. [Figure 17C] Same as above. [Figure 17D] Same as above. [Figure 17E] Same as above. [Figure 17F] Same as above. [Figure 18A]Figures 18A–E show the results for length-adjusted phenotypes compared across 62 strains in the phylogenetic study. The box plots are displayed from left to right in ascending order of median. Each panel (Figures 18A–E) corresponds to a different gait phenotype. [Figure 18B] Same as above. [Figure 18C] Same as above. [Figure 18D] Same as above. [Figure 18E] Same as above. [Figure 19] Figure 19 presents a summary list of effect sizes and FDR-adjusted p-values ​​obtained from models M1, M2, and M3 for all phenotypes and gait systems. [Figure 20] Figure 20 presents a summary list of effect sizes and FDR-adjusted p-values ​​obtained from models M1, M2, and M3 for all phenotypes and autism spectrum lines. [Figure 21A]Figures 21A-D show three optimal clusters in the phylogenetic survey data. Thirty clustering indicators were considered to select the optimal number of clusters (Bates et al., J Stat Softw (2015) 67:1). Figure 21A illustrates that the majority indicated the possibility of two or three clusters existing within the phylogenetic survey data. One key criterion for selecting the optimal number of clusters is to maximize the inter-cluster distance while keeping the intra-cluster distance small. For this purpose, the sum of squares (WSS) (shown in Figure 21B) was examined, the Calinski-Harabasz (CH) index (shown in Figure 21C) was examined [Calinski, T. & Harabasz, Communications in Statistics-theory and Methods 3, 1-27 (1974)], and a comparison was made between the gap statistic [Tibshirani, R. et al. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 63, 411-423 (2001)] (shown in Figure 21D) and the number of clusters adopted. All of these indicate that three clusters are the optimal choice. [Figure 21B] Same as above. [Figure 21C] Same as above. [Figure 21D] Same as above. [Figure 22-1] Figure 22 shows a table of significant GWAS hits related to walking and postural phenotypes. This information includes study results showing quantitative trait locus (QTL) peak SNPs, QTL peak SNP locations, QTL start locations, QTL end locations, allele 1, allele 2, allele 1 frequency, p-Wald, protein-coding genes, and the group in which the QTL was found to be significant. [Figure 22-2] Same as above. [Figure 22-3] Same as above. [Figure 22-4] Same as above. [Figure 22-5] Same as above. [Figure 22-6] Same as above. [Figure 22-7] Same as above. [Figure 22-8] Same as above. [Figure 22-9] Same as above. [Figure 22-10] Same as above. [Figure 22-11] Same as above. [Figure 22-12] Same as above. [Figure 22-13] Same as above. [Figure 22-14] Same as above. [Figure 22-15] Same as above. [Figure 22-16] Same as above. [Figure 22-17] Same as above. [Figure 22-18] Same as above. [Figure 22-19] Same as above. [Figure 22-20] Same as above. [Figure 22-21] Same as above. [Figure 22-22] Same as above. [Figure 22-23] Same as above. [Figure 22-24] Same as above. [Figure 22-25] Same as above. [Figure 22-26] Same as above. [Figure 22-27] Same as above. [Figure 22-28] Same as above. [Figure 22-29] Same as above. [Figure 22-30] Same as above. [Figure 22-31] Same as above. [Figure 23] Figure 23 is a block diagram conceptually illustrating exemplary components of a device according to an embodiment of the present disclosure. [Figure 24] Figure 24 is a block diagram conceptually illustrating exemplary components of a server according to an embodiment of the present disclosure. [Modes for carrying out the invention]

[0013] The present invention includes, in part, a method for determining differences / deviations from a control by processing video data to first track a target body part, determine data representing gait and postural indices, and then perform statistical analysis. The method and system of the present invention provide a reliable and scalable automated system for extracting gait and postural level features, dramatically reducing the time and labor costs associated with experiments on neurogenetic behavior, as well as reducing variability in such experiments.

[0014] The analysis of human and animal movement, including gait, has a long history. Aristotle was the first to write philosophical treatises on animal movement and gait, using both physical and metaphysical principles. During the Renaissance, Borelli applied the laws of physics and biomechanics to the muscles, tendons, and joints of the entire body to understand gait. Muybridge and Murray were the first to apply imaging techniques to the study of gait, taking sequential photographic images of moving humans and animals with the aim of deriving quantitative measurements of gait. Modern analytical methods for animal gait were developed in the 1970s by Hildebrandt, who classified gait based on quantified indicators. He defined the gait cycle in terms of limb contact with the ground (stance phase and swing phase). Essentially, this concept has remained unchanged for the past 40 years, and current methods for analyzing mouse gait, while improving upon Muybridge and Murray's imaging approach, are still fundamentally based on the timing of limb contact with the ground. This contrasts with human gait and posture analysis, which, since Borrelli's time, has focused on body posture and quantified whole-body movement rather than just contact with the ground. This difference between mice and humans is likely due in part to the difficulty in automatically estimating the posture of rodents, which appear as deformable objects with joint positions obscured by fur. In addition, unlike humans, parts of a mouse cannot be easily marked in terms of position using clothing. In rodents, although recent methods have advanced in determining whole-body coordination, these still require specialized equipment and make the animal walk in a fixed orientation in a walkway or treadmill, or in a narrow passage, for proper imaging and accurate determination of limb position. This is highly unnatural, and since animals often require training to perform this movement properly, it limits the use of this type of measurement method in correlating with human gait. Lateral imaging suffers from perspective impairment, but this can be overcome by restricting the animal's movement to a single depth field.Furthermore, defecation and urination by animals, or the presence of bedding, inevitably lead to obstruction, making long-term observation from this perspective impractical. In fact, behaviorally relevant gait data when animals are free to move often yield different results than treadmill-based measurements. Moreover, commercially available treadmill-based or aisle-based systems for gait analysis often produce numerous measurements that show different results in the same animal model. Identifying the precise causes of such discrepancies in a self-contained, obstructed system is difficult. Therefore, there is currently a lack of easily and widely implementable tools for measuring gait and posture in freely moving animals.

[0015] Open-field measurements are one of the oldest and most commonly used measurement methods in behavioral neurogenetics. In rodents, they have been classically used to measure intermediate phenotypes related to emotions, such as hyperactivity, anxiety, exploration, and habituation. In video-based open-field measurements, the rich and complex behaviors of animal movement are abstracted to simple points in order to extract behavioral measurements. This overly simplified abstraction is necessary primarily due to the technical limitations of not being able to accurately extract complex poses from video data. New technologies have the potential to overcome this limitation. Walking, an important indicator of neural function, is not typically analyzed in open fields in conventional systems, mainly because it is technically difficult to determine the position of the limbs when the animal is moving freely. If open-field measurements can be combined with gait and posture analysis, important insights into the neural and genetic control of animal behavior will be provided in an behaviorally relevant manner. The invention of this disclosure performs gait and posture analysis of an object in an open field by utilizing modern machine learning models such as neural networks. The present invention is invariant to the high degree of visual diversity observed in subjects such as mice, including differences in fur color, fur texture, and size. From aboveThis invention relates to a system and method for measuring gait parameters and whole-body posture parameters from this perspective. Overall, the present invention provides a system that is highly sensitive, highly accurate, and scalable, and that can detect previously undescribed differences in gait and posture in mouse models of diseases and pathological conditions.

[0016] This disclosure relates to a technique for gait analysis and posture analysis including a plurality of modular components, one of which, in some embodiments, relates to an open field. From above This is a neural network (e.g., a deep convolutional neural network) trained to perform pose estimation using video. The neural network may provide multiple two-dimensional markers (in some embodiments, 12 such markers) relating to the anatomical location (also referred to as "keypoints") of the subject for each video frame describing the subject's pose at each point in time. Another modular component may be capable of processing a time series of poses and identifying intervals representing individual strides. Yet another modular component may be capable of extracting multiple gait indices for each stride, and yet another modular component may be capable of extracting multiple postural indices. Furthermore, yet another modular component may be configured to perform statistical analysis on the gait and postural indices and to aggregate large amounts of data to provide a consensus diagram of the subject's gait structure.

[0017] The System 100 of this Disclosure may operate using various components shown in Figure 1. System 100 may include an image acquisition device 101, a device 102, and one or more systems 150, connected via one or more networks 199. The image acquisition device 101 may be part of another device (e.g., device 1600), or contained within such a device, or connected to such a device, and may also be a camera, a high-speed video camera, or other type of device capable of acquiring images or video. In addition to or instead of the image acquisition device, device 101 may include motion detection sensors, infrared sensors, temperature sensors, ambient condition detection sensors, and other sensors configured to detect various characteristics / environmental conditions. Device 102 may be a laptop, desktop, tablet, smartphone, or other type of computing device, and may include one or more components described below in relation to device 1600.

[0018] The image acquisition device 101 may acquire video (or one or more images) of one or more subjects on which a formalin assay is performed, and may transmit video data 104 representing the video to the system 150 for processing as described herein. The system 150 may include one or more components shown in Figure 1 and may be configured to process the video data 104 to determine the gait and postural behavior of the subject over time. The system 150 may determine difference data 148 representing one or more differences between the gait and / or posture of the subject and the gait and / or posture of a control. The difference data 148 may be transmitted to the device 102 for output to the user to observe the processing results of the video data 104.

[0019] Details of the components of System 150 will be described later. The various components may be located on the same physical device or on different physical devices. Communication between the various components may occur directly or via the network 199. Communication between device 101, System 150, and device 102 may occur directly or via the network 199. One or more components shown as part of System 150 may be located at device 102 or at a computing device (e.g., device 1600) connected to the image acquisition device 101. In an exemplary embodiment, System 150 may include a point tracking component 110, a gait analysis component 120, a posture analysis component 130, and a statistical analysis component 140. In other embodiments, System 150 may include fewer or more components than those shown in Figure 1 to perform the same or similar functions as those described later.

[0020] Figure 2 is a flowchart illustrating an exemplary process 200 that may be performed by the system 100 shown in Figure 1 to determine gait and posture indicators by analyzing video data 104 of a subject, according to embodiments of the present disclosure. At a high level, the process 200 is initiated by an image acquisition device 101 recording video of the subject's movements. In some embodiments, the video data 104 is related to the subject. From aboveThis is a viewpoint. In some embodiments, the subject may be in an enclosure with an open arena without using, for example, a treadmill or a tunnel to guide the subject in a particular manner. This allows the subject to be observed without the need to train the subject to perform specific actions such as walking on a treadmill or moving through a tunnel. In step 202, the system 150 may receive video data 104 from the image acquisition device 101 (or from a device 1600 connected to the image acquisition device 101, or from a device 1600 that encloses the image acquisition device 101). In step 204, the point tracking component 110 of the system 150 may determine point data 112 by processing the video data 104. The point data 112 may represent data tracking the movement of a set of subject body parts over a period of time represented in the video data 104. Further details regarding the point tracking component 110 will be described later in relation to Figure 3. In step 206, the gait analysis component 120 of the system 150 may determine index data 122 by processing the point data 112. The index data 122 may represent gait indices for the subject. Further details regarding the gait analysis component 120 will be described later in relation to Figure 4. In step 208, the posture analysis component 130 of the system 150 may determine index data 132 by processing the point data 112. The index data 132 may represent posture indices for the subject. Further details regarding the posture analysis component 130 will be described later in relation to Figure 5. In some embodiments, step 208 may be performed before step 206. In some embodiments, steps 206 and 208 may be performed in parallel, for example, while the posture analysis component 130 is processing the point data 112, the gait analysis component 120 may process the point data 112. In some embodiments, depending on the system configuration, only one of steps 206 and 208 may be performed. For example, in some embodiments, the system 150 may be configured to determine only gait indicators, thereby allowing only step 206 to be performed by the gait analysis component 120.In another example, in some embodiments, the system 150 may be configured to determine only postural indices, thereby allowing only step 208 to be performed by the gait analysis component 120. In step 210, the statistical analysis component 140 of the system 150 may determine difference data 148 by processing index data 122, index data 132, and control data 144. Further details regarding the statistical analysis component 140 will be described later.

[0021] Figure 3 is a flowchart illustrating an exemplary process 300 that may be performed by a point tracking component 110 to track a target body part in video data 104 according to an embodiment of the present disclosure. In step 302, the point tracking component 110 may identify a target body part by processing the video data 104 using a machine learning model. In step 304, the point tracking component 110 may generate a heatmap of the target body part based on processing the video data 104 using a machine learning model. The point tracking component 110 may estimate the two-dimensional pixel coordinates in which the target body part appears within the video frames of the video data 104 by using a machine learning model. The point tracking component 110 may generate a heatmap that estimates the position of one target body part for each video frame. For example, the point tracking component 110 may generate a first heatmap in which each cell may correspond to a pixel in the video frame and may represent the likelihood that a first target body part (e.g., a right forelimb) is located at each pixel. Continuing this example, the point tracking component 110 may generate a second heatmap, in which case each cell may represent the likelihood that a second target body part (e.g., the left forelimb) is located at each pixel. In step 306, the point tracking component 110 may use the generated heatmap to determine the point data 112. The heatmap cell with the highest / maximum value may identify the pixel coordinates in which each target body part is located within the video frame.

[0022] The point tracking component 110 may be configured to identify the two-dimensional coordinates of a set of subject body parts, which are identified as keypoints, in an image or video. In some embodiments, the set of subject body parts may be predetermined and based on which keypoints are visual features such as ears or noses, and / or based on which keypoints capture important information for analyzing the subject's gait and posture, such as limb joints or forelimbs. In an exemplary embodiment, the set of subject body parts may include 12 keypoints. In other embodiments, the set of subject body parts may include fewer than 12 or more than 12 keypoints. In an exemplary embodiment, the set of subject body parts may include the nose, left ear, right ear, base of the neck, left forelimb, right forelimb, mid-spine, left hindlimb, right hindlimb, base of the tail, mid-tail, and tail tip (as shown in Figure 7B).

[0023] The point tracking component 110 may implement one or more pose estimation techniques. The point tracking component 110 may include one or more machine learning models configured to process the video data 104. In some embodiments, one or more machine learning models may be neural networks such as deep neural networks, deep convolutional neural networks, or iterative neural networks. In other embodiments, one or more machine learning models may be other types of models other than neural networks. The point tracking component 110 may be configured to determine the point data 112 with high accuracy and precision, since the index data 122, 132 may be sensitive to errors in the point data 112. The point tracking component 110 may implement an architecture that maintains high-resolution features throughout the machine learning model stack, thereby preserving spatial accuracy. In some embodiments, the architecture of the point tracking component 110 may include one or more transposed convolutions to produce a match between the resolution of the heatmap output and the resolution of the video data 104. The point tracking component 110 may be configured to determine point data 112 at near real-time speed and may run on a high-performance GPU. The point tracking component 110 may be configured to allow for easy modification and expansion. In some embodiments, the point tracker component 110 may be configured to generate inference at a fixed scale rather than processing at multiple scales, thereby saving computational resources and time.

[0024] In some embodiments, the video data 104 may track the movement of a single object, and the point tracking component 110 may be configured not to perform any object detection techniques / algorithms. In other embodiments, the video data 104 may track the movement of two or more objects, and the point tracking component 110 may be configured to identify one object from another object in the video data 104 by performing object detection techniques.

[0025] In some embodiments, the point tracking component 110 may generate multiple heatmaps, each representing an inference about the location of a single keypoint representing a single body part within a frame of the video data 104. For example, the video data 104 may have 480x480 frames, and the point tracking component 110 may generate 12 480x480 heatmaps. The maximum value within each heatmap may represent the highest confidence position for each keypoint. In some embodiments, the point tracking component 110 may obtain each maximum value for the 12 heatmaps and output it as point data 112, which may contain 12 (x,y) coordinates.

[0026] In some embodiments, the point-tracking component 110 may be trained with respect to a loss function such as a Gaussian distribution centered on each keypoint. The output of the neural network of the point-tracking component 110 may be compared to a Gaussian distribution centered on the keypoints, and the loss may be calculated as the mean squared difference between each keypoint and the heatmap generated by the point-tracking component 110. In some embodiments, the point-tracking component 110 may be trained using an optimization algorithm, such as a stochastic gradient descent optimization algorithm. The point-tracking component 110 may be trained using training video data of subjects having various physical characteristics such as different fur colors, different body lengths, and different body sizes.

[0027] The point tracking component 110 may estimate a given key point with varying levels of confidence depending on the position of the body part on the target body. For example, the position of the hind limb is From above Because the forelimbs may be more obscured than the hindlimbs from the viewpoint, their position may be estimated with higher confidence than that of the forelimbs. In another example, visually prominent body parts, such as the mid-spine, may be less reliable because it may be more difficult for the point-tracking component 110 to accurately locate them.

[0028] Herein, we refer to the gait analysis component 120 and the posture analysis component 130. As used herein, gait indicators may refer to indicators derived from the movement of the subject's feet. Gait indicators may include, but are not limited to, step width, step length, stride length, velocity, angular velocity, and limb load coefficient. As used herein, posture indicators may refer to indicators derived from the movement of the subject's whole body. In some embodiments, posture indicators may be based on the movement of the subject's nose and tail. Posture indicators may include, but are not limited to, lateral displacement of the nose, lateral displacement of the tail base, lateral displacement of the tail tip, lateral displacement phase offset of the nose, displacement phase offset of the tail base, and displacement phase offset of the tail tip.

[0029] The gait analysis component 120 and the posture analysis component 130 may determine one or more gait and posture indicators for each stride. The system 150 may determine the stride interval represented within the video frame of the video data 104. In some embodiments, the stride interval may be based on the stance phase and the swing phase. Figure 4 is a flowchart illustrating an exemplary process 400 that may be performed by the gait analysis component 120 and / or the posture analysis component 130 to determine a set of multiple stride intervals for analysis.

[0030] In exemplary embodiments, the approach for detecting stride intervals is based on the periodic structure of walking. During the stride cycle, each foot may have a stance phase and a swing phase. During the stance phase, the foot supports the subject's weight and is in static contact with the ground. During the swing phase, the foot moves forward and does not support the subject's weight. Hereinafter, the transition from the stance phase to the swing phase is referred to as the toe-off event, and the transition from the swing phase to the stance phase is referred to as the foot-contact event. Figures 8A-C show examples of the stance phase, an example of the swing phase, an example of a toe-off event, and an example of a foot-contact event.

[0031] In step 402, the system 150 may determine a number of stance and swing phases represented within a given period. In an exemplary embodiment, the stance and swing phases may be determined with respect to the hind limb of the subject. The system 150 may also calculate the velocity of the forelimb and infer that the foot is in the stance phase when the velocity falls below a threshold, and that the foot is in the swing phase when the velocity exceeds the threshold. In step 404, the system 150 may determine that a foot contact event transitioning from the swing phase to the stance phase occurs in the video frame. In step 406, the system 150 may determine the stride intervals represented within a period. The stride intervals may span multiple video frames of the video data 104. For example, the system 150 may determine that a 10-second period has five stride intervals, and that one of the five stride intervals is represented within five consecutive video frames of the video data 104. In an exemplary embodiment, the left hind limb ground contact event may be defined as an event that separates / distinguishes stride intervals. In another exemplary embodiment, the right hind limb ground contact event may be defined as an event that separates / distinguishes stride intervals. In yet another exemplary embodiment, separated stride intervals may be defined by using a combination of the left hind limb ground contact event and the right hind limb ground contact event. In some other embodiments, system 150 may determine the stance phase and swing phase with respect to the forelimbs, calculate the foot velocity based on the forelimbs, and further distinguish stride intervals based on the foot ground contact events of the right and / or left forelimbs. In some other embodiments, the transition from the stance phase to the swing phase, i.e., the toe-off event, may be used to separate / distinguish stride intervals.

[0032] In some embodiments, the keypoint inference quality for the forelimbs (determined by the point tracking component 110) may be unreliable in some cases, making it preferable to determine the stride interval based on the hindlimb ground contact event rather than the forelimb ground contact event. View from aboveThis may be a result of the difficulty in accurately determining the position of the forelimbs, as they are often more hidden than the hindlimbs.

[0033] In step 408, the system 150 may determine which stride intervals were used to determine the index data 122, 132 by filtering the determined stride intervals. In some embodiments, such filtering may remove incorrect or unreliable stride intervals. In some embodiments, criteria for removing stride intervals include, but are not limited to, unreliable keypoint estimations, physiologically unrealistic keypoint estimations, missing right hind limb ground contact events, and insufficient whole-body velocity of the subject (e.g., velocity less than 10 cm / sec).

[0034] In some embodiments, stride interval filtering may be based on the confidence level used to determine the keypoints used to determine the stride intervals. For example, stride intervals determined at a confidence level below a threshold may be removed from a set of stride intervals used to determine index data 122, 132. In some embodiments, the first and last strides are removed within a continuous sequence of strides to avoid the start and stop movements adding noise to the data to be analyzed. For example, in a sequence of seven strides, a maximum of five strides would be used for analysis.

[0035] After determining the stride interval represented in the video data 104, the system 150 may determine gait indices and postural indices. Figure 5 is a flowchart of an exemplary process 500 that may be performed by the gait analysis component 120 to determine the gait indices of interest according to an embodiment of the present disclosure. The steps in process 500 may be performed in any optional order shown in Figure 5. In other embodiments, the steps in process 500 may be performed in a different order. In yet another embodiment, the steps in process 500 may be performed in parallel.

[0036] In step 502, the gait analysis component 120 may use point data 112 to determine the step length with respect to the stride interval that was determined to be analyzed in step 408 shown in Figure 4. The gait analysis component 120 may determine the step length for each stride interval during that period. In some embodiments, the point data 112 may relate to keypoints representing the left hind limb, left forelimb, right hind limb, and right forelimb. In some embodiments, the step length may be the distance between the left forelimb and the right hind limb with respect to the stride interval. In some embodiments, the step length may be the distance between the right forelimb and the left hind limb with respect to the stride interval. In some embodiments, the step length may be the distance the right hind limb moves beyond the immediate contact of the left hind limb.

[0037] In step 504, the gait analysis component 120 may use the point data 112 to determine the stride length for the stride intervals that were determined to be analyzed in step 408. The gait analysis component 120 may determine the stride length for each stride interval during that period. In some embodiments, the point data 112 may relate to keypoints representing the left hind limb, left forelimb, right hind limb, and right forelimb. In some embodiments, the stride length may be the distance between the left forelimb and left hind limb for each stride interval. In some embodiments, the stride length may be the distance between the right forelimb and right hind limb. In some embodiments, the stride length may be the total distance the left hind limb travels with respect to the stride from the toe-off event to the foot-contact event.

[0038] In step 506, the gait analysis component 120 may use the point data 112 to determine the step width for the stride intervals that were determined to be analyzed in step 408. The gait analysis component 120 may determine the step width for each stride interval during that period. In some embodiments, the point data 112 may relate to keypoints representing the left hind limb, left forelimb, right hind limb, and right forelimb. In some embodiments, the step width is the distance between the left forelimb and the right forelimb. In some embodiments, the step width is the distance between the left hind limb and the right hind limb. In some embodiments, the step width is the average of the lateral distances separating the hind limbs. This may be calculated as the length of the shortest line segment connecting the ground contact position of the right hind limb and the straight line connecting the toe-off position and subsequent foot contact position of the left hind limb.

[0039] In step 508, the gait analysis component 120 may use the point data 112 to determine the foot velocity for the stride intervals that were determined to be analyzed in step 408. The gait analysis component 120 may determine the foot velocity for each stride interval during that period. In some embodiments, the point data 112 may relate to keypoints representing the left hind limb, right hind limb, left forelimb, and right forelimb. In some embodiments, the foot velocity may be the velocity of a single foot at a given stride interval. In some embodiments, the foot velocity may be the velocity of the subject, or it may be based on the tail base of the subject.

[0040] In step 510, the gait analysis component 120 may use the point data 112 to determine the stride velocity with respect to the stride interval that was determined to be analyzed in step 408. The gait analysis component 120 may determine the stride velocity for each stride interval during that period. In some embodiments, the point data 112 may relate to keypoints representing the tail root. In some embodiments, the stride velocity may be determined by determining a set of velocity data about the subject based on the movement of the subject's tail root at a set of video frames representing a stride interval. Each velocity data in the set of velocity data may correspond to one frame in the set of video frames. The stride velocity may be calculated by averaging (or combining in other ways) the set of velocity data.

[0041] In step 512, the gait analysis component 120 may use the point data 112 to determine the limb load coefficients for the stride intervals that were determined to be analyzed in step 408. The gait analysis component 120 may determine the limb load coefficients for each stride interval during that period. In some embodiments, the point data 112 may relate to keypoints representing the right and left hind limbs. In some embodiments, the limb load coefficient for a stride interval may be the average of a first load coefficient and a second load coefficient. The gait analysis component 120 may determine the first stance time, which represents the amount of time the right hind limb is in contact with the ground during a stride interval, and then determine the first load coefficient based on the first stance time and the length of the stride interval. The gait analysis component 120 may determine the second stance time, which represents the amount of time the left hind limb is in contact with the ground during a stride interval, and then determine the second load coefficient based on the second stance time and the length of the stride interval. In other embodiments, the limb load coefficients may be based on the stance time and load coefficient of the forelimbs.

[0042] In step 514, the gait analysis component 120 may use the point data 112 to determine the angular velocity with respect to the stride interval that was determined to be analyzed in step 408. The gait analysis component 120 may determine the angular velocity for each stride interval during that period. In some embodiments, the point data 112 may relate to keypoints representing the tail root and the neck root. The gait analysis component 120 may determine a set of vectors connecting the tail root and the neck root, in which case each vector in the set corresponds to one frame of a set of frames with respect to the stride interval. The gait analysis component 120 may determine the angular velocity based on a set of vectors. The vectors may represent angles of interest, and the first derivative of the angle value may be the angular velocity with respect to the frame. In some embodiments, the gait analysis component 120 may determine the stride angular velocity by averaging the angular velocities with respect to the frames with respect to the stride interval.

[0043] Figure 6 is a flowchart illustrating an exemplary process 600 that may be performed by the posture analysis component 130 to determine a subject's posture index according to an embodiment of the present disclosure. At a high level, the posture analysis component 130 may determine the lateral displacements of the subject's nose, tail tip, and tail base with respect to individual stride intervals. Based on the lateral displacements of the nose, tail tip, and tail base, the posture analysis component 130 may determine the displacement phase offset of each subject's body part. In each case, the steps of the process 600 may be performed in an order different from that shown in Figure 6. For example, the posture analysis component 130 may determine the lateral displacement of the nose and the displacement phase offset of the nose after, or in parallel with, determining the lateral displacement of the tail tip and the displacement phase offset of the tail tip.

[0044] To determine lateral displacement, the posture analysis component 130 may first use the point data 112 in step 602 to determine a displacement vector for the stride interval to be analyzed in step 408. The posture analysis component 130 may determine a displacement vector for each stride interval during that period. In some embodiments, the point data 112 may relate to keypoints representing the mid-spine of the subject. The stride interval may span multiple video frames. In some embodiments, the displacement vector may be a vector connecting the mid-spine in the first video frame of the stride interval to the mid-spine in the last video frame of the stride interval.

[0045] In step 604, the posture analysis component 130 may use the point data 112 and the displacement vector (from step 602) to determine the lateral displacement of the subject's nose with respect to the stride interval. The posture analysis component 130 may determine the lateral displacement of the nose with respect to each stride interval during that period. In some embodiments, the point data 112 may relate to keypoints representing the mid-spine and nose of the subject. In some embodiments, the posture analysis component 130 may determine a set of multiple lateral displacements of the nose, in which case each lateral displacement of the nose may correspond to one video frame of the stride interval. The lateral displacement may be the vertical distance of the nose from the displacement vector with respect to the stride interval within each video frame. In some embodiments, the posture analysis component 130 may subtract the minimum distance from the maximum distance and divide it by the body length of the subject so that the displacement measured on a larger subject is equivalent to the displacement measured on a smaller subject.

[0046] In step 606, the posture analysis component 130 may determine the displacement phase offset of the nose using a set of lateral displacements of the nose with respect to the stride interval. The posture analysis component 130 may generate a smooth curve of the lateral displacement of the nose with respect to the stride interval by performing interpolation using a set of lateral displacements of the nose, and then use the smooth curve of the lateral displacement of the nose to determine at what point in the stride interval the maximum displacement of the nose occurs. The posture analysis component 130 may determine a percentage stride position that represents the percentage of the stride interval completed when the maximum displacement of the nose occurs. In some embodiments, the posture analysis component 130 may perform cubic spline interpolation to generate a smooth curve of displacement, and for cubic interpolation, the maximum displacement may occur at a point between video frames.

[0047] In step 608, the posture analysis component 130 may use the point data 112 and the displacement vector (from step 602) to determine the lateral displacement of the tail root of the subject with respect to the stride interval. The posture analysis component 130 may determine the lateral displacement of the tail root with respect to each stride interval during that period. In some embodiments, the point data 112 may relate to keypoints representing the mid-spine and tail root of the subject. In some embodiments, the posture analysis component 130 may determine a set of multiple lateral displacements of the tail root, in which case each lateral displacement of the tail root may correspond to one video frame of the stride interval. The lateral displacement may be the vertical distance of the tail root from the displacement vector with respect to the stride interval within each video frame. In some embodiments, the posture analysis component 130 may subtract the minimum distance from the maximum distance and divide it by the body length of the subject so that the displacement measured on a larger subject is equivalent to the displacement measured on a smaller subject.

[0048] In step 610, the posture analysis component 130 may determine the tail root displacement phase offset using a set of lateral displacements of the tail root with respect to the stride interval. The posture analysis component 130 may generate a smooth curve of the lateral displacement of the tail root with respect to the stride interval by performing interpolation using a set of lateral displacements of the tail root, and then use the smooth curve of the lateral displacement of the tail root to determine at what point in the stride interval the maximum displacement of the nose occurred. The posture analysis component 130 may determine a percentage stride position representing the percentage of the stride interval completed when the maximum displacement of the tail root occurred. In some embodiments, the posture analysis component 130 may perform cubic spline interpolation to generate a smooth curve of displacement, and for cubic interpolation, the maximum displacement may occur at a point between video frames.

[0049] In step 612, the posture analysis component 130 may use the point data 112 and the displacement vector (from step 602) to determine the lateral displacement of the tail tip of the subject with respect to the stride interval. The posture analysis component 130 may determine the lateral displacement of the tail tip with respect to each stride interval during that period. In some embodiments, the point data 112 may relate to keypoints representing the mid-spine and tail tip of the subject. In some embodiments, the posture analysis component 130 may determine a set of multiple lateral displacements of the tail tip, in which case each lateral displacement of the tail tip may correspond to one video frame of the stride interval. The lateral displacement may be the vertical distance of the tail tip from the displacement vector with respect to the stride interval within each video frame. In some embodiments, the posture analysis component 130 may subtract the minimum distance from the maximum distance and divide it by the body length of the subject so that the displacement measured on a larger subject is equivalent to the displacement measured on a smaller subject.

[0050] In step 614, the posture analysis component 130 may determine the tail root displacement phase offset using a set of lateral displacements of the tail tip with respect to the stride interval. The posture analysis component 130 may generate a smooth curve of the lateral displacement of the tail tip with respect to the stride interval by performing interpolation using a set of lateral displacements of the tail tip, and then use the smooth curve of the lateral displacement of the tail tip to determine at what point in the stride interval the maximum nose displacement occurred. The posture analysis component 130 may determine a percentage stride position representing the percentage of the stride interval completed when the maximum tail tip displacement occurred. In some embodiments, the posture analysis component 130 may perform cubic spline interpolation to generate a smooth curve of displacement, and for cubic interpolation, the maximum displacement may occur at a point between video frames.

[0051] Referring to the statistical analysis component 140 of system 150, the statistical analysis component 140 can accept index data 122 (determined by the gait analysis component 120) and index data 132 (determined by the posture analysis component 130) as inputs. In some embodiments of the present invention, the statistical analysis component 140 may accept only index data 122 as input, based on the fact that the system is configured to process only gait index data. In other embodiments, the statistical analysis component 140 may accept only index data 132 as input, based on the fact that the system is configured to process only posture index data.

[0052] The size and speed of an object can influence its gait and / or posture. For example, a faster-moving object will exhibit a different gait compared to a slower-moving object. As a further example, a larger object will exhibit a different gait compared to a smaller object. However, in some cases, differences in stride speed (differences compared to the gait of a control object) can be a definitive feature of gait and posture changes caused by genetic or pharmacological perturbations. System 150 collects multiple repeated measures for each object (via video data 104 and through objects in an open area), and each object has a different number of strides, resulting in unbalanced data. Averaging over repeated strides yields a single average value for each object, but this can be misleading because it removes variability and introduces false confidence. At the same time, classical linear models fail to distinguish between stable within-object variability and between-object fluctuations, which can lead to bias in statistical analysis. To address these issues, the statistical analysis component 140, in some embodiments, employs a linear mixed model (LMM) to separate intra-subject variability from genotype-based inter-subject variability. In some embodiments, the statistical analysis component 140 may capture main effects such as subject size, genotype, and age, and additionally, random effects relating to intra-subject variability. The technique of the present invention collects multiple repeated measures results at different ages of subjects, resulting in a nested hierarchical data structure. Examples of exemplary statistical models implemented in the statistical analysis component 140 are shown below as models M1, M2, and M3. These models follow standard LMM notation, where (genotype, body length, velocity, and age at study) represent fixed effects and (subject ID / age at study) (where age at study is nested within the subject) represents random effects. M1: Phenotype ~ Genotype + Test Age + Body Length + (1|Mouse ID / Test Age) M2: Phenotype ~ Genotype + Test Age + Speed ​​+ (1|Mouse ID / Test Age) M3: Phenotype ~ Genotype + Test Age + Speed ​​+ Body Length + (1|Mouse ID / Test Age)

[0053] Model M1 takes age and body length as inputs, Model M2 takes age and speed as inputs, and Model M3 takes age, speed, and body length as inputs. In some embodiments, the model of the statistical analysis component 140 does not include the sex of the subject as an effect because sex may have a high correlation with the body length / size of the subject. In other embodiments, the model of the statistical analysis component 140 may take the sex of the subject as input. Using point data 112 (determined by the point tracking component 110) makes it possible to determine the size and speed of the subject with respect to these models. Therefore, no additional measurements are required for these variables with respect to the models.

[0054] One or more of the data included in the index data 122, 132 may be circular variables (e.g., stride length, angular velocity, etc.), and the statistical analysis component 140 may implement a function of the linear variables using a circular-linear regression model. Linear variables such as body length and velocity may be included as covariates in the model. In some embodiments, the statistical analysis component 140 may implement a multivariate outlier detection algorithm at the individual subject level to identify subjects with injury and developmental effects.

[0055] In some embodiments, the statistical analysis component 140 may also implement linear discriminant analysis, which outputs difference data 148 by processing the index data 122 and 132 with respect to the control data 144. Linear discriminant analysis can quantitatively distinguish between the target gait indicators and / or posture indicators and the control target gait indicators and / or posture indicators.

[0056] video feed stitching In some embodiments, video data 104 may be generated using multiple video feeds that capture the movement of the object from multiple different angles / viewpoints. Video data 104 may be generated by stitching / combining a first video in a plan view of the object and a second video in a side view of the object. The first video may be captured using a first image acquisition device (e.g., device 101a), and the second video may be captured using a second image acquisition device (e.g., device 101b). Other views of the object may include a right side view, a left side view, From above This may include field of view, bottom view, front view, rear view, and other views. By combining videos from these different views to generate video data 104, a comprehensive / broad view of the object's movement may be provided, which may allow an automated phenotypic classification system to perform a more accurate and / or efficient classification of the object's behavior. In some embodiments, by combining videos from different views, the entire field of view may be provided. From above A wide field of view may be provided with a short focal length while maintaining the viewpoint. In some embodiments, video data 104 may be generated by processing multiple videos from different viewpoints using one or more ML models (e.g., neural networks). In some embodiments, the system may generate 3D video data using 2D video / images.

[0057] In some embodiments, video captured by multiple image acquisition devices 101 may be synchronized using various techniques. For example, the multiple image acquisition devices 101 may be synchronized to a central clock system or controlled by a master node. Synchronization of multiple video feeds may involve the use of various hardware and software, such as adapters, multiplexers, USB connections between image acquisition devices, wireless or wired connections to a network 199, and software to control the devices (e.g., MotionEyeOS).

[0058] In an exemplary embodiment, the image acquisition device 101 may be an ultra-wide-angle lens (i.e., a fisheye lens) capable of achieving an extremely wide field of view while generating strong visual distortion intended to create a wide panoramic or hemispherical image. In an exemplary implementation, the system for acquiring video for video data 104 consists of four fisheye lens cameras connected to four single-board computing devices (e.g., Raspberry Pi), View from above The system may include an additional image acquisition device for capturing the image. The system may use various techniques to synchronize these components. One technique involves pixel / spatial interpolation, for example, if the point of interest (e.g., a body part on an object) is located at (x, y), the system will synchronize along the x and y axes with respect to time. View from above Identify the position within the video. For example, pixel interpolation along the x-axis can be calculated by a single-board computing device according to the following formula. (Pi offset ΔX / Pi offset ΔT) × ( View from above Offset ΔT) + Initial point (x)

[0059] Next, this formula may be used to calculate the position of the point of interest with respect to the y-axis. In some embodiments, to address lens distortion during video calibration, padding may be added to one or more video feeds (instead of scaling the video feeds).

[0060] subject Some aspects of the present invention involve the use of gait analysis methods and posture analysis methods on a subject. As used herein, the term “subject” may refer to humans, non-human primates, cattle, horses, pigs, sheep, goats, dogs, cats, birds, rodents, or other suitable vertebrates or invertebrates. In certain embodiments of the present invention, the subject is a mammal, and in certain embodiments of the present invention, the subject is a human. In some embodiments, the subject used in the methods of the present invention is a rodent, including but not limited to mice, rats, gerbils, hamsters, etc. In some embodiments of the present invention, the subject is a normal, healthy subject, and in some embodiments, the subject is known to have a disease or condition, is at risk of having a disease or condition, or is suspected of having a disease or condition. In certain embodiments of the present invention, the subject is an animal model relating to a disease or condition. For example, but not intended to be limiting, in some embodiments of the present invention, the subject is a mouse forming an animal model relating to autism.

[0061] As a non-limiting example, the subjects evaluated by the methods and systems of the present invention may be subjects that constitute animal models of pathological conditions, such as models for one or more of the following: mental disorders, neurodegenerative diseases, neuromuscular diseases, autism spectrum disorders, schizophrenia, bipolar disorder, Alzheimer's disease, Rett syndrome, ALS, and Down syndrome.

[0062] In some embodiments of the methods of the present invention, the subject is a wild-type subject. As used herein, the term “wild-type” means the phenotype and / or genotype relating to the typical form of the species as it occurs in nature. In certain embodiments of the present invention, the subject is a non-wild-type subject, for example, a subject having one or more genetic modifications compared to the wild-type genotype and / or phenotype relating to the species of the subject. In some examples, the difference in the genotype / phenotype of the subject compared to the wild type is due to hereditary (germline) variants or acquired (somatic) variants. Factors that may result in a subject exhibiting one or more somatic variants include, but are not limited to, environmental factors, toxins, ultraviolet radiation, spontaneous errors occurring in cell division, teratogenic events such as radiation, maternal infection, or chemicals.

[0063] In certain embodiments of the methods of the present invention, the subject is a genetically modified organism, also referred to as the engineered subject. The engineered subject may include pre-selected and / or intentional genetic modifications, and thus exhibit one or more genotypes and / or phenotypic traits that differ from those in the unengineered subject. In some embodiments of the present invention, engineered subjects exhibiting genotype and / or phenotypic differences compared to the unengineered subject of the same species can be produced by using conventional genetic engineering techniques. As a non-limiting example, the phenotype of a genetically engineered mouse can be evaluated by using a genetically engineered mouse in which functional gene products are present at deficient or reduced levels, and by using the methods or systems of the present invention, and the results can be compared with results obtained from a control (control result).

[0064] In some embodiments of the present invention, subjects may be monitored using the gait level determination method or system of the present invention to detect the presence or absence of activity impairments or pathologies. In certain embodiments of the present invention, the response of a test subject to an activity and / or exercise pathology may be evaluated by using a test subject that constitutes an animal model of the exercise and / or exercise pathology. In addition, a test subject that constitutes an animal model of the exercise and / or exercise pathology may be administered a candidate therapeutic agent or method and monitored using the gait monitoring method and / or system of the present invention, and the results may be used to determine the effectiveness of a candidate therapeutic agent for treating the pathology. The terms “activity” and “behavior” may be used interchangeably herein.

[0065] As described elsewhere in this specification, the trained models of the present invention may be configured to detect the behavior of an object regardless of the object's physical characteristics. In some embodiments of the present invention, one or more of the object's physical characteristics may be pre-identified characteristics. For example, though not intended to be limiting, pre-identified physical characteristics may be one or more of body type, build, coat color, sex, age, and disease or pathological phenotype.

[0066] Testing and screening of control and candidate compounds Results obtained with respect to a subject using the method or system of the present invention can be compared with results with respect to a control. The method of the present invention can also be used to evaluate phenotypic differences between a subject and a control. Thus, some aspects of the present invention provide a method for determining whether or not there is a change in activity in a subject compared to a control. Some embodiments of the present invention include identifying phenotypic features related to a disease or pathological condition by using the gait analysis and posture analysis of the present invention.

[0067] Results obtained using the methods or systems of the present invention can be advantageously compared with controls. In some embodiments of the present invention, one or more subjects can be evaluated using an automated walking analysis method, and then the subjects can be retested after being administered a candidate therapeutic compound. The term “test” subject may be used herein in relation to a subject evaluated using the methods or systems of the present invention. In certain embodiments of the present invention, results obtained using an automated walking analysis method to evaluate a subject are compared with results obtained from automated walking analysis methods performed on other test subjects. In some embodiments of the present invention, results of a test subject are compared with results from automated walking analysis methods performed on the test subject at different times. In some embodiments of the present invention, results obtained using an automated walking analysis method to evaluate a test subject are compared with control results.

[0068] The control value may be a value obtained from testing multiple subjects using the gait analysis method of the present invention. As used herein, the control result may be a predetermined value and can take various forms. It may be a single cutoff value, such as a median or mean. It may be established based on a comparison group, for example, a comparison group of subjects evaluated using the automated gait analysis method of the present invention under similar conditions to the test subject, in which case the test subject is administered the candidate therapeutic agent, and the comparison group is not exposed to the candidate therapeutic agent. Another example of a comparison group may include subjects known to have a disease or condition and a group not having the disease or condition. Another comparison group may include subjects with a family history of the disease or condition and subjects from a group not having such a family history. The predetermined value may be set, for example, so that the tested population is grouped equally (or unequally) based on the test results. Those skilled in the art can select appropriate control groups and values ​​for use in the comparison method of the present invention. Non-limiting examples of the types of candidate compounds include chemical substances, nucleic acids, proteins, small molecules, antibodies, etc.

[0069] Subjects evaluated using the automated walking analysis method or system of the present invention may be monitored for any changes occurring between test conditions and control conditions. Non-limiting examples of changes in a subject include, but are not limited to, one or more changes such as movement frequency or response to external stimuli. The methods and systems of the present invention can be used on test subjects to evaluate the effects of a disease or condition, and to evaluate the effectiveness of a candidate therapeutic agent for treating a disease or condition. As a non-limiting example of using the method of the present invention to evaluate the presence or absence of changes in a test subject as a means of determining the effectiveness of a candidate therapeutic agent, a test subject known to constitute an animal model for a disease such as autism is evaluated using the automated walking analysis method of the present invention. The test subject is administered the candidate therapeutic agent and evaluated again using the automated walking analysis method. The presence or absence of changes in the test subject's results indicates whether or not the candidate therapeutic agent is effective for the autism of the test subject. Diseases and conditions that can be evaluated using the gait analysis method of the present invention include, but are not limited to, ALS, autism, Down syndrome, Rett syndrome, bipolar disorder, dementia, depression, attention deficit hyperactivity disorder, anxiety disorders, developmental disorders, sleep disorders, Alzheimer's disease, Parkinson's disease, physical injuries, and the like.

[0070] In some embodiments of the present invention, it will be understood that a test subject can serve as its own control by being evaluated two or more times, for example, using the automated gait analysis method of the present invention, and comparing the results obtained from two or more different evaluations. The methods and systems of the present invention can be used to evaluate the progression or regression of a disease or condition in a subject by identifying and comparing changes over time in the gait characteristics of the subject using two or more evaluations of the subject using embodiments of the methods or systems of the present invention.

[0071] Diseases and Disabilities The methods and systems of the present invention can be used to evaluate the activity and / or behavior of subjects known to have a disease or condition, suspected to have a disease or condition, or at risk of having a disease or condition. In some embodiments, the disease and / or condition is associated with an abnormal level of activity or behavior. In non-limiting examples, a test subject that may have anxiety, or a test subject that may form an animal model of anxiety, may have one or more anxiety-related activities or behaviors that can be detected using embodiments of the methods of the present invention. The results of evaluating a test subject can be compared with control results in evaluations such as, for example, control subjects that do not have anxiety, control subjects that do not form an animal model of anxiety, or control standards obtained from multiple subjects that do not have a condition. Differences can be compared between the results of the test subject and the control results. Some embodiments of the methods of the present invention can be used to identify subjects that have a disease or condition associated with abnormal activity and / or behavior.

[0072] The onset, progression, and / or regression of diseases or conditions associated with abnormal activity and / or behavior can also be evaluated and tracked using embodiments of the method of the present invention. For example, in certain embodiments of the method of the present invention, two, three, four, five, six, seven, or more evaluations of the activity and / or behavior of a subject are performed at different times. Comparison of two or more evaluation results performed at different times can indicate differences in the activity and / or behavior of the subject. An increase in the determined activity level or type of activity may indicate the onset and / or progression of a disease or condition associated with the evaluated activity in the subject. A decrease in the determined activity level or type of activity may indicate regression of a disease or condition associated with the evaluated activity in the subject. A determination that the activity has ceased in the subject may indicate that the disease or condition associated with the evaluated activity has ceased within the subject.

[0073] Certain embodiments of the method of the present invention can be used to evaluate the effectiveness of a therapy for treating a disease or condition associated with abnormal activity and / or behavior. For example, a test subject may be administered a candidate therapy and the method of the present invention, which are used to determine whether or not there is a change in disease or condition-related activity within the subject. A reduction in abnormal activity after administration of the candidate therapy may indicate the effectiveness of the candidate therapy for the disease or condition.

[0074] As described elsewhere in this specification, the gait analysis method of the present invention may be used to evaluate diseases or conditions in subjects, as well as to evaluate animal models of diseases and conditions. Numerous different animal models of diseases and conditions are known in the art, including, but not limited to, numerous mouse models. Subjects evaluated using the system and / or method of the present invention may be, but are not limited to, subjects that constitute animal models of diseases or conditions, such as models of diseases or conditions, including, but not limited to, neurodegenerative disorders, neuromuscular disorders, neuropsychiatric disorders, ALS, autism, Down syndrome, Rett syndrome, bipolar disorder, dementia, depression, attention deficit hyperactivity disorder, anxiety disorders, developmental disorders, sleep disorders, Alzheimer's disease, Parkinson's disease, physical injury, etc. Additional models of diseases and disorders that can be evaluated using the methods and / or systems of the present invention are known in the art, and should be seen, for example, Barrot M. Neuroscience 2012;211:39-50, Graham, DM, Lab Anim (NY) 2016;45:99-101, Sewell, RDE, Ann Transl Med 2018;6:S42.2019 / 01 / 08, and Jourdan, D., et al., Pharmacol Res 2001;43:103-110, the contents of which are incorporated herein by reference in their entirety.

[0075] In addition to testing subjects with known diseases or disorders, the method of the present invention can also be used to evaluate novel gene variants, such as artificial organisms. Therefore, the method of the present invention can be used to evaluate artificial organisms with respect to one or more characteristics of a disease or pathological condition. In this way, new biological strains, such as new mouse strains, can be evaluated, and the results can be used to determine whether the new strain is an animal model for a disease or disorder. [Examples]

[0076] Example 1. Model Development: Data Training, Testing, and Model Validation method Data training The labeled data consists of 8,910 480x480 grayscale frames, each containing one mouse in an open field, with 12 manually labeled pose keypoints per frame. The strains were selected from diverse mouse strains with different appearances, taking into account differences in coat color, body size, and obesity. Figure 8C shows a representative frame generated by the open field apparatus. The frames were generated from the same open field apparatus previously used to generate experimental data (Geuther, BQ et al., Commun Biol (2019) 2:1-11). Pose keypoint annotations were performed by multiple members of the Kumar lab. The frame images and keypoint annotations were saved together using the HDF5 format, which is used for training neural networks. The frame annotations were split into a training dataset (7,910 frames) and a validation dataset (1,000 frames) for the training.

[0077] Training a neural network The network was trained over 600 epochs, with validation performed at the end of each epoch. The training loss curve (Figure 8C) shows that the training loss converged quickly without the validation loss overfitting. Transfer learning (Weiss, K. et al., J Big Data (2016) 3:9, Tan, C. et al. 27th Inti Conference on Artificial Neural Networks (2018), 270-279, arXiv:1808.01974[cs.LG]) was used on the network to minimize labeling requirements and improve the model's generality. Initially, the ImageNet model provided by the authors of the HRNet paper (hrnet_w32-36af842e.pth) was used, with weights fixed up to the second stage during training. To further improve the network's generality, several data augmentation techniques, including rotation, flipping, scaling, brightness, contrast, and occlusion, were employed during training. The ADAM optimizer was used to train the network. The learning rate was initially 5 × 10⁻⁶. -4 It is set to 5x10 in the 400th epoch. -5 It was reduced to 5 x 10 in the 500th epoch. -6 It was reduced to this.

[0078] statistical analysis Regarding repeated measures, the following LMM model was considered.

number

[0079] The circular phase variables in Figure 14A were modeled as a function of the linear variables using a circular-linear regression model. The analysis of circular data is not straightforward, and statistical models developed for linear data are not applicable to circular data [Calinski, T. & Harabasz, Communications in Statistics - theory and Methods 3, 1-27 (1974)]. The circular response variable was assumed to be drawn from a von-Mises distribution with an unknown mean direction μ and concentration parameter κ. The mean direction parameter was related to the variable X through the following equation. Y i ~von Mises(μ i ,κ), μ i = μ + g(γ1X1 + … + γ p X p ), i = 1; …, n where g(u) = 2tan -1 (u) is a link function such that -π < g(u) < π for -∞ < u < ∞. The parameters μ; γ1...γ k and κ were estimated via the maximum likelihood method. The model was fitted using the circular package in R [Tibshirani, R. et al: Journal of the Royal Statistical Society: Series B (Statistical Methodology) 63, 411-423 (2001)].

[0080] Animal The animal strains used in the experiment are shown in Figures 14B - D.

[0081] Description and Results of the Experiment The approach to gait and posture analysis consists of multiple modular components. The toolkit is based on open-field data. From above The deep convolutional neural network was trained to perform pose estimation on video. This network provided 12 two-dimensional markers, or "keypoints," indicating the anatomical position of the mouse, for each video frame describing the mouse's pose at each point in time. Downstream components were also developed that could process the time series of poses and identify intervals representing individual strides. These strides formed the basis for almost all subsequent phenotypic and statistical analyses. Because pose information is obtained for each stride interval, this method makes it possible to extract several important gait indicators at the stride level (see Figure 14A for a list of indicators). This provides powerful capabilities for statistical analysis of stride indicators and allows for the aggregation of large amounts of data, thus enabling the provision of a consensus diagram regarding the mouse's gait structure.

[0082] Pose estimation Pose estimation was the basis for a method of quantifying and analyzing gait, as it involved identifying two-dimensional coordinates for a predefined set of keypoints within an image or video. Selected pose keypoints were either visually prominent, such as the ears or nose, or those capturing important information for understanding the pose, such as the joints of the limbs or forelimbs. Twelve keypoints were selected to capture the mouse's pose: the nose, left ear, right ear, base of the neck, left forelimb, right forelimb, mid-spine, left hindlimb, right hindlimb, base of the tail, mid-tail, and tail tip (Figure 7B).

[0083] Much effort has been expended to develop and improve pose estimation techniques for human poses (Moeslund, TB et al., Comput Vis Image Underst (2006) 104:90-126, Dang, Q. et al., Tsinghua Sci Technol (2019) 24:663-676). Conventional approaches to pose estimation relied on techniques such as the use of local body part detectors and skeletal joint modeling. These approaches had limitations in their ability to overcome complex factors such as complex configurations and occlusion of body parts. DeepPose addressed some of these shortcomings by developing a deep neural network for pose estimation (Toshev, A. & Szegedy, C., Proc IEEE Conf Comp Vis Pattern Recognit (2014), 1653-1660). DeepPose has demonstrated state-of-the-art performance improvements in pose estimation using several benchmarks. Following the release of DeepPose, the majority of successful studies on pose estimation have utilized deep convolutional neural network architectures. Several notable examples include DeeperCut (Insafutdinov, E. et al., European Conference on Computer Vision (2016), 34-50), Stacked Hourglass Networks (Newell, A. et al., European Conference on Computer Vision (2016), 483-499), and Deep High-Resolution Architecture (HRNet) (Sun, K. et al., Proc IEEE Conf Comp Vis Pattern Recognition (2019), 5693-5703). Several concepts used in high-performance pose estimation architectures developed for human pose estimation were considered in the development of the rodent pose estimation method included in the present invention.

[0084] Several important considerations underpinned the selection of the rodent pose estimation architecture. • High accuracy and precision in pose estimation: Since the gait estimation method is sensitive to errors in pose estimation, it is desirable to reduce such errors as much as possible. • Estimated speed: The ability to perform inference at real-time speed (30fps) or close to it on the latest high-end GPUs. • Architectural simplicity and versatility to facilitate modification and expansion. • Fixed-scale estimation: Since all images are at a fixed scale, approaches designed to work with multiple scales waste network capacity and estimation time. • Available open-source implementations. • Modular architecture to facilitate future upgrades.

[0085] Based on these criteria, the HRNet architecture (Sun, K. et al., Proc IEEE Conf Comp Vis Pattern Recognition (2019), 5693-5703) was selected for the network and modified for the experimental setup. The main differentiating feature of this architecture is that high-resolution features are maintained throughout the network stack, thereby maintaining spatial accuracy (Figure 7A). HRNet demonstrated very competitive performance in terms of both GPU efficiency and pause accuracy. The interface is also highly modular and is expected to allow for relatively easy network upgrades as needed. The smaller HRNet-W32 architecture was used instead of HRNet-W48 because it was shown to offer significant improvements in speed and memory with only a slight decrease in accuracy. The resolution of the heatmap output was matched to the resolution of the video input by adding two 5x5 transposed convolutions to the beginning of the network (Figure 7B). In all experiments, there was one mouse in the open field, so it was not necessary to rely on object detection for instantiation. Therefore, this step was omitted from the inference algorithm, which also resulted in a clear advantage in runtime performance. Instead of performing pose estimation after object detection, the pose of one mouse in each frame was inferred by using a full-resolution pose keypoint heatmap. This meant that 12 480x480 heatmaps were generated for each 480x480 frame of the video (one heatmap per keypoint). The maximum value within each heatmap represented the most confident position for each point. Thus, by obtaining the set of maximum points in each of the 12 heatmaps, 12 (x,y) coordinates were obtained.

[0086] To train the network, it was necessary to select a loss function and optimization algorithm. For the loss function, the approach used in the original HRNet description (Sun, K. et al., Proc IEEE Conf Comp Vis Pattern Recognition (2019), 5693-5703) was used. For each keypoint label, a two-dimensional Gaussian distribution centered on that keypoint was generated. The network output was then prepared using a Gaussian centered on the keypoints, and the loss was calculated as the mean squared difference between the labeled keypoint Gaussian and the heatmap generated by the network. The network was trained using the ADAM optimization algorithm, a variation of stochastic gradient descent (Kingma, DP & Ba, J. (2014) arXiv:1412.6980). Figure 7C shows that the validation loss converges rapidly. Labels representing diverse mouse appearances, including differences in fur color, body length, and obesity, were intentionally generated to ensure that the resulting network would function reliably across these differences. 8,910 frames across these diverse strains were manually labeled for training (see Methods). The resulting network was able to track dozens of mouse strains with different body sizes, body types, and coat colors (Geuther, BQ et al., Commun Biol (2019) 2:1-11).

[0087] Stride Inference The approach to detecting stride intervals was based on the periodic structure of gait described by Hildebrand (Figure 8A) (Hildebrand, MJ Mammalogy (1977) 58:131-156, Hildebrand, M. Bioscience (1989) 39:766). During the stride cycle, each foot has a stance phase and a swing phase (Lakes, EH & Allen, KDO Steoarthr Cartil (2016) 24:1837-1849). During the stance phase, the mouse's foot supports the mouse's weight and is in static contact with the ground. During the swing phase, the foot moves forward and does not support the mouse's weight. Following Hildebrand, the transition from the stance phase to the swing phase is called a toe-off event, and the transition from the swing phase to the stance phase is called a foot-contact event.

[0088] To calculate the stride interval, the stance and swing phases were determined for the hind limbs. Foot velocity was calculated, and it was estimated that the foot was in the stance phase when the velocity was below a threshold, and in the swing phase when the velocity was above the threshold (Figures 8C-8F). Subsequently, it was possible to determine that a foot contact event occurred in the transition frame from the swing phase to the stance phase (Figure 8C). Left hind limb contact was defined as an event that delimited the stride cycle. An example of the relationship between foot velocity and foot contact events for the hind limb is shown in Figure 8D. As shown in Figure 8E, clear high-amplitude oscillations were observed in the hind limbs, but not in the forelimbs. This difference in estimation quality between the forelimbs and hind limbs is... View from aboveThis is thought to be due to the fact that the forelimbs are obstructed more frequently than the hindlimbs, making it more difficult to accurately locate them. As shown in Figure 8G, a corresponding decrease in the confidence of inferences regarding the forelimbs was observed. For this reason, when deriving stride intervals, the forelimbs were excluded from consideration, and instead, attention was focused on the hindlimbs. In addition, extensive filtering was performed on strides to remove spurious or low-quality stride cycles from the dataset (Figure 8G). Criteria for removing strides included low confidence or physiologically unrealistic pose estimates, missing right hindlimb ground contact events, and insufficient overall mouse velocity, such as velocities less than 10 cm / sec. Figure 8G shows the confidence distribution for each keypoint. In the filtering method, a confidence threshold of 0.3 was used. Keypoints with very high confidence would be closer to 1.0. In a sequence of multiple strides, the first and last strides were always removed to avoid adding noise to the stride data from the start and stop movements (Figures 8C-D, showing A and D in tracks A and B). This means that in a sequence of seven strides, a maximum of five strides will be used for analysis. The confidence distribution of keypoints differs depending on the type of keypoint (Figure 8G). For example, for forelimbs, View from above Keypoints that were easily obstructed showed a lower confidence distribution compared to other keypoints. It was also observed that keypoints that were not visually prominent, such as the mid-spine, had lower confidence levels due to the difficulty in accurately locating them. Furthermore, instantaneous angular velocity was calculated, allowing for the determination of the direction of each stride (Figure 8F). Angular velocity was calculated by taking the first derivative of the angle formed by the line connecting the base of the mouse's tail and the base of its neck. Overall, this approach allowed for the identification of individual high-quality strides for mice in open field.

[0089] To verify that the quantification of gait was functioning properly, data from the commonly used inbred C57BL / 6NJ strain was analyzed. Using approximately one hour of open-field video per mouse, the stance and swing phase percentages were calculated from 15,667 strides from 31 animals. Data from the hind limbs were analyzed because they showed the largest amplitude oscillations during the stance and swing phases (Figures 8D, E). The data were stratified into nine angular velocity bins and eight velocity bins based on the tail base (Figures 8H and 8I, respectively). As expected, an increase in the stance phase percentage over one stride of the left hind limb was observed when the animals turned left. Conversely, when the animals turned right, the stance phase percentage of the right hind limb increased (Figure 8H). Next, by analyzing the strides within the central angular velocity bin (-20° / sec to 20° / sec), we determined whether the stance phase percentage during the stride cycle decreased as the stride velocity increased. It was confirmed that stance time decreased as stride length increased (Figure 8I). To quantify this relationship with velocity, a load factor was calculated for the hind limbs (Figure 8J). In summary, these methods allowed us to determine that From above It was concluded that stride length can be quantitatively and accurately extracted from open-field video at the viewpoint of the subject.

[0090] After the stride interval was determined, several stride indices, as defined in Figure 14A, could be derived by using frame poses in combination with stance and swing phase intervals. All relevant spatiotemporal indices could be extracted from the hind limbs and served as the primary data source for statistical analysis (Lakes, EH & Allen, KDOsteoarthr Cartil (2016) 24:1837-1849).

[0091] Estimation of full-body pose during walking cycle From aboveIn the video, the relative position of the spine could be determined using six key points (nose, base of neck, mid-spine, base of tail, mid-tail, and tail tip). These were used to extract whole-body posture during the stride cycle. Only three points (nose, base of tail, and tail tip) were used to capture lateral movement during the stride cycle (Figures 9A-C). These measurements were circular, and the phases of the nose and tail tip were reversed. For visualization, C57BL / 6J and NOR / LtJ, which have different tail tip phases during the stride cycle, were used. These phase plots could be extracted for each stride, and high sensitivity was obtained (Figures 9D-E). Since several hours of video were acquired across each strain, thousands of strides could be extracted, enabling a high level of sensitivity. By combining these in a single velocity and angular velocity bin, consensus stride phase plots could be determined for each animal and strain (Figures 9F-G). Furthermore, when these phase plots were compared between multiple strains, significant diversity in whole-body posture during the gait cycle was found.

[0092] Several metrics were related to the periodic lateral displacement observed within pose keypoints (Figures 9A-I). Lateral displacement measurements were defined as orthogonal offsets from the associated stride displacement vector. The displacement vector was defined as a line connecting the midpoint of the mouse's spine in the first frame of the stride to the midpoint of the mouse's spine in the last frame of the stride. This offset was calculated for each frame of the stride, and then cubic interpolation was performed to generate a smooth displacement curve. The phase offset of the displacement was defined as the percentage position of the stride where the maximum displacement occurred on this smoothed curve. As an example, if a value of 90 for the phase offset was not observed, it would indicate that the peak of lateral displacement occurred when the stride cycle was 90% complete. The lateral displacement metric assigned to a stride was the difference between the maximum and minimum displacement values ​​observed during the stride (Figure 9A). This analysis was highly sensitive and was able to detect subtle but very significant differences in overall posture during strides. Conventional classical spatiotemporal measurements based on Hildebrand's method were used in combination with whole-body posture metrics for the analysis. Due to the periodic nature of the phase offset indices, care was taken in the analysis to apply angular statistics to them. Other measurements were analyzed using linear methods.

[0093] Statistical analysis and genetic validation of gait measurements Following the extraction of gait and posture data, a statistical framework for data analysis was established. To validate this method, the phenotypes of three mouse models were determined, each previously shown to have gait disturbances and to be a preclinical mode of human diseases: Rett syndrome, amyotrophic lateral sclerosis (ALS or Lou Gehrig's disease), and Down syndrome. The three models—Mecp2 knockout, SOD1 G93A transgene, and Ts65Dn trisomic—were tested in a 1-hour open-field assay at two ages, using appropriate controls (Figure 14B). Gait indices showed high correlation with animal size and stride velocity (Hildebrand, M. Bioscience (1989) 39:766) (Figures 8I-J). However, in many cases, changes in stride velocity are a definitive feature of gait changes due to genetic or pharmacological perturbations. In addition, when multiple repeated measurements were collected for each subject (mouse) using this method, the number of strides differed for each subject, resulting in unbalanced data. Averaging over repeated strides yields a single mean for each subject, but this can be misleading because it removes variability and introduces a false confidence level. At the same time, classical linear models fail to distinguish between stable intra-subject variability and inter-subject fluctuations, severely biasing the estimates. To address this, a linear mixed model (LMM) was used to separate intra-subject variability from genotype-based inter-subject variability (Laird, NM & Ware, JH, Biometrics (1982) 38:963-974; Pinheiro, J. & Bates, D. Mixed-effects models in S and S-PLUS, New York: Springer-Verlag, 2000). Specifically, in addition to main effects such as animal size, genotype, and age, it includes random effects that capture intra-subject variability. Furthermore, multiple repeated measures were taken at two different age groups, resulting in a nested, hierarchical data structure.The models (M1, M2, M3) followed the standard LMM notation, where (genotype, body length, velocity, and test age) represented fixed effects, and (mouse ID / test age) (where test age is nested within the animal) represented random effects. To compare the results with previously published data that did not consider stride velocity when animal size was not considered, the results were statistically modeled using three models: one considering only age and body length (M1), one considering only age and velocity (M2), and one considering only age, velocity, and body length (M3) (Figures 10 and 14). The models were: M1: phenotype ~ genotype + test age + body length + (1|mouse ID / test age), M2: phenotype ~ genotype + test age + velocity + (1|mouse ID / test age), and M3: phenotype ~ genotype + test age + velocity + body length + (1|mouse ID / test age).

[0094] Sex was not included in the model because it correlates strongly with body length (measured using ANOVA and expressed as η, it is strong in both SOD1 (η=0.81) and Ts65Dn (η=0.16 for the whole, η=0.89 for controls, and η=0.61 for mutants)). Males and females of Mecp2 were analyzed separately. The circular phase variables in Figure 14A were modeled as functions of linear variables using a circular-linear regression model (Fisher, NI & Lee, AJ, Biometrics (1992) 48:665-677). To adjust for linear variables such as body length and velocity, they were included in the model as covariates (see also Methods). Figures 10 and 11 report the p-values ​​and normalized effect sizes. For clarity, the exact statistics are reported in detail in Figures 19 and 20.

[0095] Verification using a Rett syndrome model Rett syndrome, a hereditary neurodevelopmental disorder, is caused by mutations in the X-linked MECP2 gene (Amir, RE et al., Nat Genet (1999) 23:185-188). The study included commonly studied Mecp2 deletions that reproduced many of the features of Rett syndrome, including reduced motor skills, dysgait, limb stunting, low birth weight, and lethality (Guy, J. et al., Nature Genet, (2001) 27:322-326). Hemizygous males (n=8), heterozygous females (n=8), and litter controls (n=8 for each sex) were tested (Figure 14B). Null males were normal at birth, with an expected lifespan of approximately 50–60 days. They began to show age-dependent phenotypes by 3–8 weeks and became lethal by 10 weeks. Heterozygous females exhibit milder symptoms even at older ages (Guy, J. et al., Nature Genet, (2001) 27:322-326). Male mice were tested twice, on days 43 and 56, while female mice were tested on days 43 and 86.

[0096] Studies on this knockout have shown that stride length and stance width change in an age-dependent manner in hemizygous males (Kerr, B. et al., PLoS One (2010) 5(7): el 1534; (2010), Robinson, L. et al., Brain (2012) 135: 2699-2710). Recent analyses have shown increased step width, decreased stride length, changes in stride time, changes in step angle, and changes in overlap distance (Gadalla, KK et al., PLoS One (2014) 9(11): el 12889). However, these studies have not adjusted for the decrease in body size observed in Mecp2 hemizygous males (Figure 14D), and in some cases, have not modeled stride velocity. The most appropriate comparison between the obtained experimental data and previously published data was made using M2, which models speed but not body length (Gadalla, KK et al., PloS One (2014) 9(11): el 12889, Figure 14D). Most gait indices and some body coordination indices, including limb load coefficients, step length and stride length, step-to-stride, and temporal symmetry, were found to differ significantly between hemizygous males and controls. However, most gait indices depend on animal size, and hemizygous males are 13% smaller in body length (Figure 14D) (Guy, J. et al., Nature Genet, (2001) 27: 322-326). In addition, the analysis was limited to stride speeds of 20 cm / s to 30 cm / s, which helped reduce variability caused by speed differences. Therefore, a comparison was also made between a model that included body length instead of speed as a covariate (M1, Figure 10A) and a model that included both body length and speed (M3, Figure 15A). The results for the M2 model showed significant differences in stride speed, step width, stride length, and whole-body coordinated phenotype (tail tip amplitude, tail tip and nose phase) in hemizygous males (Figure 10B). Most phenotypes were age-dependent, with serious effects observed in males by 7 weeks (56 days) (Figure 10D).Model (M3), which included both velocity and body length, showed a significant decrease in step width, a suggestive difference in stride length, and large differences in whole-body coordination indicators (tail tip amplitude, tail tip phase, tail base phase, and nose phase) (Figure 15). In Mecp2 heterozygous females, little significant difference was observed, and this was consistent across all three models. In all three models, tail tip amplitude was consistently significantly larger, suggesting greater lateral movement in females (Figures 10A-B and 15). Taken together, these results demonstrate that this method can accurately detect the previously described differences in Mecp2. In addition, the whole-body coordination indicators were able to detect previously unexplained differences.

[0097] Verification using the ALS model Mice carrying the SOD1-G93A transgene are a preclinical model for ALS, which involves progressive loss of motor neurons (Gurney, ME et al., Science (1994) 264:1772-1775, Rosen, D et al., Nature (1993) 362:59-62). The SOD1-G93A model has been shown to exhibit changes in gait phenotype, particularly with respect to the hind limbs (Wooley, C. et al., Muscle & Nerve (2005) 32:43-50, Amende, I. et al., J Neuroeng Rehabilitation (2005) 2:20, Preisig, D. et al., Behavioural Brain Research (2016) 311:340-353, Tesla, R. et al., PNAS (2012) 109:17016-17021, Mead, R. et al., PLoS ONE (2011) 6:e23244, Vergouts, M. et al., Metabolic Brain Disease (2015) 30:1369-1377, Mancuso, R. et al., Brain Research (2011) 1406:65-73). The most prominent phenotype is an age-dependent pattern of increased stance time (load factor) and decreased stride length. However, some other studies have observed the opposite results (Wooley, C et al., Muscle & Nerve (2005) 32:43-50, Amende, I et al., J Neuroeng Rehabilitation (2005) 2:20, Mead, R J et al., PLoS ONE (2011) 6:e23244, Vergouts, M et al., Metabolic Brain Disease (2015) 30:1369-1377), and some studies have not found a significant effect on gait (Guillot, T et al., Journal of Motor Behavior (2008) 40:568-577). These studies did not adjust for differences in body size, and in some cases, did not adjust for speed.The SOD1-G93A transgene and appropriate controls were tested at 64 and 100 days of disease onset (Wooley, C. et al., Muscle & Nerve (2005) 32:43-50, Preisig, D. et al., Behavioural Brain Research (2016) 311:340-353, Vergouts, M. et al., Metabolic Brain Disease (2015) 30:1369-1377, Mancuso, R. et al., Brain Research (2011) 1406:65-73, Knippenberg, S. et al., Behavioural Brain Research (2010) 213:82-87).

[0098] Surprisingly, the phenotypic differences between the transgene carriers and controls were found to vary significantly depending on the linear mixed model used. In M1, which adjusted for body length and age but not speed, stride speed, length, and loading coefficient were significantly different (Figure 10A). However, when speed was included in the model (M2), or when speed and body length were included in the model (M3), the differences were only small changes in the phase of the tail tip and nose (Figures 10B and 15). This indicates that the changes observed in the loading coefficient and stride length using M1 were attributable to changes in stride speed. These results suggest that the primary effect of the SOD1 transgene lies in stride speed, which in turn leads to changes in stride time and loading coefficient. The slight changes in whole-body coordination are attributable to the decrease in body size (Figure 14D). These results suggest that gait changes may not be the most sensitive preclinical phenotype in this ALS model, and are consistent with reports that other phenotypes, such as visible clinical symptoms and motor learning tasks like rotarods, are more sensitive measures (Mead, RJ et al., PLoS ONE (2011) 6:e23244, Guillot, T et al., Journal of Motor Behavior (2008) 40:568-577). In summary, these test results may validate the statistical model and help explain some of the conflicting results in the literature.

[0099] Verification using a Down syndrome model Down syndrome is caused by trisomy of all or part of chromosome 21 and has a complex neurological and neurosensory phenotype (Haslam, RHD Down syndrome: living and learning in the community. New York: Wiley-Liss, 107-14 (1995)). Although various phenotypes exist, such as intellectual disability, seizures, strabismus, nystagmus, and hearing loss, the most prominent phenotype is developmental delay in fine motor skills (Shumway-Cook, A. & Woollacott, MH Physical Therapy 65:1315-1322 (1985), Morris, A. et al., Journal of Mental Deficiency Research (1982) 26:41-46). These are often explained as clumsy or uncoordinated movements (Vimercati, S. et al., Journal of Intellectual Disability Research (2015) 59:248-256, Latash, ML Perceptual-motor behavior in Down Syndrome (2000) 199-223). The Tn65Dn mouse, one of the best-studied models, is a trisomy of the mouse chromosome 16 region, synteny with human chromosome 21, and reproduces many of the characteristics of Down syndrome (Reeves, R. et al., Nat Genet (1995) 11:177-184, Herault, Y. et al., Dis Model Meeh (2017) 10:1165-1186). Tn65Dn mice have been studied for their gait phenotype using conventional inkblot footprint analysis or treadmill methods (Hampton, TG and Amende, IJ Mot Behav (2009) 42:1-4, Costa, AC et al., Physiol Behav (1999) 68:211-220, Faizi, M. et al., Neurobiol Dis (2011) 43, 397-413).Inkblot analysis revealed mice exhibiting shorter, more "unstable," and "irregular" gaits, similar to the motor coordination disorders seen in patients (Costa, AC et al., Physiol Behav (1999) 68:211-220). Treadmill-based analysis revealed further changes in stride length, step count, several motor parameters, and footprint size (Faizi, M. et al., Neurobiol Dis (2011) 43, 397-413, Hampton, T. Get al., Physiol Behav (2004) 82:381-389). These previous analyses did not study the overall posture of these mice.

[0100] Using the method of the present invention, Tn65Dn mice were analyzed along with control mice at approximately 10 and 14 weeks (Figure 14B), and consistent changes were found in all three linear mixed models M1-M3. Tn65Dn mice showed increased stride velocity (Figures 10A, C) but were not hyperactive in open fields (Figure 10C). This indicates that Tn65Dn mice had faster stepping speed but the same distance traveled as controls. Step length increased, while step length and stride length significantly decreased. The most significant deviation from controls was obtained in M3, which considered speed and body length. In particular, the whole-body coordinated phenotype was greatly affected in Tn65Dn mice. The amplitude between the tail base and tail tip, the phase of the tail base, the phase of the tail tip, and the phase of the nose were significantly reduced (Figure 15A). This result was confirmed in the phase plots of the nose and tail tip (Figure 10E). Surprisingly, significant differences in phase were found. The phase peak at the tail tip was nearly 30% of the stride period in controls and nearly 60% in mutants at multiple velocities (Figure 10E). Similar changes were observed in the phase plot for the nose. Taken together, these results confirm previously reported differences in conventional gait measurements and highlight the usefulness of a novel open-field whole-body coordination measurement method in expanding the measurable phenotypic features in human disease models. Indeed, the most striking feature of Tn65Dn gait is the change in whole-body coordination, which had previously been reported as a qualitative trait using inkblot analysis (Costa, ACet al., Physiol Behav (1999) 68:211-220), but is now quantifiable using the method of the present invention.

[0101] Characterization of autism spectrum disorder-related variants To further validate the analytical approach, we investigated gait in four autism spectrum disorder (ASD) mouse models, including Mecp2, which also corresponds to autism spectrum disorder. In humans, gait and postural defects are more common in ASD patients, and sometimes these defects precede classical deficits in verbal and social communication, as well as stereotyped behaviors (Licari, MK et al., Autism Research (2020) 13:298-306, Green et al., Dev Med Child Neurol (2009) 51:311-316). Recent studies have shown that motor changes are often undiagnosed in ASD cases (Hughes, V. Motor problems in autism move into research focus. Spectrum News (2011)). It remains unclear whether these differences are genetic or secondary, due to a lack of social interaction that could help children develop learned motor coordination (Zeliadt, N., Autism in motion: Could motor problems trigger social ones. Scientific American, Spectrum, Mental Health (2017)). Since gait defects are not well-characterized in mouse models of ASD, studies were conducted to determine whether any gait phenotypes occur in four commonly used ASD gene models, which were characterized at 10 weeks using appropriate controls (Figure 14C). Similar to the three models with known gait defects, these mutants and controls were tested in a 1-hour open-field assay to extract gait and postural indices (Figure 14A). The results were modeled using the same approach as used for the gait mutants (results for M1 and M3 are shown in Figure 11, and results for M2 are shown in Figure 16).

[0102] Cntnap2 is a member of the neurexin gene family that functions as a cell adhesion molecule between neurons and glial cells (Poliak, S. et al., Neuron (1999) 24:1037-1047). Mutations in Cntnap2 are associated with neurological disorders such as ASD, schizophrenia, bipolar disorder, and epilepsy (Toma, C. et al., PLoS Genetics (2018) 14:el007535). Cntnap2 knockout mice have been previously shown to have mild gait effects, with increased stride speed leading to decreased stride duration (Brunner, D. et al., PloS One (2015) 10(8):e0134572). Comparing our results with previous studies using Model M2, we found that Cntnap2 mice showed significant differences in most gait measurements (Figure 16). These mice were significantly smaller in body length and weight compared to controls (Figure 14D, Figure 16C). In open fields, Cntnap2 mice were not hyperactive (Figure 11C), but their stride speed was significantly increased (M1, Figures 11A, C, and 16C). These results suggest that Cntnap2 mice do not move as much, but rather, like Ts65Dn mice, have faster steps when moving.

[0103] Because Cntnap2 mice are smaller in size and have a faster stride speed, the M3 results were used to determine whether there were any changes in gait parameters after adjusting for size and stride speed (Figure 14D). The results showed that Cntnap2 mice were significantly different from controls in most aspects of conventional gait indices and whole-body coordination measurements in both the M1 and M3 models (Figure 11B). Cntnap2 mice showed decreased limb load coefficient, step length, and step width, and a significant decrease in stride length (Figures 11B, D, and 16C). The mice also showed changes in the phase of the tail tip, tail base, and nose, and the amplitudes of the tail tip, tail base, and nose also changed, albeit significantly. Another notable feature of gait in Cntnap2 mice, compared to controls, is reduced inter-animal variability, particularly with respect to limb load coefficient (Fligner-Killeen test, p<0.01), step length (Fligner-Killeen test, p<0.01), and stride length (Fligner-Killeen test, p<0.02) (Figure 11D). This may indicate that gait is more stereotypic in these mutants. Taken together, these results suggest that Cntnap2 mice are not hyperactive when measured by total distance traveled in open field, but are hyperactive at the individual stride level. Cntnap2 mice exhibit faster steps, shorter stride and step lengths, and narrower step widths. Furthermore, studies were performed to distinguish Cntnap2 mice from controls based on all combined gait measurements using unsupervised clustering. First, principal component analysis (PCA) was performed on the linear gait phenotype, and then, using a Gaussian mixture model (GMM) for PC, the animals were clustered into two separate groups. It was determined that Cntnap2 could be distinguished from the control by the gait index (Figure 11E).This analysis suggests that Cntnap2 mice can be distinguished from controls based on their gait patterns in open fields, and that these phenotypes are dramatically different compared to those previously detected (Brunner, D. et al., PloS One (2015) 10(8):e0134572).

[0104] Mutations in Shank3, a postsynaptic protein that forms a scaffold, have been found in multiple cases of ASD (Durand, CM et al., Nat Genet (2007) 39:25-27). Mutations in Fmr1, an RNA-binding protein that functions as a translation regulator, are associated with Fragile X syndrome, the most commonly inherited form of psychiatric disorder in humans (Crawford, DC et al., Genetics in Medicine (2001) 3:359-371). Fragile X syndrome has a wide range of phenotypes that overlap with the characteristics of ASD (Belmonte, MK and Bourgeron, T. Nat Neurosci (2006) 9:1221-1225). Del4Aam mice contain a 0.39 Mb deletion on mouse chromosome 7, which is synteny with human chromosome 16p11.2 (Horev, G. et al., PNAS (2011) 108:17076-17081). Copy number variations (CNVs) of human 16p11.2 are associated with various features of ASD, including intellectual disability, stereotypic behavior, social impairment, and language impairment (Weiss, LA et al., NEJM (2008) 358:667-675). Fmr1 mutant mice move more in open fields (Figure 11C) and have a greater stride velocity (Figure 11A, C). After adjusting for stride velocity and body length (M3), these mice showed a slight but significant change in limb load coefficients in M2 and M3. Both Shank3 and Del4Aam mice exhibit reduced motility in open fields compared to controls. Shank3 mice exhibited a significantly reduced stride velocity, while Del4Aam mice had a faster stride velocity (Figure 11A, C). All three statistical models showed a suggestive or significant decrease in step length in both strains. Using M3, Shank3 mice were determined to have longer step and stride lengths, while Del4Aam mice were determined to have shorter steps and strides. In whole-body coordination, Shank3 mice showed reduced nasal phase, while Del4Aam mice showed increased caudal apex phase.These results showed that, despite reduced activity levels in open fields, Shank3 exhibited slower, longer strides and steps, while Del4Aam exhibited faster strides with shorter steps and strides. Both variants had several defects in whole-body coordination. In summary, it was determined that each ASD model had some gait impairment, with Cntnap2 exhibiting the most severe phenotype. Although the direction of change and phenotypic variance differed, all showed some variation in stride velocity.

[0105] Systematic survey After validation of the analytical methods, experiments were performed to understand the range of gait and postural phenotypes in standard laboratory mouse strains in open field. Forty-four classic inbred laboratory strains, seven wild-derived inbred strains, and eleven F1 hybrid strains were investigated (1898 animals, 1740 hours of video). All animals were of the same strain, and both males and females were investigated in a one-hour open field assay (Figure 14E) (Geuther, BQ et al., Commun Biol (2019) 2:1-11). Subsequently, gait indices were extracted from each video, and the data were analyzed at the animal level (Figures 12A-B, 17, and 18). Stride data was analyzed when animals were moving in a straight line at a moderate speed (20 cm / sec to 30 cm / sec) (angular velocity of -20 degrees / sec to +20 degrees / sec). This selective analysis was possible because a large amount of data on freely moving mice could be collected and processed. Since these mice exhibited considerable variability in body size, residuals from M1 (Geuther, BQ et al., Commun Biol (2019) 2:1-11), which adjusted for body size, were used. M1 allowed for the extraction of stride velocity, which was determined to be important in ASD mutants, as a feature. To visualize the differences between strains, z-scores were calculated for the phenotype of each strain, and k-mean clustering was performed (Figure 12B). Overall, large inter-strain variability was observed in most classical gait and whole-body posture indicators, indicating high heritability of these traits. Furthermore, a novel pattern was observed in open-field walking behavior of laboratory mice, where certain strains exhibited similar behaviors.

[0106] A study was conducted to determine whether lineages could be clustered based on gait and posture phenotypes in open field. A k-means clustering algorithm was applied to the principal components obtained by performing PCA on the original linear gait features, similar to what was done with the Cntnap2 mutant. Since both PCA and the k-means clustering algorithm assume the indices are in Euclidean space, circular phase indices were not included in the clustering analysis. The first two PCs were selected as explaining 53% of the total variance in the original feature space. Four criteria were considered to evaluate the quality of the clustering, and the optimal number of clusters in the k-means clustering algorithm was selected, all of which resulted in three optimal clusters (Figure 21). It was determined that there are three clusters of lineages that can be distinguished based on gait behavior in open field (Figures 12C-12E). Cluster 1 consisted mainly of classical lineages such as A / J, C3H / HeJ, and 129S1 / SvImJ, while Cluster 2 consisted of several classical lineages and numerous wild-derived lineages such as MOLF / EiJ and CAST / EiJ. Cluster 3 mainly consisted of C57 and related lineages, including the reference C57BL / 6J. Consensus stride phase plots between the nose and tail tip were created for each cluster. Cluster 3 had a very large amplitude, while Clusters 1 and 2 had similar amplitudes but shifted phase offsets (Figure 12D). Examination of linear gait indicators revealed individual indicators that distinguished the clusters (Figure 12E). For example, Cluster 1 had longer stride and step lengths, Cluster 3 had greater lateral displacement of the tail base and tail tip, and Cluster 2 had smaller lateral displacement of the nose. Overall, analysis of individual indicators revealed significant differences in 9 out of 11 items. In summary, this analysis revealed a high level of genetic variation in the gait and overall posture of laboratory mice. Combinatorial analysis using multidimensional clustering of these indicators identified three subtypes of gait in laboratory mice.These results also demonstrate that the reference mouse strain, C57BL / 6J, differs from other common mouse strains and wild-derived strains.

[0107] GWAS Phylogenetic studies demonstrated that the measured gait characteristics were highly variable, so a study was conducted to investigate the genetic components and genetic architecture of mouse gait in open field. In human GWAS, both the mean and variance of gait traits are highly heritable (Adams, HHet al., J of Gerontol A Biol Sci Med Sci (2016) 71:740-746). To perform a GWAS and identify quantitative trait loci (QTLs) in the mouse genome, the stride of each animal was divided into four different bins according to locomotion speed (10cm / s-15cm / s, 15cm / s-20cm / s, 20cm / s-25cm / s, 25cm / s-30cm / s), and the mean and variance of each trait were calculated for each animal. Genome-wide association analysis was performed using a linear mixed model that considered sex and body length as fixed effects and population structure as a random effect, using GEMMA (Zhou, X. and Stephens, M. Nat Genet (2012) 44:821-824). Phase gait data were excluded from this analysis because the linear mixed model cannot handle circular values. Heritability was estimated by determining the proportion of phenotypic variance (PVE) explained by type-specific genotypes (left panel in Figure 13A). The heritability of gait measurements showed a wide range, and the majority of phenotypes were moderately to highly heritable. The least heritable average phenotypes were angular velocity and temporal symmetry, indicating that the variance in gait or turning behavior symmetry is not due to genetic variance in laboratory mice. In contrast, whole-body coordination measurements (amplitude measurements) and conventional gait measurements were found to be moderately to highly heritable. The phenotypic variance showed moderate heritability even for traits with low mean trait heritability (right panel in Figure 13A). For example, the mean angular velocity phenotype had low heritability (PVE < 0.1), while the variance angular velocity phenotype had moderate heritability (PVE 0.25–0.4). These heritability results indicate that gait and posture traits are appropriate for GWAS of mean and variance traits.

[0108] For significance thresholds, an empirical p-value correction was calculated for the association between SNPs and phenotypes by shuffling the values ​​between individuals (total distance traveled in the open field) 1000 times. For each permutation, the minimum p-value was extracted to find a threshold representing a corrected p-value of 0.05 (1.9 × 10⁻⁵). A combined Manhattan plot was created by obtaining the minimum p-value across all combined mean phenotypes, variance phenotypes, and both classes for each SNP (Figures 13B-D). Each SNP is colored according to the phenotype associated with the SNP with the minimum p-value. The bins for different velocities were generally consistent per phenotype, and it was decided to combine all bins for the same phenotype by obtaining the minimum p-value for the four bins for each SNP.

[0109] 239 QTLs were determined for mean traits and 239 QTLs for variance traits (Figures 13B-C). Mean angular velocity, the least heritable phenotype, showed only one significant associated genomic region, while variance of angular velocity had 53 associated genomic loci. The phenotype with the most associated loci was stride number, with 95 loci. Overall, considering all phenotypes together, 400 significant genomic regions were found associated with at least one phenotype (Figure 22), indicating that only 78 QTLs were identified for both mean and variance phenotypes. For most phenotypes, there was no overlap between the QTLs associated with the mean of a feature and the QTLs associated with its variance. Notably, there was significant overlap between the QTLs associated with mean time symmetry and the QTLs associated with variance time symmetry. Of the 28 loci associated with mean phenotypes and the 52 loci associated with variance, 10 QTLs overlapped. These data suggest that the genetic structure in the mean trait and the genetic structure in the dispersed trait are largely independent of each other. These results also begin to outline the genetic landscape of mouse gait and posture in open fields.

[0110] Consideration Gait and posture are important indicators of health and influence many neurological, neuromuscular, and neuropsychiatric disorders. The aim of these experiments was to develop a simple, reliable automated system capable of pose estimation for mice and extracting important gait and postural indicators from those poses. The information herein presents a solution that allows researchers to adapt video imaging systems used for open-field analysis to extract gait indicators. This approach has several clear advantages and limitations. In this method, the data that needs to be acquired is related to mice in an open field. From above Because it uses only grayscale video, and because subsequent pose estimation and gait indicator extraction are all fully automated, it can process large amounts of data with little effort and low cost. This method does not require expensive specialized equipment, allowing mice time to become accustomed to the open field, and enabling data collection over long periods. In addition, the method of this invention allows animals to move voluntarily (unforced behavior) within a familiar environment, resulting in a more appropriate assay from an animal behavioral perspective (Jacobs, BY et al., Curr Pain Headache Rep (2014) 18:456). Because it uses a video technique, it was not possible to measure the motor characteristics of gait (Lakes, EH & Allen, KDO Steoarthr Cartil (2016) 24:1837-1849). Furthermore, From aboveThe decision to use video meant that several pose keypoints were often obscured by the mouse's body. While the pose estimation network was robust to some degree of occlusion, as in the case of the hind limbs, for the forelimbs, they were almost always obscured during walking, and were therefore excluded from the analysis due to the high inaccuracy of the pose estimates. In any case, in all genetic models tested, hind limb data were sufficient to detect clear differences in gait and posture. In addition, the ability to analyze large amounts of data from freely moving animals proved to be highly sensitive, even with very strict heuristic rules regarding what constitutes walking.

[0111] While extracted gait indices are commonly quantified in experiments (e.g., step width and stride length), measurements of whole-body coordination, such as tail lateral displacement and phase (phase and amplitude of keypoints during stride), are not typically measured in rodent gait experiments. Gait and whole-body posture are frequently measured in humans as internal phenotypes of mental disorders (sanders2010gait, licari2020prevalence, flyckt1999neurological, walther2012motor). Results from the studies described herein using mice indicate that measurements of gait and whole-body coordination are highly heritable and confusing in disease models. Specifically, tests were performed to evaluate neurodegenerative models (Sod1), neurodevelopmental models (Down syndrome, Mecp2), and ASD models (Cntnap2, Shank3, FMR1, Del4Am), and altered gait characteristics were confirmed in all of these variants. Similar results have been found in other cases, including the neurodegenerative model machado2015quantitative. Of particular note are the data concerning Down syndrome. In humans, a lack of coordination and clumsiness are prominent features of Down syndrome. In mouse models, this lack of coordination has previously been characterized as disorder of hindlimb footprints in an inkblot gait assay. Here, analysis revealed differences in systemic coordination disorder between control mice and Tn65Dn mice. Thus, the approach described herein has made it possible to quantify traits that were previously qualitative.

[0112] Analysis of numerous mouse strains regarding gait and posture identified three distinct classes in terms of overall movement. The reference C57BL / 6J strain and related strains were found to belong to a separate cluster, distinct from other common laboratory strains and wild-derived strains. The main difference was observed in the large amplitude of tail and nose movements in the C57BL / 6 and related strains. This may be important when analyzing gait and posture in different genetic backgrounds. GWAS revealed 400 QTLs for open-field gait and posture, both in terms of mean and variance phenotypes. It was found that the mean and variance of traits were controlled by different loci. Indeed, the method of this invention identified that even for mean traits with low heritability, most variance phenotypes exhibited moderate heritability. Human GWAS, although with a low-power sample, has been conducted on gait and posture, and while the heritability estimates were good, the heritability of significantly associated loci was only slight. The results presented here for mice support the conclusion that, with well-founded research in humans, it may be possible to identify hundreds of genetic factors that control gait and posture.

[0113] Example 2 Devices and Systems One or more machine learning models in System 150 may take many forms, including neural networks. A neural network may contain several layers, from an input layer to an output layer. Each layer is configured to take a specific type of data as input and to output a different type of data. The output from one layer is introduced as input to the next layer. While the input / output data values ​​in a particular layer are unknown until the neural network actually operates at runtime, the data describing the neural network describes the structure, parameters, and operation of its layers.

[0114] One or more of the intermediate layers of a neural network may also be known as hidden layers. Each node in a hidden layer is connected to each node in the input layer and to each node in the output layer. If the neural network contains multiple intermediate layers, each node in a hidden layer will be connected to each node in the next higher layer and the next lower layer. Each node in the input layer represents a potential input to the neural network, and each node in the output layer represents a potential output from the neural network. Each connection from one node to another in the next layer may be associated with a weight or score. The neural network may output a single output or a weighted set of multiple predicted outputs.

[0115] In some embodiments, the neural network may be a convolutional neural network (CNN), which may be a normalized version of a multilayer perceptron. The multilayer perceptron may be a fully connected network, that is, a network in which each neuron in one layer is connected to all neurons in the next layer.

[0116] In one embodiment, a neural network may be constructed with iterative connections such that the output of the network's hidden layers is fed back into the hidden layers for the next set of inputs. Each node in the input layer is connected to each node in the hidden layer. Each node in the hidden layer is connected to each node in the output layer. The output of the hidden layers is fed back into the hidden layers to process the next set of inputs. A neural network incorporating iterative connections may be called an iterative neural network (RNN).

[0117] In some embodiments, the neural network may be a long-short-term memory (LSTM) network. In some embodiments, the LSTM may be a bidirectional LSTM. A bidirectional LSTM performs input from two time directions: one from past states to future states, and the other from future states to past states, where past states may correspond to features of video data for a first time frame, and future states may correspond to features of video data for a subsequent second time frame.

[0118] The processing performed by a neural network is determined by the learned weights on each node's input and the network's structure. Given a specific input, the neural network determines the output layer by layer until the output layer of the entire network is computed.

[0119] Connection weights can be initially learned by the neural network during training, associating a given input with a known output. The training data set provides various training examples into the network. In each example, the weights of the correct connections from input to output are typically set to 1, and all connections are weighted to 0. When the training data examples are processed by the neural network, the input may be sent to the network and compared with the associated output to determine how to compare the network's performance to the target performance. Training techniques such as backpropagation may be used to update the neural network's weights and reduce errors that occur when the neural network processes the training data.

[0120] Various machine learning techniques may be used to train and manipulate models for performing various steps described herein, such as user recognition feature extraction, coding, user recognition scoring, user recognition confidence determination, etc. Models may be trained and manipulated according to various machine learning techniques. Such techniques may include, for example, neural networks (deep neural networks and / or iterative neural networks, etc.), inference engines, trained classifiers, etc. Examples of trained classifiers include support vector machines (SVMs), neural networks, decision trees, AdaBoost ("Adaptive Boost") combined with decision trees, and random forests. Focusing on SVMs as an example, SVMs are supervised learning models that have association learning algorithms to analyze data and recognize patterns in the data, and these supervised learning models are commonly used for classification and regression analysis. Given a set of training examples, each marked as belonging to one of two categories, the SVM training algorithm constructs a model that assigns new examples to one of the two categories, making it a non-stochastic binary linear classifier. A more complex SVM model may be constructed with a training set that identifies three or more categories, and the SVM determines which category is most similar to the input data. The SVM model may map examples of distinct categories so that they are separated by distinct gaps. New examples are then mapped into the same space and predicted to belong to a category based on which side of the gap they fall on. The classifier may issue a “score” indicating which category the data best matches. The score may provide an indicator of how closely the data matches a category.

[0121] To apply machine learning techniques, the machine learning process itself needs to be trained. In this case, to train machine learning components such as either the first or second model, it is necessary to establish a "ground truth" regarding the training examples. In machine learning, the term "ground truth" refers to the accuracy of the classification of the training set with respect to supervised learning techniques. Various techniques, including backpropagation, statistical learning, supervised learning, semi-supervised learning, stochastic learning, or other known techniques, may be used to train the model.

[0122] Figure 23 is a block diagram conceptually illustrating device 1600 that may be used with the system. Figure 24 is a block diagram conceptually illustrating exemplary components of a remote device, such as system 150, which may assist in processing video data, identifying the behavior of an object, etc. System 150 may include one or more servers. As used herein, “server” may refer to a conventional server as understood in a server / client computing structure, but may also refer to a number of different computing components that may assist in the operations described herein. For example, a server may include one or more physical computing components (such as a rack server) that are physically and / or via a network connected to other devices / components and capable of performing computing operations. A server may also include one or more virtual machines that emulate a computer system and run on one device or across multiple devices. A server may also include other combinations of hardware, software, firmware, or similar for performing the operations described herein. The server may be configured to operate using one or more of the following computing technologies: client-server model, computer bureau model, grid computing technology, fog computing technology, mainframe technology, utility computing technology, peer-to-peer model, sandbox technology, or other computing technologies.

[0123] Multiple systems 150 may be included in the overall system of this disclosure, such as one or more systems 150 for performing keypoint / body part tracking, one or more systems 150 for gait index extraction, one or more systems 150 for posture index extraction, one or more systems 150 for statistical analysis, one or more systems 150 for training / configuring systems, and so on. When in operation, each of these systems may include computer-readable instructions and computer-executable instructions present on each device 150, as further described below.

[0124] Each of these devices (1600 / 150) may include one or more controllers / processors (1604 / 1704), each controller may include a central processing unit (CPU) for processing data and computer-readable instructions, and memory (1606 / 1706) for storing data and instructions for each device. The memory (1606 / 1706) may individually include volatile random access memory (RAM), non-volatile read-only memory (ROM), non-volatile magnetoresistive memory (MRAM), and / or other types of memory. Each device (1600 / 150) may also include a data storage component (1608 / 1708) for storing data and controller / processor executable instructions. Each data storage component (1608 / 1708) may individually include one or more non-volatile storage types, such as magnetic storage, optical storage, and solid-state storage. Each device (1600 / 150) may also be connected to removable or external non-volatile memory and / or storage (removable memory cards, memory key drives, network storage, etc.) via its respective input / output device interface (1602 / 1702).

[0125] Computer instructions for operating each device (1600 / 150) and its various components may be executed by the controller / processor (1604 / 1704) of each device, using memory (1606 / 1706) as temporary "working" storage at runtime. Computer instructions for devices may be stored non-temporarily in non-volatile memory (1606 / 1706), in storage (1608 / 1708), or in external devices. Alternatively, some or all executable instructions may be embedded in the hardware or firmware on each device, in addition to or instead of software.

[0126] Each device (1600 / 150) includes an input / output device interface (1602 / 1702). Various components may be connected via the input / output device interface (1602 / 1702), as will be further described below. In addition, each device (1600 / 150) may include an address / data bus (1624 / 1724) for transmitting data between the components of each device. Each component within a device (1600 / 150) may also be directly connected to other components, in addition to (or instead of) being connected to other components via the bus (1624 / 1724).

[0127] Referring to Figure 23, device 1600 may include an input / output device interface 1602 for connecting to various components, such as an audio output component, including a speaker 1612, a wired or wireless headset (not shown), or other components capable of outputting audio. Device 1600 may additionally include a display 1616 for displaying content. Device 1600 may further include a camera 1618.

[0128] The input / output device interface 1602 may be connected to one or more networks 199 via antenna 1614, via wireless local area network (WLAN) (such as WiFi) radio, Bluetooth, and / or wireless network radio, such as radios capable of communicating with wireless communication networks such as Long-Term Evolution (LTE) networks, WiMAX networks, 3G networks, 4G networks, and 5G networks. Wired connections such as Ethernet may also be supported. The system may be distributed across the network environment via network 199. The I / O device interfaces (1602 / 1702) may also include communication components that enable the exchange of data between devices such as different physical servers in a collection of servers or other components.

[0129] The components of device 1600 or system 150 may include their own dedicated processor, memory, and / or storage. Alternatively, one or more components of device 1600 or system 150 may utilize the I / O interface (1602 / 1702), processor (1604 / 1704), memory (1606 / 1706), and / or storage (1608 / 1708) of device 1600 or system 150, respectively.

[0130] As described above, multiple devices may be employed within a single system. In such a multi-device system, each device may contain different components for performing different aspects of the system's processing. Multiple devices may contain overlapping components. The components of device 1600 and system 150 described herein are illustrative and may be configured as standalone devices or may be included as a whole or in part as components of a larger device or system.

[0131] The concepts disclosed herein may be applied in a number of different devices and computer systems, including, for example, general-purpose computing systems, video / image processing systems, and distributed computing environments.

[0132] The above-described aspects of this disclosure are illustrative. They have been selected to illustrate the principles and uses of this disclosure and are not intended to be exhaustive or limit the disclosure. Many modifications and variations of the disclosed aspects may be apparent to those skilled in the art. Those skilled in the art in the computer and speech processing fields will recognize that the components and process steps described herein may be interchangeable with other components or steps, or with combinations of components or steps, and that the advantages and benefits of this disclosure may still be achieved. Furthermore, it will be apparent to those skilled in the art that this disclosure may be carried out without some or all of the specific details and steps disclosed herein.

[0133] The disclosed system configuration may be implemented as a computer method or as a manufactured article such as a memory device or a non-temporary computer-readable storage medium. The computer-readable storage medium may be computer-readable and may contain instructions for causing a computer or other device to perform the processes described herein. The computer-readable storage medium may be implemented by volatile computer memory, non-volatile computer memory, hard drives, solid-state memory, flash drives, removable disks, and / or other media. In addition, the system components may be implemented as firmware or hardware.

[0134] Equivalents While several embodiments of the present invention are described and illustrated herein, those skilled in the art will readily conceive of various other means and / or structures for carrying out the function and / or obtaining the results and / or one or more advantages described herein, and each of such variations and / or modifications will be considered within the scope of the present invention. More generally, those skilled in the art will readily understand that all parameters, dimensions, materials and configurations described herein are illustrative, and that the actual parameters, dimensions, materials and / or configurations will depend on the specific one or more applications in which the teachings of the present invention are used. Those skilled in the art will understand many equivalents to the specific embodiments of the present invention described herein, or will be able to verify such equivalents using only routine experiments. Thus, it will be understood that the embodiments described herein are presented only illustratively, and that the present invention may be carried out in ways other than those specifically described and described in the claims, within the scope of the appended claims and its equivalents. The present invention covers the individual features, systems, articles, materials and / or methods described herein. In addition, any combination of two or more such features, systems, articles, materials, and / or methods is included within the scope of the invention, provided that they are not mutually inconsistent. It will be understood that all definitions defined and used herein govern dictionary definitions, definitions in documents referenced by reference, and / or the common meanings of the defined terms.

[0135] As used herein in this specification and in the claims, the indefinite articles "a" and "an" will be understood to mean "at least one" unless explicitly stated to the contrary. As used herein in this specification and in the claims, the phrase "and / or" will be understood to mean "either or both" of the elements thus joined, that is, "either or both" of elements that exist as a combination in some cases and separately in others. Unless explicitly stated otherwise, other elements may exist, at their discretion, in addition to the elements specifically identified by the phrase "and / or," whether related to or unrelated to the elements specifically identified.

[0136] Conditional language used herein, in particular “can,” “could,” “might,” “may,” “eg,” and similar terms, is generally intended to convey that a particular embodiment includes certain features, elements, and / or steps, while other embodiments do not, unless otherwise specifically stated or understood in the context in which they are used. Therefore, such conditional language is not generally intended to mean that features, elements, and / or steps are required in any way with respect to one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps are included in or performed within any particular embodiment, with or without other inputs or prompts. “comprising,” “including,” “having,” and similar terms are synonymous, used comprehensively and in an open-ended manner, and do not exclude additional elements, features, actions, operations, and similar items. Furthermore, the term "or" is used in an inclusive sense (rather than an exclusive sense), so when used to connect a list of elements, for example, the term "or" can mean one element, several elements, or all elements in the list.

[0137] All documents, patents, patent applications, and publications cited or referenced in this application are incorporated herein by reference in their entirety. The claims are as follows:

Claims

1. A method performed by a computer, wherein the method is: Receiving video data representing a video that captures the movement of the target, specifically the target determination movement of the target within an open arena from an overhead perspective, By processing the aforementioned video data, point data is identified that tracks the movement of a set of multiple body parts of the subject over a certain period of time, including the intermediate portion of the subject's spine. Using the aforementioned point data, determine the multiple stance phases and corresponding multiple swing phases represented in the video data during a certain period. Based on the aforementioned plurality of stance phases and the plurality of swing phases, the plurality of stride intervals represented in the video data during a certain period are determined. Using the aforementioned point data, determine index data relating to the subject, which is based on each of the multiple stride intervals. The aforementioned indicator data relating to the subject is compared with the control indicator data, This includes determining the difference between the indicator data relating to the subject and the control indicator data based on the comparison, The stride intervals with multiple stride intervals are associated with a set of multiple frames in the video data. Determining the index data includes determining a displacement vector relating to the stride interval using the point data, wherein the displacement vector connects the mid-spine portion represented in the first frame of the set of multiple frames and the mid-spine portion represented in the last frame of the set of multiple frames. method.

2. The aforementioned set of body parts includes the nose, the base of the neck, the middle part of the spine, the left hind limb, the right hind limb, the base of the tail, the middle part of the tail, and the tip of the tail. The method according to claim 1, wherein the plurality of stance phases and the plurality of swing phases are determined based on changes in the movement speed of the left hind limb and the right hind limb.

3. Based on the toe-off event of the left hind limb or the right hind limb, the transition from the first stance phase of the multiple stance phases and from the first swing phase of the multiple swing phases is determined. The method according to claim 2, further comprising determining a transition from the second swing phase of the plurality of swing phases to the second stance phase of the plurality of stance phases based on a foot contact event of the left hind limb or the right hind limb.

4. The method according to claim 1, wherein the index data corresponds to at least one of (i) the gait measurement value of the subject at each stride interval and (ii) the posture measurement value of the subject at each stride interval.

5. The aforementioned set of multiple body parts includes the left hind limb and the right hind limb, and determining the aforementioned index data is Using the aforementioned point data, determine the step length for each stride interval, which represents the distance the right hind limb moves beyond the immediate contact of the left hind limb. Using the aforementioned point data, determine the stride length used for each stride interval, which represents the distance the left hind limb travels in each stride interval. The method according to claim 1, comprising using the aforementioned point data to determine a step width for each stride interval, which represents the distance between the left hind limb and the right hind limb.

6. The aforementioned set of multiple body parts includes the base of the tail, and determining the index data is The method according to claim 1, comprising using the point data to determine the velocity data of the object based on the movement of the tail root with respect to each stride interval.

7. The aforementioned set of multiple body parts includes the base of the tail, and determining the index data is Using the aforementioned point data, a set of multiple velocity data points relating to the object is determined based on the movement of the tail root at a set of multiple frames representing a set of stride intervals, The method according to claim 1, comprising determining the stride speed with respect to the stride interval by averaging a set of speed data.

8. The aforementioned set of multiple body parts includes the right hind limb and the left hind limb, and determining the aforementioned index data is Using the aforementioned point data, the first stance phase duration, which represents the amount of time the right hind limb is in contact with the ground during a given stride interval, is determined. Based on the duration of the first stance phase and the duration of the stride interval, the first load coefficient is determined. Using the aforementioned point data, the second stance phase duration, which represents the amount of time the left hind limb is in contact with the ground during the stride interval, is determined. Based on the duration of the second stance phase and the duration of the stride interval, the second load coefficient is determined. The method according to claim 1, comprising determining an average load coefficient with respect to the stride interval based on the first load coefficient and the second load coefficient.

9. The aforementioned set of multiple body parts includes the base of the tail and the base of the neck, and determining the aforementioned index data is Using the aforementioned point data, determine a set of vectors connecting the tail base and the neck base at a set of frames representing a set of stride intervals with multiple stride intervals, The method according to claim 1, comprising determining the angular velocity of the object with respect to the stride interval using the set of vectors.

10. The aforementioned set of multiple body parts further includes the nose of the subject, and determining the index data is: Using the aforementioned point data, a set of lateral displacements of the nose with respect to the stride interval is determined based on the vertical distance of the nose from the displacement vector for each frame of the set of frames. The method according to claim 1, wherein the lateral displacement of the nose is further based on the body length of the subject.

11. Determining the aforementioned indicator data is By performing interpolation using the set of multiple lateral displacements of the nose, a smooth curve of the lateral displacement of the nose with respect to the stride interval is generated. Using the smooth curve of the lateral displacement of the nose, determine at what point in the stride interval the maximum displacement of the nose occurred. The method according to claim 10, further comprising determining a percentage stride position representing a percentage of the stride interval completed when the maximum displacement of the nose occurs, thereby determining the displacement phase offset of the nose.

12. The aforementioned set of multiple body parts further includes the base of the tail of the subject, and determining the index data is, The method according to claim 1, comprising using the point data to determine a set of lateral displacements of the tail root with respect to the stride interval, based on the vertical distance of the tail root from the displacement vector for each frame in the set of a set of a set of frames.

13. Determining the aforementioned indicator data is By performing interpolation using the set of multiple lateral displacements of the tail base, a smooth curve of the lateral displacement of the tail base with respect to the stride interval is generated. Using the smooth curve of the lateral displacement of the tail base, determine at what point in the stride interval the maximum displacement of the tail base occurred, The method according to claim 12, further comprising determining a percentage stride position representing a percentage of the stride interval completed when the maximum displacement of the tail base occurs, thereby determining the displacement phase offset of the tail base.

14. The aforementioned set of multiple body parts further includes the tail tip of the subject, and determining the index data is, The method according to claim 1, comprising using the point data to determine a set of lateral displacements of the tail tip with respect to the stride interval based on the vertical distance of the tail tip from the displacement vector for each frame in the set of a set of a set of frames.

15. Determining the aforementioned indicator data is By performing interpolation using the set of multiple lateral displacements of the tail tip, a smooth curve of the lateral displacement of the tail tip with respect to the stride interval is generated. Using the smooth curve of the lateral displacement of the tail tip, determine at what point in the stride interval the maximum displacement of the tail tip occurred, The method according to claim 14, further comprising determining a percentage stride position representing a percentage of the stride interval completed when the maximum displacement of the tail tip occurs, thereby determining the displacement phase offset of the tail tip.

16. The method according to claim 1, wherein the control index data is obtained from one or more control organisms, the subject is an organism, the control organism and the subject organism are of the same species, and the control organism is a laboratory strain of a certain species.

17. The method according to claim 1, wherein the control index data is obtained from one or more control organisms, and a statistically significant difference in the target index data when compared with the control index data indicates a difference in the phenotype of the target when compared with the phenotype of the control organism, the difference in phenotype indicates the presence of a disease or pathological condition in the target, and the difference in phenotype indicates a difference between the genetic background of the target and the genetic background of the control organism.

18. The method according to claim 1, wherein the control index data is obtained from one or more control organisms, and a statistically significant difference between the target index data and the control index data indicates a difference in the target genotype when compared with the genotype of the control organism, and the difference between the target genotype and the genotype of the control organism indicates a difference in lineage between the target and the control organism.

19. The method according to claim 1, wherein the control index data is obtained from one or more control organisms, and a statistically significant difference between the target index data and the control index data indicates a difference in the target genotype when compared with the genotype of the control organism, and the difference between the target genotype and the genotype of the control organism indicates the presence of a disease or pathological condition in the target.

Images