Video-based tracking of clinical states in patients with movement disorders

Abstract

Devices, systems, software, and methods are provided for determining symptom severity of a subject having a movement disorder. The methods utilize video-based objective symptom quantification with classification of severity of symptoms using a machine learning algorithm. The subject methods can be used to diagnose and quantify motor symptom progression in patients with movement disorders such as Parkinson's disease, which is useful for assessing and optimizing therapeutic interventions, such as dopaminergic medications and deep brain stimulation (DBS).

Description

Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 VIDEO-BASED TRACKING OF CLINICAL STATES IN PATIENTS WITH MOVEMENT DISORDERS CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims benefit of U.S. Provisional Patent Application No.63 / 530,566, filed August 3, 2023, which application is incorporated herein by reference in its entirety. BACKGROUND OF THEINVENTION

[0002] Parkinson’s disease (PD) is a common neurodegenerative movement disorder characterized by motor symptoms (e.g., bradykinesia, rest tremor, rigidity, postural instability) that are highly disabling and significantly impair quality of life (Fasano, A. et al. BMC Neurol. 19, 1-11 (2019); Sanchez-Luengos et al. Qual. Life Res.31, 3241-3252 (2022); Bloem, B. R., Okun, M. S. & Klein, C. Parkinson’s disease. Lancet 397, 2284-2303 (2021)). The ability to quantify motor symptom progression in PD patients is crucial for assessing and optimizing therapeutic interventions, such as dopaminergic medications and deep brain stimulation (DBS) (Armstrong & Okun JAMA 323, 548- 560 (2020)). Such quantification requires accurate and continual monitoring of motor symptom severity and fluctuation; however, this objective is only partially satisfied by the status quo strategy of routine clinical assessment. Motor symptoms are classically assessed at one time point by a single clinician using the Movement Disorder Society-sponsored revision of the Unified Parkinson’s Disease Rating Scale (MDS-UPDRS) (Goetz et al. Mov. Disord.23, 2129-2170 (2008)). Day-to-day quality and fluctuation of symptoms rely on subjective recall by the patients themselves. This approach is limited by high assessment variance, imperfect recall, and recency bias (Zolfaghari et al. Mov. Disord.37, 585-597 (2022)). To overcome these limitations, the field requires reliable and objective tracking of PD clinical states.

[0003] Technology-based objective symptom quantification, such as those supported by wearable tracking devices and their ability to record kinematic (movement) data, introduces a potential solution (Huckvale et al. npj Digital Medicine 2, 1-11 (2019); Matias et al. Front. Neurol. 8, 677 (2017)). Unfortunately, due to device expenses, technical limitations, and low data availability, embodiments of these quantification systems have yet to be regularly adopted into clinical practice for PD assessment (Habets et al. Sensors 21, 21(23), 7876 (2021); Pang et al. J. Neurosci. Methods 333, 108576 (2020); Khodakarami et al. Sensors 19, (2019)). More recently, advancements in machine learning (ML) and computer vision have yielded accessible solutions for extracting kinematic information at key anatomical positions from video data without the need for physical markers (Lugaresi et al. MediaPipe: A Framework for Building Perception Pipelines. Preprint atAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 doi.org / 10.48550 / ARXIV.1906.08172 (2019); Mathis et al. Nat. Neurosci. 21, 1281-1289 (2018); Graving et al. Elife 8:e47994 (2019); Cao et al. IEEE Trans Pattern Anal Mach Intell 43(1), 172-186 (2021)). These computer vision solutions have the potential to address the shortcomings of existing methods, significantly enhance diagnostic accuracy, and open new avenues for optimizing personalized medical therapy in PD (Espay et al. Mov. Disord.34, 657-663 (2019); Sibley et al. J. Parkinsons. Dis.11, S83-S93 (2021); van Kersbergen et al. BMC Res. Notes 14, 1-6 (2021); Park et al. bNeurology 96, e1761-e1769 (2021); Morinan et al. NPJ Parkinsons Dis 9, 10 (2023)). However, to date, many video-based diagnostic tools have implementation challenges of expensive and inaccessible technology, often requiring multi-camera setups, pristine video collection setups, or additional sensors that are infeasible for conventional use. Additionally, typical “black-box” ML implementations are not tailored to be clinically interpretable, either due to complex and unintuitive algorithms or a lack of analysis on feature stability and optimality. Therefore, they are generally ineffective in generating novel clinical insights and are challenging to integrate into current clinical care or develop clinical oversight for. Finally, these tools often focus on features from a single motor modality, missing the opportunity to integrate different domains (e.g., body posture, hand movement, facial expression), which has the potential to significantly increase the accuracy and robustness of ML predictions (Baltrusaitis, et al. IEEE Trans. Pattern Anal. Mach. Intell.41, 423-443 (2019); Lim et al. NPJ Parkinson’s Disease 8, 1-11 (2022); Makarious et al. NPJ Parkinsons Dis 8, 35 (2022)). A truly valuable video-based solution for tracking PD motor symptom progression would need to be affordable, accessible, automated, transparent, and able to obtain rich and clinically relevant metrics (Lonini et al. NPJ Digital Medicine 1, 1-8 (2018); Sica et al. PLoS One 16, e0246528 (2021)). SUMMARY OF THE INVENTION

[0004] Devices, systems, software, and methods are provided for determining symptom severity of a subject having a movement disorder. The methods utilize video-based objective symptom quantification with classification of severity of symptoms using a machine learning algorithm. The subject methods can be used to diagnose and quantify motor symptom progression in patients with movement disorders such as Parkinson’s disease, which is useful for assessing and optimizing therapeutic interventions, such as dopaminergic medications and deep brain stimulation (DBS).

[0005] In one aspect, a computer implemented method for determining symptom severity of a subject having a movement disorder is provided, the computer performing steps comprising: receiving a first video of the subject’s entire body while the subject is walking and / or receiving a second video of a hand of the subject while the subject is finger-tapping; segmenting the first videoAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 and / or the second video to generate a plurality of video segments; identifying body landmarks and / or hand landmarks in the plurality of video segments; using the body landmarks and / or hand landmarks to extract a set of body movement features and / or hand movement features from the plurality of video segments based on relative movement between the landmarks with time; using a machine learning algorithm to determine symptom severity of the subject based on classification of the set of body movement features and / or hand movement features; and displaying information regarding the symptom severity of the subject.

[0006] In certain embodiments, the movement disorder is idiopathic Parkinson’s disease, or the subject has Parkinsonian symptoms associated with an atypical Parkinson’s syndrome such as progressive supranuclear palsy or multiple systems atrophy.

[0007] In certain embodiments, the body landmarks comprise a nose, a right shoulder, a left shoulder, a right elbow, a left elbow, a right wrist, a left wrist, a right hip, a left hip, a right knee, a left knee, a right ankle, and a left ankle.

[0008] In certain embodiments, the hand landmarks comprise a wrist, a thumb, an index finger, a middle finger, a ring finger, a pinky finger, base of the index finger, and base of the pinky finger.

[0009] In certain embodiments, the body movement features comprise: neck angle; arm-body angle; the distance between the left wrist and the left shoulder and / or the distance between the right wrist and the right shoulder; the distance between the left ankle and the hip; the horizontal displacement between the ankles; the horizontal displacement between the knees; the horizontal displacement between the mid-shoulder and the mid-hip; the vertical displacement between the ankles; the vertical displacement between the shoulders; and the vertical displacement between the hips.

[0010] In certain embodiments, the hand movement features comprise: the distance between the thumb and the index finger of the active hand that is finger tapping; the distance between the middle finger and the wrist of the active hand that is finger tapping; the distance between the ring finger and the wrist of the active hand that is finger tapping; and the distance between the pinky finger and the wrist of the active hand that is finger tapping.

[0011] In certain embodiments, the display further displays body movement features comprising the relative power in 0.5 Hz to 1 Hz of ankle separation, standard deviation (STD) of ankle-hip distance, relative power in 2 Hz to 4 Hz of neck angle, relative power in 4 Hz to 6 Hz of arm-body angular velocity, and mean of neck angle.

[0012] In certain embodiments, the display further displays hand movement features comprising the half-power frequency in pinky finger movement speed, relative power in 2 Hz to 4 Hz of finger tapping speed, and median absolute deviation (MAD) of finger tapping amplitude.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0

[0013] In certain embodiments, the set of body movement features and / or hand movement features are identified by performing sparsity-inducing feature selection on the plurality of video segments.

[0014] In certain embodiments, the machine learning algorithm uses a logistic regression (LR) classifier, a support vector machine (SVM) classifier, or a linear discriminant analysis (LDA) classifier. The machine learning algorithm may be supervised, semi-supervised, or unsupervised. In some embodiments, the machine learning algorithm uses artificial neural networks. In some embodiments, the machine learning algorithm uses a deep learning algorithm, which may include the use of convolutional neural networks, deep neural networks, recurrent neural networks, deep residual neural networks, long short-term memory networks, deep belief networks, multilayer perceptrons, or deep reinforcement learning, and the like.

[0015] In certain embodiments, each video segment shows multiple strides during the walking or multiple finger-tapping cycles during the finger-tapping.

[0016] In certain embodiments, each video segment is 3 to 8 seconds in length. In some embodiments, any video segment less than 3 seconds in length is discarded. In some embodiments, any video segment greater than 8 seconds in length is further segmented.

[0017] In certain embodiments, the computer implemented method further comprises comparing movement metrics for single joint or multi-joint movements of the subject to reference value ranges for a normal age-matched control subject. In some embodiments, the movement metrics are for finger-tapping or gait speed.

[0018] In certain embodiments, the computer implemented method further comprises: receiving a third video of the subject’s face; segmenting the third video to generate a plurality of video segments; identifying face landmarks in the plurality of video segments; using the face landmarks to extract a set of face movement features from the plurality of video segments based on relative movement between the face landmarks with time; using a machine learning algorithm to determine symptom severity of the subject based on classification of the set of face movement features in combination with the set of body movement features and / or hand movement features; and displaying information regarding the symptom severity of the subject. In certain embodiments, the face landmarks comprise a nose, a right eye, a left eye, a right eyelid, a left eyelid, a right eyebrow, a left eyebrow, eye corners, a mouth, an upper lip, a lower lip, a right cheek, a left cheek, a forehead, tongue, jaw, and a chin. In certain embodiments, the set of face movement features is used to detect an abnormal facial movement or an involuntary facial movement associated with a movement disorder. In some embodiments, the abnormal facial movement or the involuntary facial movement is rapid eye blinking, face twitching, face tremor, jaw tremor, lip tremor, lip-smacking, tongue twisting, tongueAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 curling, tongue thrusting, lip puckering, facial drooping, or grimacing. In certain embodiments, the movement disorder is Tardive dyskinesia, facial dystonia, hemifacial spasm, peripheral facial palsy, facial myokymia, facial bradykinesia, blepharospasm, Tourette syndrome, orofacial dyskinesia, brainstem tumor, peripheral neuropathy, chorea, multiple sclerosis, Guillain-Barré syndrome, Parkinson’s disease, congenital facial nerve palsy, acquired facial nerve palsy, Moebius syndrome, Poland syndrome, CHARGE syndrome, Hemifacial microsomia, 22q11.2 deletion syndrome, facial paralysis or paresis, or a psychogenic movement disorder.

[0019] In another aspect, a computer implemented method for determining symptom severity of a subject having a facial movement disorder is provided, the computer performing steps comprising: receiving a video of the subject’s face; segmenting the video to generate a plurality of video segments; identifying face landmarks in the plurality of video segments; using the face landmarks to extract a set of face movement features from the plurality of video segments based on relative movement between the face landmarks with time; using a machine learning algorithm to determine symptom severity of the subject based on classification of the set of face movement features; and displaying information regarding the symptom severity of the subject. In certain embodiments, the face landmarks comprise a nose, a right eye, a left eye, a right eyelid, a left eyelid, a right eyebrow, a left eyebrow, eye corners, a mouth, an upper lip, a lower lip, a right cheek, a left cheek, a forehead, tongue, jaw, and a chin. In certain embodiments, the set of face movement features is used to detect an abnormal facial movement or an involuntary facial movement associated with the facial movement disorder. In some embodiments, the abnormal facial movement or the involuntary facial movement is rapid eye blinking, face twitching, face tremor, jaw tremor, lip tremor, lip-smacking, tongue twisting, tongue curling, tongue thrusting, lip puckering, facial drooping, or grimacing. In certain embodiments, the facial movement disorder is Tardive dyskinesia, facial dystonia, hemifacial spasm, peripheral facial palsy, facial myokymia, facial bradykinesia, blepharospasm, Tourette syndrome, orofacial dyskinesia, brainstem tumor, peripheral neuropathy, chorea, multiple sclerosis, Guillain-Barré syndrome, Parkinson’s disease, congenital facial nerve palsy, acquired facial nerve palsy, Moebius syndrome, Poland syndrome, CHARGE syndrome, Hemifacial microsomia, 22q11.2 deletion syndrome, facial paralysis or paresis, or a psychogenic movement disorder.

[0020] In certain embodiments, the computer implemented method further comprises using face landmarks for facial recognition, head pose estimation, facial expression detection, or facial emotion recognition. In some embodiments, the facial emotion recognition is used to detect depression, psychosis, fatigue, fear, anger, sadness, or joy.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0

[0021] In another aspect, a non-transitory computer-readable medium comprising program instructions that, when executed by a processor in a computer, causes the processor to perform a method for determining symptom severity of a subject, described herein, is provided.

[0022] In another aspect, a kit comprising the non-transitory computer-readable medium, described herein, and instructions for determining symptom severity of a subject having a movement disorder is provided.

[0023] In another aspect, a system for determining symptom severity of a subject having a movement disorder is provided, the system comprising: a video recording device; a processor, wherein the processor is programmed to determine symptom severity of a subject having a movement disorder according to a computer implemented method, described herein; and a display component that displays information regarding the symptom severity of the subject.

[0024] In certain embodiments, the video recording device is provided by a digital camera, a smartphone, a tablet, a laptop, or a camcorder.

[0025] In certain embodiments, the video recording device comprises a laser imaging, detection, and ranging (LIDAR) sensor.

[0026] In certain embodiments, the system further comprises a tripod, wherein the video recording device is mounted on the tripod.

[0027] In certain embodiments, the system further comprises one or more additional recording devices, wherein each additional video recording device records videos of the subject from a different distance or angle.

[0028] In certain embodiments, the display further displays body movement features comprising the relative power in 0.5 Hz to 1 Hz of ankle separation, standard deviation (STD) of ankle-hip distance, relative power in 2 Hz to 4 Hz of neck angle, relative power in 4 Hz to 6 Hz of arm-body angular velocity, and mean of neck angle.

[0029] In certain embodiments, the display further displays hand movement features comprising the half-power frequency in pinky finger movement speed, relative power in 2 Hz to 4 Hz of finger tapping speed, and median absolute deviation (MAD) of finger tapping amplitude.

[0030] In certain embodiments, the display further displays information comparing movement metrics for single joint or multi-joint movements of the subject to reference value ranges for a normal age-matched control subject. In some embodiments, the movement metrics are for finger-tapping or gait speed.

[0031] In another aspect, a method of using the system, described herein, for determining symptom severity of a subject having a movement disorder is provided, the method comprising: recording aAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 first video of the subject’s entire body while the subject is walking and / or a second video of a hand of the subject while the subject is finger-tapping using a video recording device; and determining symptom severity of the subject having the movement disorder according to the computer implemented method described herein.

[0032] In certain embodiments, only one hand of the subject is actively performing finger-tapping and the subject’s other hand is resting.

[0033] In certain embodiments, the body landmarks comprise a nose, a right shoulder, a left shoulder, a right elbow, a left elbow, a right wrist, a left wrist, a right hip, a left hip, a right knee, a left knee, a right ankle, and a left ankle.

[0034] In certain embodiments, the hand landmarks comprise a wrist, a thumb, an index finger, a middle finger, a ring finger, a pinky finger, base of the index finger, and base of the pinky finger.

[0035] In certain embodiments, the body movement features comprise: neck angle; arm-body angle; distance between left wrist and left shoulder and / or distance between right wrist and right shoulder; distance between left ankle and hip; horizontal displacement between ankles; horizontal displacement between knees; horizontal displacement between mid-shoulder and mid-hip; vertical displacement between ankles; vertical displacement between shoulders; and vertical displacement between hips.

[0036] In certain embodiments, the hand movement features comprise: distance between thumb and index finger of active hand that is finger tapping; distance between middle finger and wrist of active hand that is finger tapping; distance between ring finger and wrist of active hand that is finger tapping; and distance between pinky finger and wrist of active hand that is finger tapping.

[0037] In certain embodiments, the body movement features comprise relative power in 0.5 Hz to 1 Hz of ankle separation, standard deviation (STD) of ankle-hip distance, relative power in 2 Hz to 4 Hz of neck angle, relative power in 4 Hz to 6 Hz of arm-body angular velocity, and mean of neck angle.

[0038] In certain embodiments, the hand movement features comprise half-power frequency in pinky finger movement speed, relative power in 2 Hz to 4 Hz of finger tapping speed, and median absolute deviation (MAD) of finger tapping amplitude.

[0039] In certain embodiments, the set of body movement features and / or hand movement features are identified by performing sparsity-inducing feature selection on the plurality of video segments.

[0040] In certain embodiments, the movement disorder is Parkinson’s disease or progressive supranuclear palsy.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0

[0041] In certain embodiments, the machine learning algorithm uses a logistic regression (LR) classifier, a support vector machine (SVM) classifier, or a linear discriminant analysis (LDA) classifier. The machine learning algorithm may be supervised, semi-supervised, or unsupervised. In some embodiments, the machine learning algorithm uses artificial neural networks. In some embodiments, the machine learning algorithm uses a deep learning algorithm, which may include the use of convolutional neural networks, deep neural networks, recurrent neural networks, deep residual neural networks, long short-term memory networks, deep belief networks, multilayer perceptrons, or deep reinforcement learning, and the like.

[0042] In certain embodiments, each video segment shows multiple strides during the walking or multiple finger-tapping cycles during the finger-tapping.

[0043] In certain embodiments, each video segment is 3 to 8 seconds in length.

[0044] In certain embodiments, any video segment less than 3 seconds in length is discarded.

[0045] In certain embodiments, any video segment greater than 8 seconds in length is further segmented.

[0046] In certain embodiments, the method further comprises assessing one or more motor and non-motor symptoms of the subject using a Movement Disorder Society-Sponsored Revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS), a Hoehn and Yahr (HnY) scale, or a Montreal Cognitive Assessment (MoCA) scale.

[0047] In certain embodiments, the method further comprises comparing movement metrics for single joint or multi-joint movements of the subject to reference value ranges for a normal age- matched control subject. In some embodiments, the movement metrics are for finger-tapping or gait speed.

[0048] In certain embodiments, the video recording device is provided by a digital camera, a smartphone, a tablet, a laptop, or a camcorder.

[0049] In certain embodiments, the video recording device comprises a LIDAR sensor.

[0050] In certain embodiments, the video recording device is mounted on a tripod.

[0051] In certain embodiments, each image in the first video or the second video has at least 1920 pixels horizontally and at least 1080 pixels vertically.

[0052] In certain embodiments, the video recording device shoots at least 30 frames per second.

[0053] In certain embodiments, the subject walks toward the video recording device and away from the video recording device for a minimum distance of 10 meters in each direction during said recording of the first video.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0

[0054] In certain embodiments, the first video and the second video are single-view videos shot with a single video recording device.

[0055] In certain embodiments, the first video and the second video are multi-view videos shot with at least two video recording devices.

[0056] In certain embodiments, the method is performed before and after administering a treatment to the subject for the movement disorder to evaluate efficacy of the treatment. In some embodiments, the method further comprises altering the treatment of the subject for the movement disorder if the subject shows increased symptom severity based on said classification of the set of body movement features and / or hand movement features.

[0057] In certain embodiments, the method is performed when the subject is at home or when the subject is at a hospital.

[0058] In certain embodiments, the method further comprises placing fiducial markers on the subject for said identifying body landmarks and / or hand landmarks.

[0059] In certain embodiments, the method further comprises: recording a third video of the subject’s face; segmenting the third video to generate a plurality of video segments; identifying face landmarks in the plurality of video segments; using the face landmarks to extract a set of face movement features from the plurality of video segments based on relative movement between the face landmarks with time; and using a machine learning algorithm to determine symptom severity of the subject based on classification of the set of face movement features in combination with the set of body movement features and / or hand movement features.

[0060] In certain embodiments, the face landmarks comprise a nose, a right eye, a left eye, a right eyelid, a left eyelid, a right eyebrow, a left eyebrow, eye corners, a mouth, an upper lip, a lower lip, a right cheek, a left cheek, a forehead, tongue, jaw, and a chin.

[0061] In certain embodiments, the set of face movement features is used to detect an abnormal facial movement or an involuntary facial movement associated with a movement disorder.

[0062] In certain embodiments, the abnormal facial movement or the involuntary facial movement is rapid eye blinking, face twitching, face tremor, jaw tremor, lip tremor, lip-smacking, tongue twisting, tongue curling, tongue thrusting, lip puckering, facial drooping, or grimacing.

[0063] In certain embodiments, the movement disorder is Tardive dyskinesia, facial dystonia, hemifacial spasm, peripheral facial palsy, facial myokymia, facial bradykinesia, blepharospasm, Tourette syndrome, orofacial dyskinesia, brainstem tumor, peripheral neuropathy, chorea, multiple sclerosis, Guillain-Barré syndrome, Parkinson’s disease, congenital facial nerve palsy, acquired facial nerve palsy, Moebius syndrome, Poland syndrome, CHARGE syndrome, HemifacialAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 microsomia, 22q11.2 deletion syndrome, facial paralysis or paresis, or a psychogenic movement disorder.

[0064] In certain embodiments, the method further comprises using the face landmarks for facial recognition, head pose estimation, facial expression detection, or facial emotion recognition. In some embodiments, the facial emotion recognition is used to detect depression, psychosis, fatigue, fear, anger, sadness, or joy.

[0065] In certain embodiments, recording the first video, the second video, or the third video is performed by the subject, a caregiver, or a physician.

[0066] In certain embodiments, the video recording device is a smartphone.

[0067] In certain embodiments, recording the first video, the second video, or the third video is performed by the smartphone automatically whenever the subject uses the smartphone.

[0068] In another aspect, a computer implemented method for diagnosing a movement disorder in a subject is provided, the computer performing steps comprising: receiving a first video of the subject’s entire body while the subject is walking and / or receiving a second video of a hand of the subject while the subject is finger-tapping; segmenting the first video and / or the second video to generate a plurality of video segments; identifying body landmarks and / or hand landmarks in the plurality of video segments; using the body landmarks and / or hand landmarks to extract a set of body movement features and / or hand movement features from the plurality of video segments based on relative movement between the landmarks with time; using a machine learning algorithm to detect a symptom of the movement disorder, if present, based on classification of the set of body movement features and / or hand movement features; and displaying information regarding the diagnosis of the subject based on whether the symptom of the movement disorder is detected.

[0069] In another aspect, a system for diagnosing a movement disorder is provided, the system comprising: a video recording device; a processor, wherein the processor is programmed to determine symptom severity of a subject having a movement disorder according to a computer implemented method for diagnosing a movement disorder, as described herein; and a display component that displays information regarding the diagnosis of the subject.

[0070] In another aspect, a method of using the system for diagnosing a movement disorder in a subject is provided, the method comprising: recording a first video of the subject’s entire body while the subject is walking and / or a second video of a hand of the subject while the subject is finger- tapping using the video recording device; and detecting whether the subject has a symptom of the movement disorder according to the computer implemented method diagnosing a movement disorder, as described herein; and diagnosing the subject with the movement disorder if the symptomAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 of the movement disorder is detected. In some embodiments, the method further comprises treating the subject for the movement disorder if the subject has a positive diagnosis for the movement disorder based on detection of the symptom. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] FIGS.1A-1C. Schematic overview of automatic feature extraction fromvideo recordings for classification of Parkinson’s disease motor symptom severity. (FIG. 1A) From recordings of participants performing prescribed motor assessment tasks, we extracted movement (kinematic) time series at key landmarks using the pose estimation library MediaPipe. Relative movement measurements were computed based on the extracted signals, from which various temporal and spectral metrics were computed as features. Pairwise combinations of samples from the same patient under the same medication state were performed on the body and hand feature sets to form the bimodality combination feature set. (FIG. 1B) To obtain objective measurements of classification performances, we performed a leave-one-subject-out cross-validation (CV), where samples from one patient are held out as the training set for each CV iteration, on each dataset. The CV process is repeated 16 times to account for variabilities in models trained. During CV, a small subset of features with high predictive power with regard to the assigned group labels was selected via least absolute shrinkage and selection operator (LASSO) feature selection based on the training set. The fitted feature selection is then applied to both the training and validation sets. The selected features of the training and validation sets were used to train and validate, respectively, the various machine learning (ML) models. (FIG.1C) LASSO feature selection consistently identified small sets of salient features during each CV iteration. The average number of features selected at each iteration is reported as mean ± standard deviation.

[0072] FIGS.2A-2B. Classification performances of seven selected ML classification models. SVM support vector machine, LR logistic regression, LDA linear discriminant analysis, RF random forest, AB adaptive-boosted trees, KNN K-nearest neighbors, GNB Gaussian naive Bayes ROC AUC area under receiver operating characteristics curve. (FIG.2A) Here we show the average ROC curves of the trained ML models during 16-times repeated leave-on-subject-out cross-validation (CV). The corresponding AUC scores are reported in the right margins as mean ± standard deviation. Overall, integrating body and hand features led to sustained or mildly improved performances in most models. (FIG.2B) Here we show histograms of the classification accuracies of all models aggregated over all CV iterations. The accuracy scores are reported as mean ± standard deviation at the base of the histogram bars. Similar to the findings when reading the ROC plots, models trained with combinedAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 features had comparable accuracies with those trained only on body features, which were more accurate than those trained only on hand features.

[0073] FIGS. 3A-3C. Group differences in stable features and their contribution to ML model predictions. STD standard deviation, MAD median absolute deviation. The stable features were chosen according to the criteria that they must be selected in at least 50% of all models trained during cross-validation (CV). (FIG.3A) Point plot showing the statistically significant differences (p≪0.01) in group means for the selected features between groups with low and high levels of motor impairment. The feature values are z-score normalized. (FIG.3B) The swarm plots on the left show individual SHAP values of feature values encountered during CV training of a logistic regression (LR) classifier. The histograms on the right show the mean absolute SHAP values of each feature. A higher SHAP value indicates a stronger bias toward predicting the “less severe” class label. A higher mean absolute SHAP value for each feature corresponds to its level of feature attribution in relation to the predicted label. (FIG.3C) Visualizations of landmark relationships from which kinematic metrics were derived. The body and hand skeletons also show important landmarks and other landmark relationships were based on in our extraction pipeline.

[0074] FIGS. 4A-4B. Scatter plots comparing 2D projections of datasets with and without stable feature selection. The projections were computed using MDS (multidimensional scaling). (FIG.4A) Poor separability of class clusters in 2D projection space when all features in the datasets were used. (FIG.4B) Improved separation (especially when both body and hand features were included) of class clusters when only stable features were used.

[0075] FIGS.5A-5L. Histograms showing distributions of clinical and demographic characteristics between low and high motor symptom severity groups according to diagnosis (FIG.5A), age (FIG. 5B), education (FIG.5C), female gender (FIG.5D), cognitive profile (MoCA) (FIG.5E), handedness (FIG.5F), disease duration (FIG.5G), UPDRS part I score (FIG.5H), UPDRS part II score (FIG.5I), (FIG.4J), Hoehn and Yahr scale (FIG.5K), and on or off levodopa (FIG.5L).

[0076] FIGS.6A-6D. UPDRS Part III subscores with significant differences between PD and PSP patients of the same severity group. Patient subscores are shown for (FIG.6A) arising from a chair, (FIG.6B) gait, (FIG.6C) rest tremor amplitude, and (FIG.6D) constancy of rest tremor. LUE = left upper extremities. Both PD and PSP patients belong to the dichotomized group with higher motor impairment. Significant differences (^ < 0.05) in sample means is determined via a Permutation Test (100,000 repetitions).

[0077] FIGS.7A-7C. Histograms showing distributions of video segment statistics. (FIGS.7A-7B) Time segment counts per recording and their durations are roughly normally distributed. (FIG.7C)Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 There are no statistically significant (^ < 0.05) differences in video segment duration by clinical states in either segments derived from body recordings (top) or segments derived from hand recordings (bottom). Sample means are compared using a two-sample t-test.

[0078] FIG.8. Average confusion matrices for selected classifiers. Numbers in the confusion matrices represent proportions of classification results that are: (top left) true positive; (top right) false negative; (bottom left) false positive; (bottom right) true negative. Compared to classifiers trained with gait features, classifiers trained with finger-tapping features misclassified the high motor impairment class more. Classifiers trained with both gait and finger-tapping features had more confidence and accuracy when predicting the motor impairment labels. LDA = linear discriminant analysis; LR = logistic regression; SVM = support vector machine; RF = random forest; AB = adaptive-boosted trees; KNN = K-nearest neighbors; GNB = Gaussian naive Bayes.

[0079] FIGS.9A-9B. Classification performances of PD samples only, including (FIG.9A) ROC AUC and (FIG.9B) classification accuracy. SVM = support vector machine; LR = logistic regression; LDA = linear discriminant analysis; RF = random forest; AB = adaptive-boosted trees; KNN = K-nearest neighbors; GNB = Gaussian naive Bayes; ROC AUC = area under receiver operating characteristics curve. For the body feature dataset, we dichotomized the samples into 65 low- and 41 high-severity groups. For the hand feature dataset, we dichotomized the samples into 83 low- and 79 high-severity groups. For the combined dataset, there are 417 low- and 334 high-severity samples.

[0080] FIG.10. Average relative PSDs of arm-body lateral angular velocity of all participants. The shaded areas correspond to ± 1 standard deviation.

[0081] FIGS. 11A-11F. Pinky movement features of at-rest hand show no significant difference between low and high motor symptom severity groups in low frequency (< 4 Hz) regimes for spectral relative power of 0.5 to 1 (FIG.11A), 1 to 2 (FIG.11B), or 2 to 4 (FIG.11C). Two-sided Mann-Whitney U test; significance threshold ^ = 0.05. Movement features for other digits of the resting hand also showed no statistically significant differences between the severity groups in these regimes for spectral relative power of 0.5 to 1 (FIG.11D), 1 to 2 (FIG.11E), 2 to 4 (FIG.11F). These results suggest that the main contributor to differential high frequency pinky movement patterns between PD severity states is likely not dyskinesia but intrinsic sharp movements due to loss of movement fluidity.

[0082] FIG.12. Body and hand skeletons with key landmarks used in analysis labeled. For simplicity, we only labeled relevant landmarks on one side of the body and one of the hands. Some labels were renamed and differed from the official nomenclatures provided by MediaPipe.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 DETAILED DESCRIPTION OF THE INVENTION

[0083] Devices, systems, software, and methods are provided for determining symptom severity of a subject having a movement disorder. The methods utilize video-based objective symptom quantification with classification of severity of symptoms using a machine learning algorithm. The subject methods can be used to diagnose and quantify motor symptom progression in patients with movement disorders such as Parkinson’s disease, which is useful for assessing and optimizing therapeutic interventions, such as dopaminergic medications and deep brain stimulation (DBS).

[0084] Before the present devices, systems, software, and methods are described, it is to be understood that this invention is not limited to the particular devices, systems, software, and methods described, as such may, of course, vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting, since the scope of the present invention will be limited only by the appended claims.

[0085] Where a range of values is provided, it is understood that each intervening value, to the tenth of the unit of the lower limit unless the context clearly dictates otherwise, between the upper and lower limits of that range is also specifically disclosed. Each smaller range between any stated value or intervening value in a stated range and any other stated or intervening value in that stated range is encompassed within the invention. The upper and lower limits of these smaller ranges may independently be included or excluded in the range, and each range where either, neither or both limits are included in the smaller ranges is also encompassed within the invention, subject to any specifically excluded limit in the stated range. Where the stated range includes one or both of the limits, ranges excluding either or both of those included limits are also included in the invention.

[0086] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can be used in the practice or testing of the present invention, some potential and preferred methods and materials are now described. All publications mentioned herein are incorporated herein by reference to disclose and describe the methods and / or materials in connection with which the publications are cited. It is understood that the present disclosure supersedes any disclosure of an incorporated publication to the extent there is a contradiction.

[0087] As will be apparent to those of skill in the art upon reading this disclosure, each of the individual embodiments described and illustrated herein has discrete components and features which may be readily separated from or combined with the features of any of the other severalAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 embodiments without departing from the scope or spirit of the present invention. Any recited method can be carried out in the order of events recited or in any other order which is logically possible.

[0088] It must be noted that as used herein and in the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "an image" includes a plurality of such images and reference to "the video" includes reference to one or more videos and equivalents thereof, such as visual recordings, films, and footage (e.g., containing a series of moving images of a subject), known to those skilled in the art, and so forth.

[0089] The publications discussed herein are provided solely for their disclosure prior to the filing date of the present application. Nothing herein is to be construed as an admission that the present invention is not entitled to antedate such publication by virtue of prior invention. Further, the dates of publication provided may be different from the actual publication dates which may need to be independently confirmed. Definitions

[0090] The term "about," particularly in reference to a given quantity, is meant to encompass deviations of plus or minus five percent.

[0091] The term "movement disorder" refers to any type of neurological disorder that causes either increased movements or reduced or slow movements. Movement disorders include, but are not limited to, Parkinson's disease, parkinsonism, progressive supranuclear palsy, ataxia, cervical dystonia, chorea, dystonia, functional movement disorder, Huntington's disease, multiple system atrophy, myoclonus, tardive dyskinesia, Tourette syndrome, tremor, restless legs syndrome, and Wilson's disease. Symptoms may include, but art not limited to, tremor, involuntary movements, slowness of movement (bradykinesia), rigidity, postural instability, twisting movements, poor balance, irregularity of movements, stumbling, and difficulty with walking. In some cases, a movement disorder is caused by genetic and / or environmental factors, head trauma, infections, inflammation, metabolic disturbances, toxins, adverse reactions to medications, or stressful life events.

[0092] The term also includes movement disorders that causes abnormal facial movement, involuntary facial movement, or paralysis or partial paralysis of facial muscles. Facial movement disorders include, but are not limited to, Tardive dyskinesia, facial dystonia, hemifacial spasm, peripheral facial palsy, facial myokymia, facial bradykinesia, blepharospasm, Tourette syndrome, orofacial dyskinesia, brainstem tumor, peripheral neuropathy, chorea, multiple sclerosis, Guillain-Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 Barré syndrome, Parkinson’s disease, congenital facial nerve palsy, acquired facial nerve palsy, Moebius syndrome, Poland syndrome, CHARGE syndrome, hemifacial microsomia, 22q11.2 deletion syndrome, facial paralysis or paresis, and psychogenic movement disorders. Symptoms of facial movement disorders may include, but are not limited to, rapid eye blinking, face twitching, face tremor, jaw tremor, lip tremor, lip-smacking, tongue twisting, tongue curling, tongue thrusting, lip puckering, facial drooping, or grimacing.

[0093] The terms “individual”, “subject”, “recipient”, and “patient” are used interchangeably herein and refer to any mammalian subject for whom diagnosis, treatment, or therapy is desired, particularly humans. "Mammal" for purposes of treatment refers to any animal classified as a mammal, including human and non-human mammals such as non-human primates, including chimpanzees and other apes and monkey species; laboratory animals such as mice, rats, rabbits, hamsters, guinea pigs, and chinchillas; domestic animals such as dogs and cats; and farm animals such as sheep, goats, pigs, horses and cows.

[0094] The term “user” as used herein refers to a person that interacts with a device and / system disclosed herein for performing one or more steps of the presently disclosed methods. The user may be the patient being diagnosed or receiving treatment for a movement disorder. The user may be a health care practitioner, such as the patient’s physician.

[0095] The terms "treatment", "treating", "treat" and the like are used herein to generally refer to obtaining a desired pharmacologic and / or physiologic effect. The effect can be prophylactic in terms of completely or partially preventing a disease or symptom(s) thereof and / or may be therapeutic in terms of a partial or complete stabilization or cure for a disease and / or adverse effect attributable to the disease. The term “treatment" encompasses any treatment of a disease in a mammal, particularly a human, and includes: (a) preventing the disease and / or symptom(s) from occurring in a subject who may be predisposed to the disease or symptom but has not yet been diagnosed as having it; (b) inhibiting the disease and / or symptom(s), i.e., arresting their development; or (c) relieving the disease symptom(s), i.e., causing regression of the disease and / or symptom(s). Those in need of treatment include those already inflicted (e.g., those with a movement disorder) as well as those in which prevention is desired those with a genetic predisposition to developing a movement disorder, those with increased susceptibility to developing a movement disorder, those suspected of having a movement disorder, etc.).

[0096] A therapeutic treatment is one in which the subject is inflicted prior to administration and a prophylactic treatment is one in which the subject is not inflicted prior to administration. In some embodiments, the subject has an increased likelihood of becoming inflicted or is suspected of beingAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 inflicted prior to treatment. In some embodiments, the subject is suspected of having an increased likelihood of becoming inflicted.

[0097] A "therapeutically effective dose" or “therapeutic dose” is an amount sufficient to effect desired clinical results (i.e., achieve therapeutic efficacy). A therapeutically effective dose can be administered in one or more administrations.

[0098] "Pharmaceutically acceptable excipient or carrier" refers to an excipient that may optionally be included in the compositions of the invention and that causes no significant adverse toxicological effects to the patient.

[0099] "Pharmaceutically acceptable salt" includes, but is not limited to, amino acid salts, salts prepared with inorganic acids, such as chloride, sulfate, phosphate, diphosphate, bromide, and nitrate salts, or salts prepared from the corresponding inorganic acid form of any of the preceding, e.g., hydrochloride, etc., or salts prepared with an organic acid, such as malate, maleate, fumarate, tartrate, succinate, ethylsuccinate, citrate, acetate, lactate, methanesulfonate, benzoate, ascorbate, para-toluenesulfonate, palmoate, salicylate and stearate, as well as estolate, gluceptate and lactobionate salts. Similarly salts containing pharmaceutically acceptable cations include, but are not limited to, sodium, potassium, calcium, aluminum, lithium, and ammonium (including substituted ammonium).

[0100] "Diagnosis" as used herein generally includes determination as to whether a subject is likely affected by a given disease, disorder or dysfunction. The skilled artisan often makes a diagnosis on the basis of one or more diagnostic indicators, i.e., a biomarker, the presence, absence, or amount of which is indicative of the presence or absence of the disease, disorder or dysfunction.

[0101] "Prognosis" as used herein generally refers to a prediction of the probable course and outcome of a clinical condition or disease. A prognosis of a patient is usually made by evaluating factors or symptoms of a disease that are indicative of a favorable or unfavorable course or outcome of the disease. It is understood that the term "prognosis" does not necessarily refer to the ability to predict the course or outcome of a condition with 100% accuracy. Instead, the skilled artisan will understand that the term "prognosis" refers to an increased probability that a certain course or outcome will occur; that is, that a course or outcome is more likely to occur in a patient exhibiting a given condition, when compared to those individuals not exhibiting the condition.

[0102] The term “responsive” as used herein means that the treatment is having the desired effect such as reducing symptom severity caused by a movement disorder. When the individual does not improve in response to the treatment, it may be desirable to seek a different therapy or treatment regime for the individual.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0

[0103] The term “continue treatment” (i.e., continue therapy) is used herein to mean that the current course of treatment (e.g., continued administration of a therapy) is to continue. If the current course of treatment is not effective in treating the movement disorder, the treatment may be altered. “Altering therapy” is used herein to mean “discontinuing therapy” or “changing the therapy” (e.g., changing the type of treatment, changing the particular dose and / or frequency of administration of medication, e.g., increasing the dose and / or frequency). In some cases, therapy can be altered until the individual is deemed to be responsive. In some embodiments, altering therapy means changing which type of treatment is administered, discontinuing a particular treatment altogether, etc.

[0104] “Providing an analysis” is used herein to refer to the delivery of an oral or written analysis (i.e., a document, a report, etc.). A written analysis can be a printed or electronic document. A suitable analysis (e.g., an oral or written report) provides any or all of the following information: identifying information of the subject (name, age, etc.), a description of video recordings of the subject’s entire body while walking and / or a hand while finger-tapping, and / or face during abnormal or involuntary facial movements, assessment of the symptom severity of the subject based on classification of the set of body movement features and / or hand movement features and / or facial movements from the video recordings, assessment of mood or facial emotion based on facial expression detection or facial emotion recognition analysis of the video recordings of the subject’s face, a recommendation for treatment, and / or to continue or alter therapy, a recommended strategy for additional therapy, etc. The report can be in any format including, but not limited to printed information on a suitable medium or substrate (e.g., paper); or electronic format. If in electronic format, the report can be in any computer readable medium, e.g., diskette, compact disk (CD), flash drive, and the like, on which the information has been recorded. In addition, the report may be present as a website address which may be used via the internet to access the information at a remote site. Video-Based Methods of Determining Symptom Severity

[0105] The ability to readily quantify motor symptom progression in patients with movement disorders will greatly aid in assessing and optimizing therapeutic interventions, including prescription of appropriate medications and deep brain stimulation (DBS). The subject methods utilize videos of full-body movement and / or hand movement and / or facial movement together with a machine learning algorithm to automatically determine motor symptom severity in patients with movement disorders. No expensive specialized equipment is required. The videos can be recorded on consumer-grade devices such as smartphones and digital cameras, which are relatively inexpensive and readily available for conventional use. Furthermore, the subject methods have the advantage ofAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 being able to extract meaningful features from only a few seconds of video snippets. The methods, disclosed herein, are capable of quantifying clinically meaningful motor features useful for both machine learning predictions and clinical analysis.

[0106] In one aspect, a method of determining symptom severity of a subject having a movement disorder is provided, the method comprising: recording a first video of the subject’s entire body while the subject is walking and / or a second video of a hand of the subject while the subject is finger- tapping using a video recording device; segmenting the first video and / or the second video to generate a plurality of video segments; identifying body landmarks and / or hand landmarks in the plurality of video segments; using the body landmarks and / or hand landmarks to extract a set of body movement features and / or hand movement features from the plurality of video segments; and using a machine learning algorithm to determine symptom severity of the subject based on classification of the set of body movement features and / or hand movement features.

[0107] In some embodiments, the machine learning algorithm uses a logistic regression (LR) classifier, a support vector machine (SVM) classifier, or a linear discriminant analysis (LDA) classifier. The machine learning algorithm may be supervised, semi-supervised, or unsupervised. In some embodiments, the machine learning algorithm uses artificial neural networks. In some embodiments, the machine learning algorithm uses a deep learning algorithm, which may include the use of convolutional neural networks, deep neural networks, recurrent neural networks, deep residual neural networks, long short-term memory networks, deep belief networks, multilayer perceptrons, or deep reinforcement learning, and the like. For a description of deep learning algorithms, see, e.g., Pedrycz et al. Deep Learning: Algorithms and Applications (Studies in Computational Intelligence Book 865, Springer, 2019), Goodfellow et al. Deep Learning (Adaptive Computation and Machine Learning series, The MIT Press, 2016), and Various Deep Learning Algorithms in Computational Intelligence (edited by Oscar Humberto Montiel Ross, Mdpi AG, 2023); herein incorporated by reference in their entireties.

[0108] By "movement disorder" is meant any type of neurological disorder that causes either increased movements or reduced or slow movements. Movement disorders include, but are not limited to, Parkinson's disease, parkinsonism, progressive supranuclear palsy, ataxia, cervical dystonia, chorea, dystonia, functional movement disorder, Huntington's disease, multiple system atrophy, myoclonus, tardive dyskinesia, Tourette syndrome, tremor, restless legs syndrome, and Wilson's disease. In some cases, a movement disorder is caused by head trauma, infections, inflammation, metabolic disturbances, toxins, or adverse reactions to medications.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0

[0109] For body recordings, the subject should be fully standing and roughly facing forward or backward relative to the video recording device. To verify that the subject is standing, video segments can be checked to determine that the apparent lower leg length (ankle-knee distance) is at most twice the apparent upper leg length (hip-knee distance). In certain embodiments, the subject walks toward the video recording device and away from the video recording device for a minimum distance of 10 meters in each direction during the recording of a full-body walking / gait video.

[0110] For recording of the hand video, preferably only one hand is raised and engaged in active finger-tapping while the other hand is resting. In some embodiments, the subject taps their index finger on their thumb at least ten times with maximally achievable speed and amplitude. The resting hand should have little to no movement and can have its time series discarded. Having only one hand actively finger-tapping simplifies analysis by eliminating the need to distinguish between left and right-sided hand movements.

[0111] The method includes segmenting full-length body / hand videos into video segments for subsequent feature extraction and analysis. Each video segment that is retained should show multiple strides during walking or multiple finger-tapping cycles during finger-tapping of the subject. In certain embodiments, each video segment is 3 to 8 seconds in length, including any number of seconds within this range such as 3 seconds, 4 seconds, 5 seconds, 6 seconds, 7 seconds, or 8 seconds. In some embodiments, any video segment less than 3 seconds in length is discarded. In some embodiments, any video segment greater than 8 seconds in length is further segmented.

[0112] In certain embodiments, the set of body movement features and / or hand movement features used in analysis of symptom severity are identified by performing sparsity-inducing feature selection on the plurality of video segments. This type of analysis can be used to identify the features that are most important for severity prediction among the body, hand, and combined features for a particular patient.

[0113] Body landmarks include, without limitation, a nose, a right shoulder, a left shoulder, a right elbow, a left elbow, a right wrist, a left wrist, a right hip, a left hip, a right knee, a left knee, a right ankle, and a left ankle. In some embodiments, the body movement features comprise: the neck angle; the arm-body angle; the distance between the left wrist and the left shoulder and / or the distance between the right wrist and the right shoulder; the distance between the left ankle and the hip; the horizontal displacement between the ankles; the horizontal displacement between the knees; the horizontal displacement between the mid-shoulder and the mid-hip; the vertical displacement between the ankles; the vertical displacement between the shoulders; and the vertical displacement between the hips. In some embodiments, the body movement features comprise the relative powerAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 in 0.5 Hz to 1 Hz of ankle separation, standard deviation (STD) of ankle-hip distance, relative power in 2 Hz to 4 Hz of neck angle, relative power in 4 Hz to 6 Hz of arm-body angular velocity, and mean of neck angle.

[0114] Hand landmarks include, without limitation, a wrist, a thumb, an index finger, a middle finger, a ring finger, a pinky finger, base of the index finger, and base of the pinky finger. In certain embodiments, the hand movement features comprise: the distance between the thumb and the index finger of the active hand that is finger tapping; the distance between the middle finger and the wrist of the active hand that is finger tapping; the distance between the ring finger and the wrist of the active hand that is finger tapping; and the distance between the pinky finger and the wrist of the active hand that is finger tapping. In some embodiments, the hand movement features comprise the half-power frequency in pinky finger movement speed, relative power in 2 Hz to 4 Hz of finger tapping speed, and median absolute deviation (MAD) of finger tapping amplitude.

[0115] In some embodiments, a movement disorder causes abnormal facial movement, involuntary facial movement, or paralysis or partial paralysis of facial muscles. Facial movement disorders include, but are not limited to, Tardive dyskinesia, facial dystonia, hemifacial spasm, peripheral facial palsy, facial myokymia, facial bradykinesia, blepharospasm, Tourette syndrome, orofacial dyskinesia, brainstem tumor, peripheral neuropathy, chorea, multiple sclerosis, Guillain-Barré syndrome, Parkinson’s disease, congenital facial nerve palsy, acquired facial nerve palsy, Moebius syndrome, Poland syndrome, CHARGE syndrome, hemifacial microsomia, 22q11.2 deletion syndrome, facial paralysis or paresis, and psychogenic movement disorders.

[0116] For embodiments in which a patient has a movement disorder causing abnormal or involuntary facial movements, the method for determining symptom severity of the subject may further comprises: recording a video of the subject’s face using a video recording device; segmenting the video to generate a plurality of video segments; identifying face landmarks in the plurality of video segments; using the face landmarks to extract a set of face movement features from the plurality of video segments; and using a machine learning algorithm to determine symptom severity of the subject based on classification of the set of face movement features, either by themselves or in combination with body movement features and / or hand movement features.

[0117] Face landmarks include, without limitation, a nose, a right eye, a left eye, a right eyelid, a left eyelid, a right eyebrow, a left eyebrow, eye corners, a mouth, an upper lip, a lower lip, a right cheek, a left cheek, a forehead, tongue, jaw, and a chin. In some embodiments, the face movement features are used to detect a symptom of a facial movement disorder such as, but not limited to, rapid eyeAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 blinking, face twitching, face tremor, jaw tremor, lip tremor, lip-smacking, tongue twisting, tongue curling, tongue thrusting, lip puckering, facial drooping, or grimacing.

[0118] In certain embodiments, the face landmarks are further analyzed using a machine learning algorithm to detect facial expression and / or facial emotion. For example, a machine learning algorithm can be used to classify facial expressions for facial emotion recognition to detect emotional states such as, but not limited to, depression, psychosis, fatigue, fear, anger, sadness, or joy.

[0119] Segmenting of videos of the face for subsequent feature extraction and analysis can be performed similarly to segmenting of full-length body or hand videos as described above. Each face video segment that is retained should show multiple involuntary or abnormal facial movements. In certain embodiments, each video segment is 3 to 8 seconds in length, including any number of seconds within this range such as 3 seconds, 4 seconds, 5 seconds, 6 seconds, 7 seconds, or 8 seconds. In some embodiments, any video segment less than 3 seconds in length is discarded. In some embodiments, any video segment greater than 8 seconds in length is further segmented.

[0120] Any convenient technique may be used for recognizing landmarks in a video of a subject. In some cases, identifying landmarks on the body, hand, or face of a subject may entail utilizing artificial intelligence-based techniques for recognizing features of the subject. Such artificial intelligence- based techniques may comprise any convenient image processing or recognition algorithms, such as those comprising deep learning methods, convolutional neural networks, and the like. For example, the Pose and Hands tracking solutions from MediaPipe (Lugaresi et al. (2019) arxiv- vanity.com / papers / 1906.08172), DeepLabCut (Mathis et al. (2018) Nat. Neurosci.21, 1281-1289), and OpenPose (Cao, et al. (2021) IEEE Trans. Pattern Anal. Mach. Intell.43, 172-186) can be used for identifying landmarks on the body or hand of a subject in a video image. Alternatively, fiducial markers can be attached to a subject and worn during video recordings to identify body landmarks and / or hand landmarks and / or face landmarks in video images.

[0121] The video recording device may be any commercially available, off the shelf, video recording device. In certain embodiments, the video recording device is provided by a commercially available digital camera, smartphone, tablet, laptop, or camcorder. The video recording device is preferably mounted on a tripod to avoid any movement of the video recording device during video recordings of the full body and / or hand of a subject.

[0122] In some embodiments, the video recording device records a series of images at a rate ranging from 24 frames / images per second to 240 frames / images per second, 24 to 100 frames / images per second, 24 to 60 frames / images per second, or 24 to 30 frames / images per second, including any rate within these ranges such as 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44,Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 46, 48, 50, 52, 54, 56, 58, 60, 70, 80, 90, 100, 120, 140, 160, 180, 200, 220, or 240 frames / images per second. In some embodiments, the video recording device records a series of images at a rate of at least 30 frames / images per second.

[0123] In certain embodiments, the video recording device records standard-definition (SD) digital video with a display resolution of at least 720 × 480 (i.e., at least 720 pixels horizontally and at least 480 pixels vertically). In certain embodiments, the video recording device records SD digital video with a display resolution of at least 720 × 576 (i.e., at least 720 pixels horizontally and at least 576 pixels vertically). In certain embodiments, the video recording device records high-definition (HD) digital video with a display resolution of at least 1280 × 720 (i.e., at least 1280 pixels horizontally and at least 720 pixels vertically). In certain embodiments, the video recording device records HD digital video with a display resolution of at least 1920 × 1080 (i.e., at least 1920 pixels horizontally and at least 1080 pixels vertically). In some embodiments, the video recording device records 4K ultra-high- definition (4K UHD) digital video with a display resolution of at least 3840 × 2160 (i.e., at least 3840 pixels horizontally and at least 2160 pixels vertically). In some embodiments, the video recording device records 8K ultra-high-definition (8K UHD) digital video with a display resolution of at least 7680 × 4320 (i.e., at least 7680 pixels horizontally and at least 4320 pixels vertically).

[0124] In certain embodiments, the body, hand, or face videos are single-view videos shot with a single video recording device. In other embodiments, the body, hand, or face videos are multi-view videos shot with at least two video recording devices. In some embodiments, two or more, three or more, four or more, five or more, six or more, seven or more, eight or more, nine or more, or ten or more video recording devices are used to record videos of the subject while walking or finger tapping or during abnormal or involuntary facial movements, wherein each video recording device records videos of the subject’s full-body, hand, or face from a different angle or at a different distance. In embodiments in which more than one video recording device is employed, each of the video recording devices may be oriented to capture different features of the subject. For example, a first video recording device may be configured to record videos of the subject’s full-body, a second video recording device may be configured to record videos of the subject’s hand, and a third video recording device may be configured to record videos of the subject’s face. In other cases, multiple video recording devices may be oriented to capture different views of the same features of the subject. For example, a first video recording device may be configured to record videos of a subject’s body from a front or rear view, and a second video recording device may be configured to record videos of a subject’s body from a side view. In still other cases, multiple video recording devices may be oriented to record videos of the same features of the subject at different levels of zoom. In someAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 cases, a first video recording device may be configured to record videos of the subject’s entire body, and a second video recording device may be configured to record videos of the subject’s hand or face, exclusively. Any combination of different configurations of multiple video recording devices may be employed.

[0125] In certain embodiments, the video recording device comprises a laser imaging, detection, and ranging (LIDAR) sensor. Any type of LIDAR sensor capable of producing images that can be used to measure body, hand, or facial feature movements may be used. LIDAR images can be used, for example, for detection of landmarks, feature tracking, and distance measurements. The LIDAR sensor may scan in a single direction or multiple directions (e.g., 3D laser scanning). Exemplary types of LIDAR sensors include, without limitation, a spindle-type LIDAR, which spins to give a 360- degree view; a solid-state LIDAR having a fixed field of view, which may use either micro-electro- mechanical systems (MEMS) mirror scanning or optical phased arrays to steer the laser beams; and a flash LIDAR, which spreads a flash of light over the field of view before the signal bounces back to a detector. Either a static or a mobile system may be used. For example, the LIDAR system may be mounted on a tripod, a handheld device, or other stationary device. Alternatively, the LIDAR system may be mounted on a moving vehicle. In some embodiments, the LIDAR sensor is provided on a laptop, a smartphone, or a tablet. For a description of LIDAR systems, see, e.g., Gimmestad et al. Lidar Engineering: Introduction to Basic Principles Cambridge University Press, 2023), McManamon LiDAR Technologies and Systems (SPIE--The International Society for Optical Engineering, 2019), Wang et al. (2020) Micromachines (Basel) 11(5):456, Hu et al. (2023) Sensors (Basel) 23(13):5920, Murray et al. (2022) Opt Express 30(15):27164-27170, Huang et al. (2017) Sensors (Basel) 17(9):1932, Bahraini et al. (2019) Sensors (Basel) 19(17):3699; herein incorporated by reference.

[0126] Although videos may be recorded at a hospital, videos may also be conveniently recorded at the home of a patient, or almost anywhere the patient goes and can even be recorded by the patients themselves. In some cases, caregivers or physicians may record the videos or assist the patient in recording the videos. In some embodiments, the videos are recorded by a smartphone automatically whenever a patient uses the smartphone to allow daily capture of motor and non-motor symptoms to allow evaluation of symptom severity from passive phone usage.

[0127] The recorded videos may be transmitted to a processor and analyzed to determine the severity of symptoms according to a computer implemented method, as described further below. In some cases, the severity of symptoms of a movement disorder may be further assessed using a visual analog scale or a verbal rating scale. In certain embodiments, the method further comprisesAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 assessing one or more motor and / or non-motor symptoms of the subject using a Movement Disorder Society-Sponsored Revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS), a Hoehn and Yahr (HnY) scale, or a Montreal Cognitive Assessment (MoCA) scale.

[0128] The methods described herein may be used to determine an appropriate treatment regimen for a patient and whether a patient should be treated for a movement disorder. For example, a patient may be selected for treatment for a movement disorder if the symptoms are severe enough to warrant treatment of the movement disorder based on the assessment of symptom severity using the methods described herein.

[0129] Treatment of Parkinson’s disease may comprise, for example, administering a medication such as, but not limited to, L-DOPA (l-3,4-dihydroxyphenylalanine, also known as levodopa), carbidopa (N-amino-α-methyl-3-hydroxy-L-tyrosine monohydrate), carbidopa-levodopa (Rytary, Sinemet, Duopa), a dopamine agonist, including, without limitation, pramipexole (Mirapex ER), rotigotine, apomorphine (Apokyn), and amantadine (Gocovri); a monoamine oxidase B (MAO-B) inhibitor, including, without limitation, selegiline (Zelapar), rasagiline (Azilect) and safinamide (Xadago); a catechol O-methyltransferase (COMT) inhibitor, including, without limitation, entacapone (Comtan), opicapone (Ongentys), and Tolcapone (Tasmar); an anticholinergic agent, including, without limitation, benztropine (Cogentin) and trihexyphenidyl; an adenosine receptor antagonist, including, without limitation an A2A receptor antagonist such as istradefylline (Nourianz), or an antipsychotic, including, without limitation, nuplazid (Pimavanserin), or any combination thereof. Alternatively or additionally, treatment of Parkinson’s disease may comprise deep brain stimulation or magnetic resonance imaging (MRI)-guided focused ultrasound (MRgFUS), which are typically administered to patients who have advanced Parkinson's disease or who are not responsive to treatment with medication (e.g., levodopa). In some cases, deep brain stimulation is administered to a patient for treatment of tremor that is not responsive or inadequately responsive to treatment with levodopa. MRgFUS uses high intensity focused ultrasonic waves to destroy (ablate) tissue in areas of the brain from which the tremors initiate.

[0130] The determination of symptom severity of a subject having a movement disorder using video- based tracking of clinical symptoms of patients, as described herein, and analysis using a machine learning algorithm to determine severity is an active clinical application of the correlation between body / hand movements and the disease. For example, “determining” requires the active step of reviewing the data, which is produced during the active step(s) of recording videos of the subject’s entire body while the subject is walking and / or a hand of the subject while the subject is finger- tapping, and / or face of the subject while having abnormal or involuntary facial movements;Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 segmenting the videos to generate a plurality of video segments; identifying body landmarks and / or hand landmarks and / or face landmarks in the plurality of video segments; using the body landmarks and / or hand landmarks and / or face landmarks to extract a set of body movement features and / or hand movement features and / or face movement features from the plurality of video segments, and using a machine learning algorithm to determine symptom severity of the subject based on classification of the set of body movement features and / or hand movement features and / or face movement features, as described herein. Additionally, in some cases, a decision is made to proceed with the current treatment (i.e., therapy), or instead to alter the treatment. In some cases, the subject methods include the step of continuing therapy or altering therapy.

[0131] In some embodiments, the video-based methods described herein are used for monitoring symptom severity of a subject having a movement disorder. For example, a first set of body (i.e., while subject is walking) and / or hand (i.e., while the subject is finger-tapping) and / or face videos (i.e., during abnormal or involuntary facial movements) can be recorded of the patient at a first time point, and a second set of body and / or hand, and / or face videos can be recorded of the subject at a second (later) time point. A machine learning algorithm is used to determine symptom severity of the subject at the first time point and the second time point based on classification of a set of body movement features and / or hand movement features and / or face movement features, as described herein, wherein increased severity of the symptoms at the second time point compared to the first time point indicates the subject is worsening, and decreased severity of the symptoms at the second time point compared to the first time point indicates the subject is improving.

[0132] In certain embodiments, the method is performed before and after administering a treatment to the subject for the movement disorder to evaluate efficacy and determine if the subject is responsive to the treatment. The term “responsive” as used herein means that the treatment is having the desired effect i.e., decreasing severity of the symptoms as determined according to the video-based methods described herein. If the symptoms of the individual do not improve in response to the treatment, it may be desirable to seek a different therapy or treatment regimen for the individual.

[0133] The term “continue treatment” (i.e., continue therapy) is used herein to mean that the current course of treatment (e.g., continued administration of a therapy) is to continue. If the current course of treatment is not effective in reducing severity of symptoms of the movement disorder, the treatment may be altered. “Altering therapy” is used herein to mean “discontinuing therapy” or “changing the therapy” (e.g., changing the type of treatment, changing the particular dose and / or frequency of administration of medication, e.g., increasing the dose and / or frequency). In someAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 cases, therapy can be altered until the individual is deemed to be responsive. In some embodiments, altering therapy means changing which type of treatment is administered, discontinuing a particular treatment altogether, etc.

[0134] As a non-limiting illustrative example, a patient may be initially treated for Parkinson’s disease by administering carbidopa-levodopa. Then to “continue treatment” would be to continue with this type of treatment. If the current course of treatment is not effective, the treatment may be altered, e.g., increasing the dose or frequency of administration of the carbidopa-levodopa, adding one or more additional medications such as a dopamine agonist, a MAO-B inhibitor, a COMT inhibitor, an anticholinergic agent, and / or an adenosine receptor antagonist, or changing to a different type of treatment such as deep brain stimulation.

[0135] In other words, the subject may be monitored using the video-based methods, described herein, in order to determine when to continue therapy and / or when to alter therapy. As such, body / hand videos can be recorded of the subject after any of the administrations and the videos can be assayed to determine symptom severity. Accordingly, the subject methods can be used to determine whether an individual being treated for a movement disorder is responsive or is maintaining responsiveness to a treatment.

[0136] The therapy can be administered to an individual any time after a pre-treatment body video and / or hand video and / or face video is recorded of the individual, but it is preferable for the therapy to be administered simultaneous with or as soon as possible (e.g., about 7 days or less, about 3 days or less, e.g., 2 days or less, 36 hours or less, 1 day or less, 20 hours or less, 18 hours or less, 12 hours or less, 9 hours or less, 6 hours or less, 3 hours or less, 2.5 hours or less, 2 hours or less, 1.5 hours or less, 1 hour or less, 45 minutes or less, 30 minutes or less, 20 minutes or less, 15 minutes or less, 10 minutes or less, 5 minutes or less, 2 minutes or less, or 1 minute or less) after a pre-treatment body and / or hand video is recorded.

[0137] In some cases, more than one type of therapy may be administered to the individual. For example, a subject who has Parkinson’s disease may initially be treated with carbidopa-levodopa and a dopamine agonist. Additional medication may be added to the treatment as the disease progresses such as a MAO-B inhibitor, a COMT inhibitor, an anticholinergic agent, or an adenosine receptor antagonist, or a combination thereof. A subject who has more severe disease may be treated more aggressively. For example, treatment of more severe disease may include deep brain stimulation or MRgFUS.

[0138] In some embodiments, the subject methods include providing an analysis indicating the severity of symptoms of a movement disorder and / or whether the individual is determined to have aAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 movement disorder. The analysis may further provide an analysis of whether an individual is responsive or not responsive to a treatment for a movement disorder, or whether the individual is determined to be maintaining responsiveness or not maintaining responsiveness to a treatment for a movement disorder. As described above, an analysis can be an oral or written report (e.g., written or electronic document). The analysis can be provided to the subject, to the subject’s physician, to a testing facility, etc. The analysis can also be accessible as a website address via the internet. In some such cases, the analysis can be accessible by multiple different entities (e.g., the subject, the subject’s physician, a testing facility, etc.). Systems and Computer Implemented Methods

[0139] The present disclosure provides systems and computer implemented methods which find use in practicing the subject methods. In some embodiments, the system may include: a processor programmed to determine symptom severity of a subject having a movement disorder or to diagnose a movement disorder; one or more video recording devices for recording body videos and / or hand videos, and / or face videos, and a display component for displaying information regarding the symptom severity or diagnosis of the subject. The system may also comprise one or more graphic boards for processing and outputting graphical information to the display component. For example, the display may be used to display an image or video of the subject walking or finger-tapping or the face of the subject while having abnormal or involuntary facial movements.

[0140] In some embodiments, a computer implemented method is used for determining symptom severity of a subject having a movement disorder. The processor can be programmed to perform steps of a computer implemented method comprising: receiving a first video of the subject’s entire body while the subject is walking and / or receiving a second video of a hand of the subject while the subject is finger-tapping; segmenting the first video and / or the second video to generate a plurality of video segments; identifying body landmarks and / or hand landmarks in the plurality of video segments; using the body landmarks and / or hand landmarks to extract a set of body movement features and / or hand movement features from the plurality of video segments based on relative movement between the landmarks with time; using a machine learning algorithm to determine symptom severity of the subject based on classification of the set of body movement features and / or hand movement features; and displaying information regarding the symptom severity of the subject.

[0141] Alternatively or additionally, the computer implemented method may comprise: receiving a video of the subject’s face; segmenting the video of the subject’s face to generate a plurality of video segments; identifying face landmarks in the plurality of video segments; using the face landmarks toAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 extract a set of face movement features from the plurality of video segments based on relative movement between the face landmarks with time; using a machine learning algorithm to determine symptom severity of the subject based on classification of the set of face movement features, either by themselves or in combination with a set of body movement features and / or hand movement features; and displaying information regarding the symptom severity of the subject.

[0142] In some embodiments, a computer implemented method is used for diagnosing a movement disorder in a subject. The processor can be programmed to perform steps of a computer implemented method comprising: receiving a first video of the subject’s entire body while the subject is walking and / or receiving a second video of a hand of the subject while the subject is finger-tapping; segmenting the first video and / or the second video to generate a plurality of video segments; identifying body landmarks and / or hand landmarks in the plurality of video segments; using the body landmarks and / or hand landmarks to extract a set of body movement features and / or hand movement features from the plurality of video segments based on relative movement between the landmarks with time; using a machine learning algorithm to detect a symptom of the movement disorder, if present, based on classification of the set of body movement features and / or hand movement features; and displaying information regarding the diagnosis of the subject based on whether the symptom of the movement disorder is detected.

[0143] Alternatively or additionally, the computer implemented method for diagnosing a movement disorder in a subject may comprise: receiving a video of the subject’s face; segmenting the video of the subject’s face to generate a plurality of video segments; identifying face landmarks in the plurality of video segments; using the face landmarks to extract a set of face movement features from the plurality of video segments based on relative movement between the landmarks with time; using a machine learning algorithm to detect a symptom of the movement disorder, if present, based on classification of the set of face movement features; and displaying information regarding the diagnosis of the subject based on whether the symptom of the movement disorder is detected.

[0144] In some embodiments, a computer implemented method further comprises using face landmarks for facial recognition, head pose estimation, facial expression detection, or facial emotion recognition. In some embodiments, a machine learning algorithm is used to classify facial expressions for facial emotion recognition to detect emotional states such as, but not limited to, depression, psychosis, fatigue, fear, anger, sadness, or joy. In certain embodiments, the machine learning algorithm, used in the computer implemented methods, applies a logistic regression (LR) classifier, a support vector machine (SVM) classifier, or a linear discriminant analysis (LDA) classifier. The machine learning algorithm may be supervised,Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 semi-supervised, or unsupervised. In some embodiments, the machine learning algorithm uses artificial neural networks. In some embodiments, the machine learning algorithm uses a deep learning algorithm, which may include the use of convolutional neural networks, deep neural networks, recurrent neural networks, deep residual neural networks, long short-term memory networks, deep belief networks, multilayer perceptrons, or deep reinforcement learning, and the like. See, e.g., Pedrycz et al. Deep Learning: Algorithms and Applications (Studies in Computational Intelligence Book 865, Springer, 2019), Goodfellow et al. Deep Learning (Adaptive Computation and Machine Learning series, The MIT Press, 2016), and Various Deep Learning Algorithms in Computational Intelligence (edited by Oscar Humberto Montiel Ross, Mdpi AG, 2023); herein incorporated by reference in their entireties.

[0145] Full-length body videos and / or hand videos and / or face videos are segmented into video segments for subsequent feature extraction and analysis. In certain embodiments, each video segment is 3 to 8 seconds in length, including any number of seconds within this range such as 3 seconds, 4 seconds, 5 seconds, 6 seconds, 7 seconds, or 8 seconds. In some embodiments, any video segment less than 3 seconds in length is discarded. In some embodiments, any video segment greater than 8 seconds in length is further segmented. Each video segment that is retained should show multiple strides during walking, multiple finger-tapping cycles during finger-tapping, or multiple abnormal or involuntary facial movements of the subject.

[0146] In certain embodiments, the computer implemented method further comprises performing sparsity-inducing feature selection on the plurality of video segments. As described above, the body landmarks may comprise: a nose, a right shoulder, a left shoulder, a right elbow, a left elbow, a right wrist, a left wrist, a right hip, a left hip, a right knee, a left knee, a right ankle, and a left ankle; and the hand landmarks comprise a wrist, a thumb, an index finger, a middle finger, a ring finger, a pinky finger, base of the index finger, and base of the pinky finger. The body movement features may comprise: neck angle; arm-body angle; the distance between the left wrist and the left shoulder and / or the distance between the right wrist and the right shoulder; the distance between the left ankle and the hip; the horizontal displacement between the ankles; the horizontal displacement between the knees; the horizontal displacement between the mid-shoulder and the mid-hip; the vertical displacement between the ankles; the vertical displacement between the shoulders; and the vertical displacement between the hips. In certain embodiments, the hand movement features comprise: the distance between a thumb and index finger of the active hand that is finger tapping; the distance between the middle finger and the wrist of the active hand that is finger tapping; the distance between the ring finger and the wrist of the active hand that is finger tapping; and the distance between theAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 pinky finger and the wrist of active hand that is finger tapping. In certain embodiments, the body movement features comprise relative power in 0.5 Hz to 1 Hz of ankle separation, standard deviation (STD) of ankle-hip distance, relative power in 2 Hz to 4 Hz of neck angle, relative power in 4 Hz to 6 Hz of arm-body angular velocity, and mean of neck angle. In certain embodiments, the hand movement features comprise half-power frequency in pinky finger movement speed, relative power in 2 Hz to 4 Hz of finger tapping speed, and median absolute deviation (MAD) of finger tapping amplitude. The face landmarks may comprise: a nose, a right eye, a left eye, a right eyelid, a left eyelid, a right eyebrow, a left eyebrow, eye corners, a mouth, an upper lip, a lower lip, a right cheek, a left cheek, a forehead, tongue, jaw, and a chin. In some embodiments, the face movement features are used to detect rapid eye blinking, face twitching, face tremor, jaw tremor, lip tremor, lip-smacking, tongue twisting, tongue curling, tongue thrusting, lip puckering, or grimacing.

[0147] Additional relevant clinic metrics may also be stored in the system, including, without limitation, the subject’s age, height, body-mass-index (BMI), gender and / or weight, or other metrics applicable to the present techniques. In some cases, such metrics may be obtained from the subject’s electronic medical records (EMR) or another applicable cloud-based storage technique, or, in other cases, may be measured and subsequently stored in the subject’s electronic medical records or another applicable cloud-based storage technique.

[0148] In some embodiments, the computer implemented method further comprises correlating the symptom severity of the subject determined based on classification of the set of body movement features and / or hand movement features with a Movement Disorder Society-Sponsored Revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS) or a Hoehn and Yahr (HnY) scale, a Parkinson’s Disease Composite Scale (PDCS), or a Schwab and England Activities of Daily Living (ADL) Scale.

[0149] In some embodiments, the computer implemented method further comprises comparing movement metrics for single joint or multi-joint movements of the subject to reference value ranges for a normal age-matched control subject (i.e, healthy subject, subject not having a movement disorder). In some embodiments, the movement metrics are for finger-tapping or gait speed.

[0150] The methods can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, a data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or any combination thereof.

[0151] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0152] In a further aspect, the system for performing the computer implemented method, as described, may include a processor, a storage component (i.e., memory), a display component, and other components typically present in general purpose computers. In some embodiments, the processor is provided by a computer or handheld device (e.g., a cell phone or tablet). The storage component stores information accessible by the processor, including instructions that may be executed by the processor and data that may be retrieved, manipulated or stored by the processor.

[0153] The storage component includes instructions. For example, the storage component includes instructions for determining symptom severity of a subject having a movement disorder and / or diagnosing a subject with a movement disorder according to the methods described herein. The computer processor is coupled to the storage component and configured to execute the instructions stored in the storage component in order to receive a video of the subject’s entire body while the subject is walking and / or a video of a hand of the subject while the subject is finger-tapping and analyze the video images according to one or more algorithms, as described herein.

[0154] The processor and / or memory may be operably connected to a display device, for example, via a wired, such as a Universal Serial Bus (USB) connection, or wireless connection, such as a Bluetooth connection. Any convenient display device, such as a liquid crystal display (LCD), light- emitting diode (LED) display, plasma (PDP) display, quantum dot (QLED) display or cathode ray tube display device may be used. The display component displays information regarding the symptom severity and / or diagnosis of the subject. In some embodiments, the display further displays the body and / or hand videos or video segments thereof. In some embodiments, the display further displays body movement features comprising relative power in 0.5 Hz to 1 Hz of ankle separation,Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 standard deviation (STD) of ankle-hip distance, relative power in 2 Hz to 4 Hz of neck angle, relative power in 4 Hz to 6 Hz of arm-body angular velocity, and mean of neck angle. In some embodiments, the display further displays hand movement features comprising half-power frequency in pinky finger movement speed, relative power in 2 Hz to 4 Hz of finger tapping speed, and median absolute deviation (MAD) of finger tapping amplitude. In some embodiments, the display further displays information comparing movement metrics for single joint or multi-joint movements of the subject to reference value ranges for a normal age-matched control subject (i.e, healthy subject, subject not having a movement disorder). In some embodiments, the movement metrics are for finger-tapping or gait speed.

[0155] The storage component may be of any type capable of storing information accessible by the processor, such as a hard-drive, memory card, ROM, RAM, DVD, CD-ROM, USB Flash drive, write- capable, and read-only memories. The processor may be a general purpose processor, a graphics processor unit, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor, but in the alternative, the processor can be a controller, microcontroller, or state machine, combinations of the same, or the like. A processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Although described herein primarily with respect to digital technology, a processor can also include primarily analog components. A computing environment can include any type of computer system, including, but not limited to, a computer system based on a microprocessor, a graphics processor unit, a mainframe computer, a digital signal processor, a portable computing device, a personal organizer, a device controller, and a computational engine within an appliance, to name a few.

[0156] The steps of a method, process, or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module, engine, and associated databases can reside in memory resources such as in RAM memory, FRAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of non-transitory computer-readable storage medium, media, or physical computer storage known in the art. An exemplary storage medium can be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In theAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium can reside as discrete components in a user terminal.

[0157] The instructions may be any set of instructions to be executed directly (such as machine code) or indirectly (such as scripts) by the processor. In that regard, the terms "instructions," "steps" and "programs" may be used interchangeably herein. The instructions may be stored in object code form for direct processing by the processor, or in any other computer language including scripts or collections of independent source code modules that are interpreted on demand or compiled in advance.

[0158] Data may be retrieved, stored or modified by the processor in accordance with the instructions. For instance, although the system is not limited by any particular data structure, the data may be stored in computer registers, in a relational database as a table having a plurality of different fields and records, XML documents, or flat files. The data may also be formatted in any computer-readable format such as, but not limited to, binary values, ASCII or Unicode. Moreover, the data may comprise any information sufficient to identify the relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories (including other network locations) or information which is used by a function to calculate the relevant data.

[0159] In certain embodiments, the processor and storage component may comprise multiple processors and storage components that may or may not be stored within the same physical housing. For example, some of the instructions and data may be stored on removable CD-ROM and others within a read-only computer chip. Some or all of the instructions and data may be stored in a location physically remote from, yet still accessible by, the processor. Similarly, the processor may comprise a collection of processors which may or may not operate in parallel.

[0160] In some embodiments, the method can be performed using a cloud computing system. In these embodiments, videos of the subject’s body while the subject is walking and / or videos of a hand of the subject while the subject is finger-tapping, and the programming can be exported to a cloud computer, which runs the program, and returns an output to the user.

[0161] Components of systems for carrying out the presently disclosed methods are further described in the examples below.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 Kits

[0162] Also provided are kits comprising software for carrying out the computer implemented methods, described herein, for determining symptom severity of a subject having a movement disorder or diagnosing a subject with a movement disorder. In some embodiments, the kit comprises a non-transitory computer-readable medium and instructions for determining symptom severity of a subject having a movement disorder and / or diagnosis of a subject, as described herein. In some embodiments, the kit comprises a system comprising a processor programmed to determine symptom severity of a subject having a movement disorder and / or diagnosis of a subject according to a computer implemented method described herein; and a display component for displaying information regarding the symptom severity and / or diagnosis of the subject.

[0163] In addition to the above components, the subject kits may further include (in certain embodiments) instructions for practicing the subject methods. These instructions may be present in the subject kits in a variety of forms, one or more of which may be present in the kit. One form in which these instructions may be present is as printed information on a suitable medium or substrate, e.g., a piece or pieces of paper on which the information is printed, in the packaging of the kit, in a package insert, and the like. Yet another form of these instructions is a computer readable medium, e.g., diskette, compact disk (CD), flash drive, and the like, on which the information has been recorded. Yet another form of these instructions that may be present is a website address which may be used via the internet to access the information at a removed site. Utility

[0164] Assessment of symptom severity in patients having movement disorders has typically relied on the subjective recall of the patients themselves, which is limited by high assessment variance, day-to-day fluctuation of symptoms, imperfect recall, and recency bias. The subject methods improve quantification of motor symptom progression in patients with movement disorders, which is useful for assessing and optimizing therapeutic interventions, such as dopaminergic medications and deep brain stimulation (DBS). In some embodiments, the methods and systems described herein find use in clinical settings such as any clinical setting where routine clinical assessment of a movement disorder is applied. In other embodiments, the methods and systems described herein find use in remote medicine settings such as in telemedicine contexts, where monitoring symptom severity may be enabled and made more convenient by application of the present methods and systems. In some cases, the subject methods and systems find use in improving diagnosis of a movement disorder and providing convenient regular monitoring of symptoms.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 Examples of Non-Limiting Aspects of the Disclosure

[0165] Aspects, including embodiments, of the present subject matter described above may be beneficial alone or in combination, with one or more other aspects or embodiments. Without limiting the foregoing description, certain non-limiting aspects of the disclosure numbered 1-144 are provided below. As will be apparent to those of skill in the art upon reading this disclosure, each of the individually numbered aspects may be used or combined with any of the preceding or following individually numbered aspects. This is intended to provide support for all such combinations of aspects and is not limited to combinations of aspects explicitly provided below. 1. A computer implemented method for determining symptom severity of a subject having a movement disorder, the computer performing steps comprising: receiving a first video of the subject’s entire body while the subject is walking and / or receiving a second video of a hand of the subject while the subject is finger-tapping; segmenting the first video and / or the second video to generate a plurality of video segments; identifying body landmarks and / or hand landmarks in the plurality of video segments; using the body landmarks and / or hand landmarks to extract a set of body movement features and / or hand movement features from the plurality of video segments based on relative movement between the landmarks with time; using a machine learning algorithm to determine symptom severity of the subject based on classification of the set of body movement features and / or hand movement features; and displaying information regarding the symptom severity of the subject. 2. The computer implemented method of aspect 1, wherein the body landmarks comprise a nose, a right shoulder, a left shoulder, a right elbow, a left elbow, a right wrist, a left wrist, a right hip, a left hip, a right knee, a left knee, a right ankle, and a left ankle. 3. The computer implemented method of aspect 1 or 2, wherein the hand landmarks comprise a wrist, a thumb, an index finger, a middle finger, a ring finger, a pinky finger, base of the index finger, and base of the pinky finger. 4. The computer implemented method of any one of aspects 1-3, wherein the body movement features comprise:Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 neck angle; arm-body angle; distance between left wrist and left shoulder and / or distance between right wrist and right shoulder; distance between left ankle and hip; horizontal displacement between ankles; horizontal displacement between knees; horizontal displacement between mid-shoulder and mid-hip; vertical displacement between ankles; vertical displacement between shoulders; and vertical displacement between hips. 5. The computer implemented method any one of aspects 1-4, wherein the hand movement features comprise: distance between thumb and index finger of active hand that is finger tapping; distance between middle finger and wrist of active hand that is finger tapping; distance between ring finger and wrist of active hand that is finger tapping; and distance between pinky finger and wrist of active hand that is finger tapping. 6. The computer implemented method of any one of aspects 1-5, wherein the body movement features further comprise relative power in 0.5 Hz to 1 Hz of ankle separation, standard deviation (STD) of ankle-hip distance, relative power in 2 Hz to 4 Hz of neck angle, relative power in 4 Hz to 6 Hz of arm-body angular velocity, and mean of neck angle. 7. The computer implemented method of any one of aspects 1-6, wherein the hand movement features further comprise half-power frequency in pinky finger movement speed, relative power in 2 Hz to 4 Hz of finger tapping speed, and median absolute deviation (MAD) of finger tapping amplitude. 8. The computer implemented method of any one of aspects 1-7, wherein the set of body movement features and / or hand movement features are identified by performing sparsity- inducing feature selection on the plurality of video segments.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 9. The computer implemented method of any one of aspects 1-8, wherein the movement disorder is Parkinson’s disease, progressive supranuclear palsy, or multiple systems atrophy. 10. The computer implemented method of any one of aspects 1-9, wherein the machine learning algorithm uses a logistic regression (LR) classifier, a support vector machine (SVM) classifier, or a linear discriminant analysis (LDA) classifier. 11. The computer implemented method of any one of aspects 1-10, wherein the machine learning algorithm uses an artificial neural network. 12. The computer implemented method of any one of aspects 1-11, wherein the machine learning algorithm uses a deep learning algorithm. 13. The computer implemented method of aspect 12, wherein the deep learning algorithm uses a convolutional neural network, a deep neural network, a recurrent neural network, a deep residual neural network, a long short-term memory network, a deep belief network, a multilayer perceptron, or deep reinforcement learning. 14. The computer implemented method of any one of aspects 1-13, wherein the machine learning algorithm is supervised, semi-supervised, or unsupervised. 15. The computer implemented method of any one of aspects 1-14, wherein each video segment shows multiple strides during the walking or multiple finger-tapping cycles during the finger- tapping. 16. The computer implemented method of any one of aspects 1-15, wherein each video segment is 3 to 8 seconds in length. 17. The computer implemented method of any one of aspects 1-16, wherein any video segment less than 3 seconds in length is discarded. 18. The computer implemented method of any one of aspects 1-17, wherein any video segment greater than 8 seconds in length is further segmented.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 19. The computer implemented method of any one of aspects 1-18, further comprising correlating the symptom severity of the subject based on classification of the set of body movement features and / or hand movement features with a Movement Disorder Society-Sponsored Revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS) or a Hoehn and Yahr (HnY) scale, a Parkinson’s Disease Composite Scale (PDCS), or a Schwab and England Activities of Daily Living (ADL) Scale. 20. The computer implemented method of any one of aspects 1-19, further comprising comparing movement metrics for single joint or multi-joint movements of the subject to reference value ranges for a normal age-matched control subject. 21. The computer implemented method of aspect 20, wherein the movement metrics are for finger-tapping or gait speed. 22. The computer implemented method of any one of aspects 1-21, further comprising: receiving a third video of the subject’s face; segmenting the third video to generate a plurality of video segments; identifying face landmarks in the plurality of video segments; using the face landmarks to extract a set of face movement features from the plurality of video segments based on relative movement between the face landmarks with time; using a machine learning algorithm to determine symptom severity of the subject based on classification of the set of face movement features in combination with the set of body movement features and / or hand movement features; and displaying information regarding the symptom severity of the subject. 23. The computer implemented method of aspect 22, wherein the face landmarks comprise a nose, a right eye, a left eye, a right eyelid, a left eyelid, a right eyebrow, a left eyebrow, eye corners, a mouth, an upper lip, a lower lip, a right cheek, a left cheek, a forehead, tongue, jaw, and a chin.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 24. The computer implemented method of aspect 22 or 23, wherein the set of face movement features is used to detect an abnormal facial movement or an involuntary facial movement associated with a movement disorder. 25. The computer implemented method of aspect 24, wherein the abnormal facial movement or the involuntary facial movement is rapid eye blinking, face twitching, face tremor, jaw tremor, lip tremor, lip-smacking, tongue twisting, tongue curling, tongue thrusting, lip puckering, facial drooping, or grimacing. 26. The computer implemented method of any one of aspects 22-25, wherein the movement disorder is Tardive dyskinesia, facial dystonia, hemifacial spasm, peripheral facial palsy, facial myokymia, facial bradykinesia, blepharospasm, Tourette syndrome, orofacial dyskinesia, brainstem tumor, peripheral neuropathy, chorea, multiple sclerosis, Guillain-Barré syndrome, Parkinson’s disease, congenital facial nerve palsy, acquired facial nerve palsy, Moebius syndrome, Poland syndrome, CHARGE syndrome, Hemifacial microsomia, 22q11.2 deletion syndrome, facial paralysis or paresis, or a psychogenic movement disorder. 27. The computer implemented method of any one of aspects 22-26, further comprising using the face landmarks for facial recognition, head pose estimation, facial expression detection, or facial emotion recognition. 28. The computer implemented method of aspect 27, wherein the facial emotion recognition is used to detect depression, psychosis, fatigue, fear, anger, sadness, or joy. 29. A computer implemented method for determining symptom severity of a subject having a facial movement disorder, the computer performing steps comprising: receiving a video of the subject’s face; segmenting the video to generate a plurality of video segments; identifying face landmarks in the plurality of video segments; using the face landmarks to extract a set of face movement features from the plurality of video segments based on relative movement between the face landmarks with time; using a machine learning algorithm to determine symptom severity of the subject based on classification of the set of face movement features; andAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 displaying information regarding the symptom severity of the subject. 30. The computer implemented method of aspect 29, wherein the face landmarks comprise a nose, a right eye, a left eye, a right eyelid, a left eyelid, a right eyebrow, a left eyebrow, eye corners, a mouth, an upper lip, a lower lip, a right cheek, a left cheek, a forehead, tongue, jaw, and a chin. 31. The computer implemented method of aspect 29 or 30, wherein the set of face movement features is used to detect an abnormal facial movement or an involuntary facial movement associated with the facial movement disorder. 32. The computer implemented method of aspect 31, wherein the abnormal facial movement or the involuntary facial movement is rapid eye blinking, face twitching, face tremor, jaw tremor, lip tremor, lip-smacking, tongue twisting, tongue curling, tongue thrusting, lip puckering, or grimacing. 33. The computer implemented method of any one of aspects 29-32, wherein the facial movement disorder is Tardive dyskinesia, facial dystonia, hemifacial spasm, peripheral facial palsy, facial myokymia, facial bradykinesia, blepharospasm, Tourette syndrome, orofacial dyskinesia, brainstem tumor, peripheral neuropathy, chorea, multiple sclerosis, Guillain-Barré syndrome, Parkinson’s disease, congenital facial nerve palsy, acquired facial nerve palsy, Moebius syndrome, Poland syndrome, CHARGE syndrome, Hemifacial microsomia, 22q11.2 deletion syndrome, facial paralysis or paresis, or a psychogenic movement disorder. 34. The computer implemented method of any one of aspects 29-33, further comprising using the face landmarks for facial recognition, head pose estimation, facial expression detection, or facial emotion recognition. 35. The computer implemented method of aspect 34, wherein the facial emotion recognition is used to detect depression, psychosis, fatigue, fear, anger, sadness, or joy.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 36. A non-transitory computer-readable medium comprising program instructions that, when executed by a processor in a computer, causes the processor to perform the method of any one of aspects 1-35. 37. A kit comprising the non-transitory computer-readable medium of aspect 36 and instructions for determining symptom severity of a subject having a movement disorder. 38. A system comprising: a video recording device; a processor, wherein the processor is programmed to determine symptom severity of a subject having a movement disorder according to the computer implemented method of any one of aspects 1-35; and a display component that displays information regarding the symptom severity of the subject. 39. The system of aspect 38, wherein the video recording device is provided by a digital camera, a smartphone, a tablet, a laptop, or a camcorder. 40. The system of aspects 38 or 39, further comprising a tripod, wherein the video recording device is mounted on the tripod. 41. The system of any one of aspects 38-40, further comprising one or more additional recording devices, wherein each additional video recording device records videos of the subject from a different angle. 42. The system of any one of aspects 38-41, wherein the video recording device comprises a laser imaging, detection, and ranging (LIDAR) sensor. 43. The system of any one of aspects 38-42, wherein the display further displays information comparing movement metrics for single joint or multi-joint movements of the subject to reference value ranges for a normal age-matched control subject. 44. The system of aspect 43, wherein the movement metrics are for finger-tapping or gait speed.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 45. A method of using the system of any one of aspects 38-44 for determining symptom severity of a subject having a movement disorder, the method comprising: recording a first video of the subject’s entire body while the subject is walking and / or a second video of a hand of the subject while the subject is finger-tapping using the video recording device; and determining symptom severity of the subject having the movement disorder according to the computer implemented method. 46. The method of aspect 45, wherein only one hand of the subject is actively performing finger-tapping and the subject’s other hand is resting. 47. The method of aspect 45 or 46, wherein the body landmarks comprise a nose, a right shoulder, a left shoulder, a right elbow, a left elbow, a right wrist, a left wrist, a right hip, a left hip, a right knee, a left knee, a right ankle, and a left ankle. 48. The method of any one of aspects 45-47, wherein the hand landmarks comprise a wrist, a thumb, an index finger, a middle finger, a ring finger, a pinky finger, base of the index finger, and base of the pinky finger. 49. The method of any one of aspects 45-48, wherein the body movement features comprise: neck angle; arm-body angle; distance between left wrist and left shoulder and / or distance between right wrist and right shoulder; distance between left ankle and hip; horizontal displacement between ankles; horizontal displacement between knees; horizontal displacement between mid-shoulder and mid-hip; vertical displacement between ankles; vertical displacement between shoulders; and vertical displacement between hips.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 50. The method of any one of aspects 45-49, wherein the hand movement features comprise: distance between thumb and index finger of active hand that is finger tapping; distance between middle finger and wrist of active hand that is finger tapping; distance between ring finger and wrist of active hand that is finger tapping; and distance between pinky finger and wrist of active hand that is finger tapping. 51. The method of any one of aspects 45-50, wherein the body movement features comprise relative power in 0.5 Hz to 1 Hz of ankle separation, standard deviation (STD) of ankle-hip distance, relative power in 2 Hz to 4 Hz of neck angle, relative power in 4 Hz to 6 Hz of arm-body angular velocity, and mean of neck angle. 52. The method of any one of aspects 45-51, wherein the hand movement features comprise half-power frequency in pinky finger movement speed, relative power in 2 Hz to 4 Hz of finger tapping speed, and median absolute deviation (MAD) of finger tapping amplitude. 53. The method of any one of aspects 45-52, wherein the set of body movement features and / or hand movement features are identified by performing sparsity-inducing feature selection on the plurality of video segments. 54. The method of any one of aspects 45-53, wherein the movement disorder is Parkinson’s disease, progressive supranuclear palsy, or multiple systems atrophy. 55. The method of any one of aspects 45-54, wherein the machine learning algorithm uses a logistic regression (LR) classifier, a support vector machine (SVM) classifier, or a linear discriminant analysis (LDA) classifier. 56. The method of any one of aspects 45-55, wherein the machine learning algorithm uses an artificial neural network. 57. The method of any one of aspects 45-56, wherein the machine learning algorithm uses a deep learning algorithm.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 58. The method of aspect 57, wherein the deep learning algorithm uses a convolutional neural network, a deep neural network, a recurrent neural network, a deep residual neural network, a long short-term memory network, a deep belief network, a multilayer perceptron, or deep reinforcement learning. 59. The method of any one of aspects 45-58, wherein the machine learning algorithm is supervised, semi-supervised, or unsupervised. 60. The method of any one of aspects 45-59, wherein each video segment shows multiple strides during the walking or multiple finger-tapping cycles during the finger-tapping. 61. The method of any one of aspects 45-60, wherein each video segment is 3 to 8 seconds in length. 62. The method of any one of aspects 45-61, wherein any video segment less than 3 seconds in length is discarded. 63. The method of any one of aspects 45-62, wherein any video segment greater than 8 seconds in length is further segmented. 64. The method of any one of aspects 45-63, further comprising correlating the symptom severity of the subject based on classification of the set of body movement features and / or hand movement features with a Movement Disorder Society-Sponsored Revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS) or a Hoehn and Yahr (HnY) scale, a Parkinson’s Disease Composite Scale (PDCS), or a Schwab and England Activities of Daily Living (ADL) Scale. 65. The method of any one of aspects 45-64, further comprising assessing one or more motor and non-motor symptoms of the subject using a Movement Disorder Society-Sponsored Revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS), a Hoehn and Yahr (HnY) scale, or a Montreal Cognitive Assessment (MoCA) scale.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 66. The method of any one of aspects 45-65, further comprising comparing movement metrics for single joint or multi-joint movements of the subject to reference value ranges for a normal age-matched control subject. 67. The method of aspect 66, wherein the movement metrics are for finger-tapping or gait speed. 68. The method of any one of aspects 45-67, further comprising correlating the symptom severity of the subject based on classification of the set of body movement features and / or hand movement features with a Progressive Supranuclear Palsy Clinical Deficits Scale (PSP-CDS), a Psychogenic Movement Disorders (PMD) scale, a Unified Huntington's Disease Rating Scale-Total Motor Score (UHDRS-TMS). 69. The method of any one of aspects 45-68, wherein each image in the first video or the second video has at least 1920 pixels horizontally and at least 1080 pixels vertically. 70. The method of any one of aspects 45-69, wherein the video recording device shoots at least 30 frames per second. 71. The method of any one of aspects 45-70, wherein the subject walks toward the video recording device and away from the video recording device for a minimum distance of 10 meters in each direction during said recording of the first video. 72. The method of any one of aspects 45-71, wherein the first video and the second video are single-view videos shot with a single video recording device. 73. The method of any one of aspects 45-72, wherein the first video and the second video are multi-view videos shot with at least two video recording devices. 74. The method of any one of aspects 45-73, wherein the method is performed before and after administering a treatment to the subject for the movement disorder to evaluate efficacy of the treatment.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 75. The method of any one of aspects 45-74, further comprising altering the treatment of the subject for the movement disorder if the subject shows increased symptom severity based on said classification of the set of body movement features and / or hand movement features. 76. The method of any one of aspects 45-75, wherein the method is performed when the subject is at home or when the subject is at a hospital. 77. The method of any one of aspects 45-76, further comprising placing fiducial markers on the subject for said identifying body landmarks and / or hand landmarks. 78. The method of any one of aspects 45-77, further comprising: recording a third video of the subject’s face; segmenting the third video to generate a plurality of video segments; identifying face landmarks in the plurality of video segments; using the face landmarks to extract a set of face movement features from the plurality of video segments based on relative movement between the face landmarks with time; and using a machine learning algorithm to determine symptom severity of the subject based on classification of the set of face movement features in combination with the set of body movement features and / or hand movement features. 79. The method of aspect 78, wherein the face landmarks comprise a nose, a right eye, a left eye, a right eyelid, a left eyelid, a right eyebrow, a left eyebrow, eye corners, a mouth, an upper lip, a lower lip, a right cheek, a left cheek, a forehead, tongue, jaw, and a chin. 80. The method of aspect 78 or 79, wherein the set of face movement features is used to detect an abnormal facial movement or an involuntary facial movement associated with a movement disorder. 81. The method of aspect 80, wherein the abnormal facial movement or the involuntary facial movement is rapid eye blinking, face twitching, face tremor, jaw tremor, lip tremor, lip- smacking, tongue twisting, tongue curling, tongue thrusting, lip puckering, facial drooping, or grimacing.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 82. The method of any one of aspects 78-81, wherein the movement disorder is Tardive dyskinesia, facial dystonia, hemifacial spasm, peripheral facial palsy, facial myokymia, facial bradykinesia, blepharospasm, Tourette syndrome, orofacial dyskinesia, brainstem tumor, peripheral neuropathy, chorea, multiple sclerosis, Guillain-Barré syndrome, Parkinson’s disease, congenital facial nerve palsy, acquired facial nerve palsy, Moebius syndrome, Poland syndrome, CHARGE syndrome, Hemifacial microsomia, 22q11.2 deletion syndrome, facial paralysis or paresis, or a psychogenic movement disorder. 83. The method of any one of aspects 78-82, further comprising using the face landmarks for facial recognition, head pose estimation, facial expression detection, or facial emotion recognition. 84. The method of aspect 83, wherein facial emotion recognition is used to detect depression, psychosis, fatigue, fear, anger, sadness, or joy. 85. The method of any one of aspects 45-84, wherein said recording the first video, the second video, or the third video is performed by the subject, a caregiver, or a physician. 86. The method of any one of aspects 45-85, wherein the video recording device is a digital camera, a smartphone, a tablet, a laptop, or a camcorder. 87. The method of aspect 86, wherein said recording the first video, the second video, or the third video is performed by the smartphone automatically whenever the subject uses the smartphone. 88. A computer implemented method for diagnosing a movement disorder in a subject, the computer performing steps comprising: receiving a first video of the subject’s entire body while the subject is walking and / or receiving a second video of a hand of the subject while the subject is finger-tapping; segmenting the first video and / or the second video to generate a plurality of video segments; identifying body landmarks and / or hand landmarks in the plurality of video segments;Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 using the body landmarks and / or hand landmarks to extract a set of body movement features and / or hand movement features from the plurality of video segments based on relative movement between the landmarks with time; using a machine learning algorithm to detect a symptom of the movement disorder, if present, based on classification of the set of body movement features and / or hand movement features; and displaying information regarding the diagnosis of the subject based on whether the symptom of the movement disorder is detected. 89. The computer implemented method of aspect 88, wherein the body landmarks comprise a nose, a right shoulder, a left shoulder, a right elbow, a left elbow, a right wrist, a left wrist, a right hip, a left hip, a right knee, a left knee, a right ankle, and a left ankle. 90. The computer implemented method of aspect 88 or 89, wherein the hand landmarks comprise a wrist, a thumb, an index finger, a middle finger, a ring finger, a pinky finger, base of the index finger, and base of the pinky finger. 91. The computer implemented method of any one of aspects 88-90, wherein the body movement features comprise: neck angle; arm-body angle; distance between left wrist and left shoulder and / or distance between right wrist and right shoulder; distance between left ankle and hip; horizontal displacement between ankles; horizontal displacement between knees; horizontal displacement between mid-shoulder and mid-hip; vertical displacement between ankles; vertical displacement between shoulders; and vertical displacement between hips. 92. The computer implemented method any one of aspects 88-91, wherein the hand movement features comprise: distance between thumb and index finger of active hand that is finger tapping;Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 distance between middle finger and wrist of active hand that is finger tapping; distance between ring finger and wrist of active hand that is finger tapping; and distance between pinky finger and wrist of active hand that is finger tapping. 93. The computer implemented method of any one of aspects 88-92, wherein the display further displays body movement features comprising relative power in 0.5 Hz to 1 Hz of ankle separation, standard deviation (STD) of ankle-hip distance, relative power in 2 Hz to 4 Hz of neck angle, relative power in 4 Hz to 6 Hz of arm-body angular velocity, and mean of neck angle. 94. The computer implemented method of any one of aspects 88-93, wherein the display further displays hand movement features comprising half-power frequency in pinky finger movement speed, relative power in 2 Hz to 4 Hz of finger tapping speed, and median absolute deviation (MAD) of finger tapping amplitude. 95. The computer implemented method of any one of aspects 88-94, wherein the set of body movement features and / or hand movement features are identified by performing sparsity- inducing feature selection on the plurality of video segments. 96. The computer implemented method of any one of aspects 88-95, wherein the movement disorder is Parkinson’s disease, progressive supranuclear palsy, or multiple systems atrophy. 97. The computer implemented method of any one of aspects 88-96, wherein the machine learning algorithm uses a logistic regression (LR) classifier, a support vector machine (SVM) classifier, or a linear discriminant analysis (LDA) classifier. 98. The computer implemented method of any one of aspects 88-97, wherein the machine learning algorithm uses an artificial neural network. 99. The computer implemented method of any one of aspects 88-98, wherein the machine learning algorithm uses a deep learning algorithm.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 100. The computer implemented method of aspect 99, wherein the deep learning algorithm uses a convolutional neural network, a deep neural network, a recurrent neural network, a deep residual neural network, a long short-term memory network, a deep belief network, a multilayer perceptron, or deep reinforcement learning. 101. The computer implemented method of any one of aspects 88-100, wherein the machine learning algorithm is supervised, semi-supervised, or unsupervised. 102. The computer implemented method of any one of aspects 88-101, wherein each video segment shows multiple strides during the walking or multiple finger-tapping cycles during the finger- tapping. 103. The computer implemented method of any one of aspects 88-102, wherein each video segment is 3 to 8 seconds in length. 104. The computer implemented method of any one of aspects 88-103, wherein any video segment less than 3 seconds in length is discarded. 105. The computer implemented method of any one of aspects 88-104, wherein any video segment greater than 8 seconds in length is further segmented. 106. A non-transitory computer-readable medium comprising program instructions that, when executed by a processor in a computer, causes the processor to perform the method of any one of aspects 88-105. 107. A kit comprising the non-transitory computer-readable medium of aspect 106 and instructions for diagnosing a movement disorder. 108. A system comprising: a video recording device; a processor, wherein the processor is programmed to determine symptom severity of a subject having a movement disorder according to the computer implemented method of any one of aspects 88-105; andAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 a display component that displays information regarding the diagnosis of the subject. 109. The system of aspect 108, wherein the video recording device is provided by a digital camera, a smartphone, a tablet, a laptop, or a camcorder. 110. The system of aspect 108 or 109, further comprising a tripod, wherein the video recording device is mounted on the tripod. 111. The system of any one of aspects 108-110, further comprising one or more additional recording devices, wherein each additional video recording device records videos of the subject from a different angle. 112. The system of any one of aspects 108-111, wherein the video recording device comprises a laser imaging, detection, and ranging (LIDAR) sensor. 113. The system of any one of aspects 108-112, wherein the display further displays information comparing movement metrics for single joint or multi-joint movements of the subject to reference value ranges for a normal age-matched control subject. 114. The system of aspect 113, wherein the movement metrics are for finger-tapping or gait speed. 115. A method of using the system of any one of aspects 108-114 for diagnosing a movement disorder in a subject, the method comprising: recording a first video of the subject’s entire body while the subject is walking and / or a second video of a hand of the subject while the subject is finger-tapping using the video recording device; and detecting whether the subject has a symptom of the movement disorder according to the computer implemented method; and diagnosing the subject with the movement disorder if the symptom of the movement disorder is detected.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 116. The method of aspect 115, further comprising treating the subject for the movement disorder if the subject has a positive diagnosis for the movement disorder based on detection of the symptom. 117. The method of aspect 115 or 116, wherein only one hand of the subject is actively performing finger-tapping and the subject’s other hand is resting. 118. The method of any one of aspects 115-117, wherein the body landmarks comprise a nose, a right shoulder, a left shoulder, a right elbow, a left elbow, a right wrist, a left wrist, a right hip, a left hip, a right knee, a left knee, a right ankle, and a left ankle. 119. The method of any one of aspects 115-118, wherein the hand landmarks comprise a wrist, a thumb, an index finger, a middle finger, a ring finger, a pinky finger, base of the index finger, and base of the pinky finger. 120. The method of any one of aspects 115-119, wherein the body movement features comprise: neck angle; arm-body angle; distance between left wrist and left shoulder and / or distance between right wrist and right shoulder; distance between left ankle and hip; horizontal displacement between ankles; horizontal displacement between knees; horizontal displacement between mid-shoulder and mid-hip; vertical displacement between ankles; vertical displacement between shoulders; and vertical displacement between hips. 121. The method of any one of aspects 115-120, wherein the hand movement features comprise: distance between thumb and index finger of active hand that is finger tapping; distance between middle finger and wrist of active hand that is finger tapping;Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 distance between ring finger and wrist of active hand that is finger tapping; and distance between pinky finger and wrist of active hand that is finger tapping. 122. The method of any one of aspects 115-121, wherein the body movement features comprise relative power in 0.5 Hz to 1 Hz of ankle separation, standard deviation (STD) of ankle-hip distance, relative power in 2 Hz to 4 Hz of neck angle, relative power in 4 Hz to 6 Hz of arm-body angular velocity, and mean of neck angle. 123. The method of any one of aspects 115-122, wherein the hand movement features comprise half-power frequency in pinky finger movement speed, relative power in 2 Hz to 4 Hz of finger tapping speed, and median absolute deviation (MAD) of finger tapping amplitude. 124. The method of any one of aspects 115-123, wherein the set of body movement features and / or hand movement features are identified by performing sparsity-inducing feature selection on the plurality of video segments. 125. The method of any one of aspects 115-124, wherein the movement disorder is Parkinson’s disease, progressive supranuclear palsy, or multiple systems atrophy. 126. The method of any one of aspects 115-125, wherein the machine learning algorithm uses a logistic regression (LR) classifier, a support vector machine (SVM) classifier, or a linear discriminant analysis (LDA) classifier. 127. The method of any one of aspects 115-126, wherein the machine learning algorithm uses an artificial neural network. 128. The method of any one of aspects 115-127, wherein the machine learning algorithm uses a deep learning algorithm. 129. The method of aspect 128, wherein the deep learning algorithm uses a convolutional neural network, a deep neural network, a recurrent neural network, a deep residual neural network, a long short-term memory network, a deep belief network, a multilayer perceptron, or deep reinforcement learning.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 130. The method of any one of aspects 115-129, wherein the machine learning algorithm is supervised, semi-supervised, or unsupervised. 131. The method of any one of aspects 115-130, wherein each video segment shows multiple strides during the walking or multiple finger-tapping cycles during the finger-tapping. 132. The method of any one of aspects 115-131, wherein each video segment is 3 to 8 seconds in length. 133. The method of any one of aspects 115-132, wherein any video segment less than 3 seconds in length is discarded. 134. The method of any one of aspects 115-133, wherein any video segment greater than 8 seconds in length is further segmented. 135. The method of any one of aspects 115-134, further comprising correlating the symptom severity of the subject based on classification of the set of body movement features and / or hand movement features with a Movement Disorder Society-Sponsored Revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS) or a Hoehn and Yahr (HnY) scale, a Parkinson’s Disease Composite Scale (PDCS), or a Schwab and England Activities of Daily Living (ADL) Scale. 136. The method of any one of aspects 115-135, further comprising assessing one or more motor and non-motor symptoms of the subject using a Movement Disorder Society-Sponsored Revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS), a Hoehn and Yahr (HnY) scale, or a Montreal Cognitive Assessment (MoCA) scale. 137. The method of any one of aspects 115-136, further comprising correlating the symptom severity of the subject based on classification of the set of body movement features and / or hand movement features with a Progressive Supranuclear Palsy Clinical Deficits Scale (PSP-CDS), a Psychogenic Movement Disorders (PMD) scale, a Unified Huntington's Disease Rating Scale-Total Motor Score (UHDRS-TMS)Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 138. The method of any one of aspects 115-137, wherein each image in the first video or the second video has at least 1920 pixels horizontally and at least 1080 pixels vertically. 139. The method of any one of aspects 115-138, wherein the video recording device shoots at least 30 frames per second. 140. The method of any one of aspects 115-139, wherein the subject walks toward the video recording device and away from the video recording device for a minimum distance of 10 meters in each direction during said recording of the first video. 141. The method of any one of aspects 115-140, wherein the first video and the second video are single-view videos shot with a single video recording device. 142. The method of any one of aspects 115-141, wherein the first video and the second video are multi-view videos shot with at least two video recording devices. 143. The method of any one of aspects 115-142, wherein the method is performed when the subject is at home or when the subject is at a hospital. 144. The method of any one of aspects 115-143, further comprising placing fiducial markers on the subject for said identifying body landmarks and / or hand landmarks.

[0166] It will be apparent to one of ordinary skill in the art that various changes and modifications can be made without departing from the spirit or scope of the invention. EXPERIMENTAL

[0167] The following examples are put forth so as to provide those of ordinary skill in the art with a complete disclosure and description of how to make and use the present invention, and are not intended to limit the scope of what the inventors regard as their invention nor are they intended to represent that the experiments below are all or the only experiments performed. Efforts have been made to ensure accuracy with respect to numbers used (e.g. amounts, temperature, etc.) but some experimental errors and deviations should be accounted for. Unless indicated otherwise, parts areAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 parts by weight, molecular weight is weight average molecular weight, temperature is in degrees Centigrade, and pressure is at or near atmospheric.

[0168] All publications and patent applications cited in this specification are herein incorporated by reference as if each individual publication or patent application were specifically and individually indicated to be incorporated by reference.

[0169] The present invention has been described in terms of particular embodiments found or proposed by the present inventor to comprise preferred modes for the practice of the invention. It will be appreciated by those of skill in the art that, in light of the present disclosure, numerous modifications and changes can be made in the particular embodiments exemplified without departing from the intended scope of the invention. For example, due to codon redundancy, changes can be made in the underlying DNA sequence without affecting the protein sequence. Moreover, due to biological functional equivalency considerations, changes can be made in protein structure without affecting the biological action in kind or amount. All such modifications are intended to be included within the scope of the appended claims. Example 1 Interpretable Video-Based Tracking and Quantification of Parkinsonism Clinical Motor States Introduction

[0170] Parkinson’s disease (PD) is a common neurodegenerative disorder characterized by progressive motor symptoms (e.g., bradykinesia, rest tremor, rigidity, postural instability) that can be disabling and significantly impair quality of life1-3. The ability to quantify motor symptom progression in PD patients is crucial for assessing and optimizing therapeutic interventions, such as dopaminergic medications and deep brain stimulation (DBS)4. Such quantification requires accurate and continual monitoring of motor symptom severity and fluctuations. Currently this objective is only partially satisfied by the status quo strategy of intermittent motor assessments assessed at one time point by a single clinician using the Movement Disorder Society-sponsored revision of the Unified Parkinson’s Disease Rating Scale (MDS-UPDRS)5. Clinical signs such as finger tapping slowness and decrement (bradykinesia) within the MDS-UPDRS have been discovered by clinical heuristics and codified by expert consensus, precluding well-validated, objective, data-driven and quantifiable assessment of patients. Day- to-day fluctuation of symptoms relies on subjective recall from patients often captured by motor diaries6. These approaches are limited by high assessment variance,Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 imperfect recall, and recency bias7. To overcome these limitations, the field needs more reliable and objective tracking of PD clinical states.

[0171] Technology-based objective symptom quantification, such as those supported by wearable tracking devices and their ability to record kinematic (movement) data, introduces a potential solution8-10. Unfortunately, due to device expenses and technical limitations, embodiments of these quantification systems have yet to be regularly adopted into clinical practice for PD assessment11-13. More recently, advancements in machine learning (ML) and computer vision have yielded accessible solutions for extracting kinematic information at key anatomical positions from video data without the need for physical marker systems14-17. These computer vision solutions have the potential to address the shortcomings of existing methods, significantly enhance diagnostic accuracy, and open new avenues for optimizing personalized medical therapy in PD18-22. However, to date, many video-based diagnostic tools have implementation challenges of expensive and inaccessible technology, often requiring multi-camera setups, pristine video collection protocols, or additional sensors that are infeasible for conventional use. Additionally, typical “black-box” ML implementations are not tailored to be clinically interpretable, either due to complex and unintuitive algorithms or a lack of analysis on feature stability and optimality. Therefore, they are generally ineffective in generating novel clinical insights and are challenging to integrate into current clinical care or critically, to develop clinical oversight for. Finally, these tools often focus on prolonged videos from a formal clinical examination or features from a single motor modality, increasing the burden of video acquisition and missing the opportunity to integrate different domains (e.g., body posture, hand movement, facial expression), which has the potential to significantly increase the accuracy and robustness of ML predictions23-26. They also typically only predict metrics directly corresponding to a single modality (i.e., predict only MDS-UPDRS finger tapping score)22,27. A truly valuable video-based solution for tracking PD motor symptom progression would need to be affordable, accessible, automated, transparent, and able to obtain rich and clinically relevant metrics for holistic evaluation of PD symptoms28,29.

[0172] In this study, we address these needs by (1) publicly releasing a comprehensive kinematics dataset from 31 patients all with parkinsonism, and (2) developing a video-based framework to automatically predict PD motor symptom high versus low severity according to the MDS-UPDRS Part III metrics (total score). Following interpretable ML principles30,31, our primary contribution is to enforce model robustness and interpretability by integrating (1) automatic, data-driven kinematic metric evaluation guided by pre-defined digital features of movement, (2) combination of bi-domain (body and hand) kinematic features, and (3) sparsity-inducing and stability-inducing ML analysis with simple-to-interpret models. We perform a comprehensive kinematic feature stability analysis toAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 identify conserved features across ensemble of models32-34and feature contributions to model outputs via tree SHAP (Shapley Additive exPlanations) analysis35. These elements in our design ensure that the model quantifies clinically meaningful motor features providing new clinical insights for quantification of PD severity, including gait features and finger features, not specifically examined in a classical movement disorders examination. Our framework is enabled by retrospective, single- view videos recorded on consumer-grade devices such as smartphones and digital cameras, thereby eliminating the requirement for specialized equipment. In addition, the framework has the advantage of being able to extract rich and meaningful features from just three to seven seconds of video for efficient training and accurate prediction. Validation of features by ML models are enabled by a leave- one-subject-out cross-validation (CV) scheme, which selects a different patient each time and isolates all samples related to that patient as test samples. This approach has been mirrored in other PD studies to protect against data leakage and ensure unbiased results36,37. Results Clinical and demographic characteristics

[0173] Clinical data were obtained from 31 participants all with parkinsonism and who were evaluated at UCSF as part of a multi-day deep phenotyping cohort study. The protocol included standardized video recordings taken in both “on” and “off” dopaminergic medication states while clinical rating scales were performed. Video kinematic data were retrospectively extracted from the individual subject’s clinical videos. These kinematic data is publicly released as part of this study. The video clips were collected with a single tablet camera, which is mounted on a fixed stand, in a clinical setting and not initially formalized or collected with optimized settings for a computer vision- ML pipeline. Disease severity was scored using the MDS-UPDRS at the time of assessment. All patients had Parkinsonian symptoms at the time of evaluation. Seven patients were later determined to have Progressive Supranuclear Palsy (PSP) and one patient did not meet clear diagnostic criteria and was classified as PUCS (Parkinsonism of uncertain clinical significance)38. Four participants were not taking dopaminergic medication and were only assessed in the “off” medication state. Ten participants exhibited on-state dyskinesia. In terms of wake time spent with dyskinesias (MDS- UPDRS 4.1 score), seven reported less than 25% of their waking day (score = 1) and three reported between 25% to 50% of their waking day (score = 2). In terms of functional impact of dyskinesias on activities and social interactions (MDS-UPDRS 4.2 score), five reported no impact (score = 0) and five reported slight impact (score = 1).Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0

[0174] Patients were dichotomized into two groups associated with low and high Parkinsonian motor symptom severity based on the sample median MDS-UPDRS Part III (motor) score of 32, consistent with the cut-off threshold for mild and moderate-to-severe disease severity recommended in literature39. The average UPDRS III scores of 18.5 and 43.4 for the low and high impairment groups fall under the mild and moderate-to-severe categories, respectively. Table 1 summarizes the clinical and demographic characteristics of patients with low (n = 33) and high (n = 25) severity motor symptoms (see also FIG.5).

[0175] The dichotomized groups demonstrated well-balanced characteristics with only age and cognitive profile (MoCA) also showing a difference between the high and the lower severity cohort. The group with high motor symptom severity included patients diagnosed as having PSP, a neurodegenerative disorder with similar Parkinsonian motor symptoms to PD but with more rapid progression4041PSP may also exhibit subtle differences in its patterns of motor deficits as compared to PD (FIG. 6).42Select patients could appear in both high and low severity groups if levodopa medication significantly altered their motor symptom severity to move them from high to low severity. Note that the leave-one-subject-out CV protocol ensures that data from the same subject, irrespective of severity group assignment, may be used only during training or validation and never both. Therefore, the risks of dependency on hidden covariates and potential data leakage are implicitly addressed during training and validation by ensuring that the data used for validation stemmed from patients unseen in training. Automatic extraction of motor features

[0176] We designed a computational framework to automatically extract a large array of features representing movement characteristics in raw, unedited video recordings of Parkinsonian patients performing motor tasks (FIG.1A). A small but highly predictive subset of these features was then selected for training and validation of our ML to predict motor symptom severity quantified by MDS- UPDRS Part III metrics (FIG. 1B). Associated with each patient record was a full-body video of walking / gait, a video of the fingertapping task, or both, collected during the same visit with a standard digital camera. In total, we extracted 40 full-body walking / gait recordings from 25 participants and 48 hand recordings from 27 participants. A comprehensive computer vision pipeline based on deep learning techniques (see Methods) was used to extract kinematic time series from each video recording. The video backgrounds are different but similar clinical scenarios. Since the computer vision pipeline, and CNN-based algorithms in general, is also robust against different backgrounds43, video backgrounds should not be a significant source of noise variance duringAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 extraction. We extracted kinematic time series from 13 major body landmarks14 from each full-body video and kinematic time-series from 8 major hand landmarks from each hand video recording. The landmarks extracted are 3D, but we chose to discard the noisier depth (z) information in favor of normalization for consistent frame-by-frame scaling and comparability of features. This was followed by filtering and sub-segmentation of the videos. As a result, all full-length videos were segmented into short, salient, and error-free video segments (for examples). From full-body videos, we produced 132 video segments with a mean duration of 5.1 ± 1.1 s. From hands videos, we produced 195 video segments with a mean duration of 5.7 ± 1.0 s (FIG.7). Removing those with improper or missing associated data entries, we retained 126 full-body video segments and 189 hand video segments. Based on the clinical dichotomization, 65 body video segments and 83 hands video segments were labeled as exhibiting less severe motor symptoms, whereas 61 and 106, respectively, were labeled as exhibiting more severe motor symptoms. The class assignments for the video segments were roughly balanced with marginally higher membership in the “more severe” category.

[0177] Once the landmarks have been extracted, simple first-order relationships between two or three landmarks (e.g., arm-body lateral angle, thumb-index distance; see Methods for complete list of relationships) were defined, computed, and used for subsequent analysis, provided that they make physiological sense and are not occluded in the videos. The concept behind this design was that some or all of the relative kinematic time series should capture aspects of high-level movement features such as stride patterns or finger-tapping consistency. Then, motor features used in classification were computed based on the relationship time series and their time derivatives based on select temporal and spectral kinematic metrics (see Methods). In total, our framework generated 339 features for each of the 126 video segments from the body and 105 features for each of the 189 video segments from the hands. Moreover, to unify the motor modalities, we also integrated body and hand feature sets via pairwise concatenation of body and hand feature vectors associated with the same patient with the same medication status. This procedure created 895 combined feature vectors with 496 features. 417 of the combined vectors are labeled as “less severe” and 478 as “more severe”. In all cases, large numbers of salient features were extracted from each video recording without any manual tracking. However, the high-dimensional nature of the feature sets posed a challenge to feature interpretability. To identify a minimal and optimal feature subset for the classification task, we introduced a sparsity-inducing feature selection module based on the least absolute shrinkage and selection operator (LASSO)44 technique to the classification framework. This module identified on average 60.48 ± 16.52, 8.60 ± 6.47, and 31.65 ± 29.02 features most important for severity prediction among the body, hand, and combined features, respectively (FIG.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 1C). The significantly reduced feature set sizes and enabled further analysis and interpretation of trained models for generating clinically relevant insights. Classification of motor symptom severity based on extracted features

[0178] To demonstrate that the automatically extracted and sparsified motor features have high predictive power in discriminating between low and high parkinsonian motor symptom severity states, we trained seven different ML models with the generated features (see Methods). We then quantified the classification performances using classification accuracy and average area under receiver operating characteristics curve (AUC) scores estimated with leave-one-subject-out cross- validation (CV) (FIG.2, FIG.8; see Methods)45. In the interest of retaining sufficient sample sizes, we chose to retain data from patients with clinical parkinsonism secondary to PSP and PUCS as well. We also repeated the analysis with only PD patients, which demonstrated only mildly reduction of classification performances across the board (FIG.9), which could be the combined effects of (1) less statistical power from fewer samples; (2) removing significant influence of PSP and PUCS motor patterns; (3) increased imbalance in low versus high motor symptom groups.

[0179] When the models were trained and evaluated on the body features, logistic regression (LR), support vector machine (SVM), and Gaussian naive Bayes (GNB) classifiers achieved the highest average AUC of 0.76, 0.75, and 0.75, respectively. They were closely followed by linear discriminant analysis (LDA) and k-nearest neighbors (KNN) classifiers, at 0.72 and 0.71 average AUC scores, respectively. The remaining ensemble models, random forest (RF) and adaptive boosted trees (AB), had the chance-level average AUC scores of 0.53 and 0.49. Performance measured by classification accuracy were similar in relative ranking, with SVM and LR achieving the highest average classification accuracy of 71% and 70%, respectively. Lower but comparable classification accuracies of 68%, 67%, and 67% were observed in LDA, KNN, and GBN classifiers, respectively. RF and AB achieved near-random accuracies of 54% and 51%. Overall, linear models trained with body features demonstrated decent classification performances. This suggests that the body features, parsed from only a few seconds of walking footage, formed an acceptable representation of the motor characteristics of the corresponding patients.

[0180] For the models trained and evaluated using hand features, most classifiers achieved average AUC scores between 0.67 and 0.69, with LR being the most performant at 0.69, closely followed by AB, SVM, and RF at 0.68, 0.67, and 0.67 respectively. LDA and KNN were the least performant at 0.62 and 0.61 AUC. Similarly, most classifiers achieved the average classification accuracy of 66- 67%, with the exception of LDA and KNN at 63% and 61%, respectively. Similarly, overall, theAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 classification performances of most models trained with hand features were slightly poorer compared to models trained with body features. This may be explained by the fact that movement characteristics measurable by the finger-tapping task are specific and limited, whereas whole-body motor assessments, such as the walking task used in this investigation, may contain richer information for diagnosing the overall level of motor impairment.

[0181] Integrating both body and hand features into a single model sustained or improved the classification performances. SVM and LR achieved the highest average AUC scores at 0.78 and 0.79, outperforming the single modality models. LDA, KNN, and GNB achieved lower AUC scores of 0.71, 0.68, and 0.68, consistent with or slightly lower than their counterparts trained with only body or hand features. RF and AB achieved the lowest scores of 0.63 and 0.61. In terms of classification accuracy, SVM and LR achieved an average accuracy of 72%, with LDA, KNN, and GNB following closely at 68%. The least performant RF and AB classifiers achieved average accuracies of 61% and 58%, which still offered an improved lower bound on the accuracies compared to training only with body features. Overall, integrating body and hand features led to mildly improved performances in the most accurate models and similar performances in the remaining models. The addition of hand features reduced the size of the feature set to evaluate but may or may not have contributed significantly to the overall additional variance explained in the motor characteristics. Clinical insights from feature stability analysis

[0182] The classification performances demonstrated that our framework is capable of extracting optimal features for discriminating low and high motor symptom severity. However, direct interpretation of trained ML models is usually challenging due to variance in LASSO feature selection. The variance was a consequence of distinct data partitioning during CV. To allow insight into the most important features and their contributions to model outputs, we performed ensemble feature stability analysis, aggregating selection counts of features over all leave-one-subject-out CV iterations for each type of classifier model. Specifically, we considered features selected in at least 50% of all iterations as “stable” based on the ensemble paradigm32, which is a trade-off between retaining all potentially relevant features selected in any CV iteration and retaining only features consistently selected in all iterations to facilitate interpretation.

[0183] Performing the analysis on the most performant combined feature set, we identified 9 body and 5 hand features with high stability for interpretation. For all identified features, there exist statistically significant differences (p ≪ 0.05; independent 2-sample t-test) in group means betweenAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 the low and high motor symptom severity groups (FIG. 3A). Visualizations of the landmark relationships from which metrics were derived can be seen in FIG.3C.

[0184] The disparities between feature values in the dichotomized groups and their impact on deciding the severity state of patients are compatible with the current clinical understanding of PD and parkinsonism symptoms. Specifically, in patients with higher symptom severity, the model pulled out higher variability in the inter-ankle distance during walking, suggestive of unsteadiness. Additionally, higher movement of the pinkie finger was seen during the finger tapping task which is strictly meant to be restricted to index finger movements only. This may reflect an increase in rigidity or loss of movement discriminability between fingers in more severe clinical Parkinsonian cases. This feature is not classically specifically interrogated in conventional clinical examination of Parkinson’s disease. More severe cases also had lower hip-ankle variability during gait, reflective of hip flexion reductions and gait “shuffling”. Increased axial rigidity was supported by reduced movement in the neck (relative power in the 2-4 Hz of neck angle) during gait as well as reductions in finger tapping as measured by the finger tapping speed and amplitude.

[0185] We also found that the power spectral density for arm-body lateral angular velocity has a distribution shift with the severity of the motor symptoms. Specifically, the relative power in 4-6 Hz of arm-body lateral angular velocity is lower for patients with more severe motor symptoms, while the relative power in 0.5-1 Hz is lower for patients with less severe symptoms (FIG.10). This suggests that with more severe PD motor symptoms, patients have increased lower-frequency changes and reduced higher-frequency changes in arm body lateral angle, suggestive of axial rigidity.

[0186] Similarly, there is a distribution shift for the power spectral density of finger tapping speed. Patients with more severe symptoms have higher relative power in 4-6 Hz of their finger tapping speed suggesting jerky / sudden interruptions to regular tapping as this frequency range is higher than the actual tapping frequency. The relative power in 2-4 Hz of the finger tapping speed decreases with severity consistent with incomplete finger tapping for patients with more severe motor symptoms.

[0187] The directions of the group differences were consistent with the SHAP values, which measure the impact feature values have on model outputs, estimated with a separate LR classifier using all 14 features (FIG.3B; see also Table 2 for effect sizes of features). In this case, a higher SHAP value biases the model towards assigning a sample the “less severe” label. Features whose values have a positive correlation with the estimated SHAP values were consistent with features with higher group means for participants with less severe motor symptoms. Moreover, based on the mean absoluteAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 SHAP value of each feature, which reflects overall feature attribution, body features had the most predictive power, higher than hand features associated with finger tapping.

[0188] To further validate the optimality of the selected stable features, we compared the 2D projection results of each feature set (body, hand, combined) with and without stable feature selection (FIG.4). Here, the projection algorithm used was MDS (multidimensional scaling), which tends to preserve the global structures present in the data. When all features generated by our framework were retained prior to projection, the final class clusters were poorly separated in all three cases (FIG. 4A). In contrast, the class clusters saw improved linear separability when only stable features were retained prior to projection. Samples containing body or combined features experienced the most significant increase in separability, consistent with the high classification performance in models trained with them. (FIG.4B). These results suggest that the stable features identified through ensemble analysis well-characterized the low and high motor impairment groups and are optimal for the classification task at hand. In conclusion, this form of ensemble analysis not only unified the CV iterations and increased model interpretability but also helped to identify and preserve the most important features. Discussion

[0189] Our study demonstrates that an interpretable computer vision-ML pipeline is able to accurately classify Parkinsonian motor severity (high versus low) using very brief video recordings of gait and finger tapping. The classification accuracy was mildly improved using a combined pipeline that incorporated both walking and finger movements, which also condensed the number of stable movement features to evaluate for easy interpretability. The combined model classification was competitive to previous studies that have relied upon complex and extensive hardware solutions including wearable sensors and prolonged, formalized clinical videos. Notably, our study was performed using only very short video clips from a retrospective clinical library; video recordings were not performed in a laboratory under standardized conditions nor optimized for computer vision or machine learning. This supports translation and scalability to real-world applications both in-clinic and potentially extended to outside the clinic.

[0190] Clinical movement disorder diagnosis and tracking has shown limited fundamental conceptual advancement over the course of the last 200 years, relying mainly on subjective (expert) recognition and classification of symptoms from visual inspection. This is unsurprisingly significantly limited by inter-rater variability and lack of reproducibility. However, in addition to providing an accurate, objective, evaluation of Parkinsonian motor severity, unlike many ML schemes, ourAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 algorithm has the potential to provide, rather than obscure, clinical insight, through bespoke feature stability analysis. Here the algorithm identified a number of classic Parkinsonian features including finger tapping speed and arm swing during gait which serve as useful validation. Moreover, our pipeline identified other features including pinky finger stability during the finger tapping task (likely independent from dyskinesia; see FIG.11) and neck angle features that appear to drive classification and might serve as useful features for clinical classification, that may to date have been underappreciated in medical training. Overall, the stability-driven approach serves to (1) allow clinical validation of identified features (by checking against clinical knowledge), (2) objective quantification of feature importance, and (3) identification of new features with clinical potential. This serves to build clinical confidence in the pipeline, but this technique could also support improved clinical education with objective validation of clinical phenotyping. A bidirectional relationship between clinical expertise and ML-led feature identification has the potential to improve classification accuracies and clinical knowledge.

[0191] Here our algorithm was used to classify high versus low Parkinsonian severity instead of predicting MDS-UPDRS scores directly with regressive models due to limited data availability. In the future, with larger datasets this could be further trained to provide objective, quantitative, diagnostic evaluations that could support classical clinical diagnostic pathways. In addition to single time point evaluations, this approach could in principle be extended to at-home implementation. This has the potential to support chronic symptom tracking in order to monitor naturalistic motor fluctuations over time toward personalized optimization of therapy. Online pose estimation (with immediate deletion of raw video footage) plus automated person recognition have the potential to address potential privacy and security concerns.

[0192] The accuracies reported in our study are not directly comparable to existing studies which reported relatively higher classification accuracy differentiating PD from healthy control using movement data (average accuracy of 89.1 ± 8.3%)46for three primary reasons: First, our study classifies severity in patients diagnosed with Parkinsonian symptoms (rather than against healthy controls) and attempts to predict high versus low total MDS-UPDRS Part III score. This is a more difficult task compared to the classification of finger tapping or gait scores, which is primarily used in past studies. Second, our approach is designed to use short (~ 5) second-long video segments captured by consumer-grade devices such as smartphones, tablets, simple digital cameras. Out-of- the-box pose estimation by Media- Pipe may also, but not always, be suboptimal for subjects under pathological conditions47-49. The pipeline may benefit from tracking algorithms trained specifically on precision-labeled, disease-specific data to meet the demands of high accuracy and specificity forAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 clinical use. Previous studies, on the other hand, benefit from the movement data collected using a variety of camera-based, sensor-based, or other miscellaneous recording devices captured over multi-minute-long recordings. Therefore, they lacked the technical simplicity offered by our framework50. Third, we prioritize model interpretability over performance to determine highly stable and predictive features of hand and body, providing reliable clinical insights. Tangentially, preliminary analyses showed that the inclusion of non-motor confounding variables, such as cognitive metrics (e.g., MoCA, UPDRS.1) and demographic information (e.g., age, sex), may markedly improve the accuracy of the pipeline as a diagnostic tool. Specifically, the performance-boosting capacity of cognitive metrics illustrates the multi-systemic and cross-domain interactive characteristics of PD. Nevertheless, inclusion of confounding variables could distract from the detection of salient motor features that may inform clinical decisions. Therefore, we chose not to include them in the main analysis. But their potential in aiding automated diagnosis, particularly if interpretability is less of focus, are important and warrant further investigation.

[0193] There are a number of limitations to the present study. First, the relatively small cohort size might limit or bias the model performances reported. The distribution of MDS-UPDRS Part III scores of our cohort does not cover the entire score space (0 to 132 points), which could bias the classification performances and feature findings reported. However, UPDRS scoring is a non-linear scale with significant right-sided skew and clinical evidence and published data supporting a median score on the lower side, consistent with our score distribution. Many findings presented herein are also consistent with the current understandings of PD motor characteristics at lower vs. higher PD severity states in general39,51,52. These suggest that the pipeline can find clinically valid and meaningful features despite presence of fewer more advanced severity states. Nevertheless, a larger cohort of patients is required to fully validate the framework’s reported performance and objectivity, particularly with respect to important features identified. Ideally, the cohort should contain more diverse subjects, with patients at all stages of disease progression and with healthy controls.

[0194] Having a larger cohort could enable fine-grain analysis such as regression on UPDRS scores to validate the framework’s utility in obtaining detailed clinical diagnoses. Moreover, inclusion of healthy controls could allow for direct comparative analysis against competing methods in the context of performance and clinical relevance. Second, the performances reported may be affected by the quality of the video data, which was not collected in a manner specifically designed and optimized for a computer vision task. This has resulted in some suboptimal data unsuitable for ML analysis. Standardization of video acquisition and analysis protocols will be beneficial for the framework’s objective assessment of PD symptoms, as well as adaptability and scalability. However, this alsoAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 demonstrates the robustness of our technique to real-world clinical application. Additional pre- processing should also be explored to increase the pose position estimate validity, including evaluation of a wider suite of available pose estimation computer vision software packages. However, this dataset represents a floor to classification accuracy. Finally, we chose to perform stability analysis based on selection frequency by LASSO during training to promote feature interpretability. However, the method might be overly conservative and ignorant of some important features, especially those with collinearity. Future developments could consider methods such as creating “proto-features” from clusters of correlated features53,54, which might better preserve important features.

[0195] In conclusion, our framework effectively expands upon previous research in PD quantification and addresses many of the shortcomings for a simple yet comprehensive video-based solution. Analyzing a cohort of Parkinsonism patients with the proposed framework, we showed that our approach extracted and identified salient kinematic features that could be used to train accurate ML models for predicting low- and high-severity states for motor impairment with high accuracy. Follow- up studies should focus on further refining the framework, increasing the degree of automation, and validating it in larger, representative cohorts. The framework should also be extended to incorporate additional motor modalities, such as facial expressions and speech, as well as non-motor modalities such as neurological data. Future directions should also include exploring the framework’s utility in predicting other clinically relevant outcomes in PD and in application to other neurological movement disorders such as dystonia and essential tremor. Methods Participants and assessment of motor symptoms

[0196] A cohort of patients presenting with Parkinsonism symptoms, including idiopathic PD and PSP, was recruited at the UCSF Movement Disorder and Neuromodulation Center. Qualitative and Quantitative assessments of motor and non-motor symptoms of participating patients were conducted by a movement disorders neurologist based on MDS-UPDRS, the H&Y scale, and the MoCA scale. The main assessment metric used in this study for measuring motor symptom severity was the MDS-UPDRS Part III score22,55. The MDD-UPDRS uses an ordinal scale ranging from 0 to 132, where higher values indicate greater motor impairment. Motor assessments using MDS- UPDRS Part III were conducted for the “off” and “on” dopaminergic medication states. For the “off” state, participants were withdrawn from their medications for a minimum of twelve hours prior toAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 assessment. For the “on” state, participants were given a standard morning dose of Levodopa medication and evaluated one hour later. Not all patients were prescribed dopaminergic medication, and thus some were only assessed in the “off” state. Concurrently with the clinical assessments, video recordings of participants were captured with a consumer-grade iPad tablet recording device at full HD (1920 × 1080) resolution and 30 frames per second, mounted on a tripod in the clinical care area. During recording, participants were instructed to perform finger-tapping and gait tasks as described in MDS-UPDRS sections 3.4 and 3.10. For the finger-tapping task, participants were instructed to tap their index finger on their thumb a minimum of ten times with maximally achievable speed and amplitude. For the gait task, participants were instructed to walk toward and away from the camera for a minimum of 10 meters (30 feet) each way. The resultant videos underwent no additional editing. All participants provided written consent forms for the use of personal health information in research and release. Privacy and confidentiality protection have been explicitly addressed with UCSF IRB approval. The raw videos were not authorized for release. Pose estimation and signal processing

[0197] For each video, kinematic (movement) time series at select body and hand landmarks (FIG. 12) were extracted using the Pose and Hands tracking solutions from MediaPipe14. MediaPipe is an open-sourced framework for building multimodal machine learning pipelines. It is cross-platform (server, iOS, Android) and uses a graph-based pipeline to perform processing and inference functions on multimodal input streams, such as vision and audio. Through this framework, we employ pre-trained hand and pose models, which were trained on ~30 k and 85 K (25 k of which performing fitness exercises), respectively. All of these images were annotated by humans56. The hand model infers 213D landmarks of a hand from a single frame, while the pose model predicts the location of 33 pose landmarks. We found empirically that the depth (z) components of the extracted 3D landmarks are noisier and more inaccurate, thereby not informative to the analysis. Therefore, the depth components are discarded upon extraction, resulting in 2D x,y-coordinates to be normalized downstream. The two-stage pipeline used by MediaPipe for pose and hand tracking consists of (1) an autoencoder detector, similar to feature pyramid networks, for finding bounding box for pose or hand57, and (2) an convolutional encoder tracker for landmark localization informed by the bounding box. While current state-of-the-art approaches rely primarily on powerful desktop environments for inference, MediaPipe pose and hand models achieve real-time performance on mobile phones. MediaPipe, as well as alternatives such as DeepLabCut and OpenPose, have been adopted for accurate extraction of kinematic data from video recordings for clinical analysis of PD24,58-60.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0

[0198] Moderate Gaussian smoothing was applied to the extracted signals to reduce random fluctuations due to tracking inaccuracies, implemented as a weighted sum over a 5-point rolling Gaussian window (σ = 0.5) with the python data manipulation library pandas (version 1.4.0)61. Time points at which major tracking errors in any time series, e.g. missing data, invalid numerics, severe flickering, occurred were identified and marked for removal. Specifically, severe flickering was detected by checking for rapid zero-crossing in distance between shoulders. Time points at which specific posing requirements were not met were also marked for removal. For body recordings, the requirements were that the subject being filmed must be fully standing and roughly facing forward or backward relative to the camera. We determine if a subject is standing by verifying that the apparent lower leg length (ankle-knee distance) is at most twice the apparent upper leg length (hip-knee distance). We approximately determine if a subject is facing forward or backward by verifying that the horizontal distances of left-wrist- to-left-hip and right-wrist-to-right-hip are of opposite signs, i.e. the hands are on opposite sides of the body. For hand recordings, the requirement was that one and only one hand must be raised (and assumed to be engaged in active finger-tapping). We determine this by verifying that the vertical distances of left and right index-finger-to-wrist are of opposite signs, i.e. one hand is pointed up and the other down. For each invalid time point identified, a time window within 0.333 s (10 time points) from the invalid point was removed from all extracted time series.

[0199] As a byproduct, the filtering operation effectively segmented the movement data, resulting in sets of short time series at key landmarks grouped by video segments. To equalize the video segments with respect to duration (and by extension the amount of contained information), segments with a duration less than 3 s were discarded and segments with a duration greater than 8 s were further sub-segmented. The minimum and maximum duration values were empirically chosen such that each segment contained sufficient kinematic information, i.e. multiple stride or finger-tapping cycles, for non-trivial extraction of motor features while maximizing the number of samples available for subsequent model training and classification. Although the segmentation process is partially driven by tracking error, there are no statistically significant differences in segment durations between low and high severity groups (FIG.7C). Computing relative movement measurements

[0200] Similar to strategies seen in existing literature60, we generated relational time series from the filtered and segmented kinematic time series based on predefined interactions between two or more landmarks. In most studies, a small subset of distance or angular relationships between select landmarks may be chosen to be evaluated with select few (sometimes complex) metrics based onAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 experimental or clinical expectations. Here, we chose to include many first-order relationships between available landmarks as long as they have physical and physiological meaning (e.g. limb or body angle, distance, speed) and are not occluded in the videos, followed by blanket application of a relatively large set of kinematic and spectral metrics (e.g., mean, standard deviation, spectral entropy). These relationships between landmarks, which are time series, collectively capture aspects of healthy and pathological movement characteristics without explicit tailoring of feature metrics to each relationship. This process enables automation, increases information to include, and encourages novel feature discoveries. The list of time series selected is provided here: 1. Neck angle (angle between nose, mid-shoulder, and left shoulder) 2. Arm-body lateral angle (angle between elbow, shoulder, and hip, lateral to body) 3. Left / right wrist-shoulder distance 4. Left / right ankle-hip distance 5. Ankle separation (horizontal displacement between ankles) 6. Knee separation (horizontal displacement between knees) 7. Body sway (horizontal displacement between mid-shoulder and mid-hip) 8. Leg raise (vertical displacement between ankles) 9. Shoulder tilt (vertical displacement between shoulders) 10. Hip tilt (vertical displacement between hips) 11. Finger tapping (distance between thumb and index finger of active hand) 12. Middle finger movement (distance between middle finger and wrist of active hand) 13. Ring finger movement (distance between ring finger and wrist of active hand) 14. Pinky finger movement (distance between pinky finger and wrist of active hand)

[0201] The pose features listed above reflect distance, displacement, and angle measures between landmarks. To produce information on speed, velocity, and angular velocity for analysis, we also compute the first-order derivatives of the time series, resulting in equivalent numbers of additional time series for further processing. Note that the filtering step ensured that only one hand will be actively performing finger-tapping for any given time segment; therefore, distinguishing between left and right-sided hand movements was unnecessary as the resting hand will necessarily have little to no movement and could have its time series discarded.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 Time series assessment with kinematic metrics

[0202] Once we computed the relational time series, as the apparent body and hand sizes change depending on the distance between the subject and the camera, we normalized the new time series at each time point to equalize the scale and allow direct comparison. For body signals, normalization was achieved by dividing the time series by the body length (distance between mid-shoulder and mid-hip) and centered mid-hip. For hand signals, normalization was achieved by dividing the time series by the palm length (distance between wrist and midpoint between bases of index and pinky fingers) and centering at the wrist. The normalization was followed by kinematic metric evaluation of each time series, during which various temporal (distribution-based) and spectral (spectral density- based) metrics were used to provide a statistical summary of the body and hand movement characteristics (Table 3). This process was repeated for the derivatives of the relational time series, calculated as the rate of change between adjacent time points.

[0203] The computed metric values formed the feature vectors associated with each time segment and were used in subsequent ML training and classification. Since only one hand is performing active finger-tapping at any time in the videos, features from the non-active hand are discarded. Considering the common unilateral development of PD symptoms, we needed to retain both left and right-sided body features. This posed a challenge to compare features between subjects whose manifestation of unilateral features might be on opposite sides of the body. To address this, we recategorized left and right-sided features of the same type as “minimum” and “maximum” features by ranking numerically, thereby eliminating the sidedness of features. In addition, to unify the body and hand kinematic features and increase classification performance, we generated combined feature vectors from all valid combinations of body and hand feature vectors, so long as they were associated with the same patient record, i.e. same patient under the same medication status. Feature selection and classification

[0204] The scikit-learn package (version 1.1.2) in Python was used for feature selection and classification on low and high motor symptom severity for all three feature sets generated by our framework. To generate training and validation data in a robust manner, we conducted the aforementioned procedure with a custom leave-one-subject-out CV, repeated 16 times to account for model variability due to hyperparameter tuning. The leave-one-subject-out protocol selects a different patient at each CV iteration and isolates all samples related to that patient as the testing set. This form of CV is necessary to prevent dependency on hidden confounding variables and thus data leakage. Once partitioned, the training and validation sets were centered and normalizedAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 feature-wise based on means and standard deviations from the training set. Then, to reduce feature dimensionality, address multicollinearity, and improve interpretability, we applied to all feature sets as a pre-training step LASSO feature selection, which is an L1-regularized, sparsity-inducing algorithm. Finally, classification performances were evaluated using seven different ML algorithms: (1) linear discriminant analysis (LDA)62; (2) logistic regression (LR)63; (3) support vector machine (SVM)64; (4) random forest (RF)65; (5) adaptive-boosted trees (AdaBoost; AB)66; (6) K-nearest neighbors (KNN)67; and, (7) Gaussian naive Bayes (GNB)68. The regularization parameter for LASSO feature sparsity, as well as relevant hyperparameters for each classification model, were chosen via hyperparameter optimization with nested leave-one- subject-out CV during each CV iteration. Classification accuracies and AUC scores were recorded for each classifier, where accuracy measures the model’s ability to classify on the defined class labels and AUC measures the model’s sensitivity and generalizability by looking at all probabilities for assigning labels. Accuracy and AUC scores during CV were aggregated and reported as mean ± standard deviation. Feature stability analysis

[0205] To address the challenge of interpreting membership variance in salient feature subsets selected via LASSO during CV, we performed feature stability analysis based on frequencies of selection. Here, we defined a feature as being stable if it was selected by LASSO in at least 50% of all CV iterations. Once the stable features have been identified, their contributions to model predictions were evaluated with SHAP analysis on the most performant model using the python package SHAP (version 0.40.0). Additional validation was performed by comparing the 2D projections of the datasets with and without stable feature selection. The algorithm used for projection was multidimensional scaling (MDS), an unsupervised dimensionality reduction algorithm that preserves global structures in the data. Statistical analysis

[0206] Continuous variables were presented as mean ± standard deviation and compared between low and high motor scoring groups with an independent 2-sample t-test if normally distributed or with the Mann-Whitney U-test if otherwise. Categorical variables were presented as counts and compared between scoring groups with Fisher’s exact test, which is analogous to the Chi-squared test but suitable for small-sized samples. A p-value less than 0.05 was considered statistically significant.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 References

[0207] 1. Fasano, A. et al. Characterizing advanced Parkinson’s disease: OBSERVE-PD observational study results of 2615 patients. BMC Neurol.19, 50 (2019).

[0208] 2. Sanchez-Luengos, I. et al. Predictors of health-related quality of life in Parkinson’s disease: the impact of overlap between health-related quality of life and clinical measures. Qual. Life Res. Int. J. Qual. Life Asp. Treat. Care Rehabil.31, 3241-3252 (2022).

[0209] 3. Bloem, B. R., Okun, M. S. & Klein, C. Parkinson’s disease. Lancet Lond. Engl.397, 2284- 2303 (2021).

[0210] 4. Armstrong, M. J. & Okun, M. S. Diagnosis and Treatment of Parkinson Disease: A Review. JAMA 323, 548-560 (2020).

[0211] 5. Goetz, C. G. et al. Movement Disorder Society-sponsored revision of the Unified Parkinson’s Disease Rating Scale (MDS-UPDRS): scale presentation and clinimetric testing results. Mov. Disord. J. Mov. Disord. Soc.23, 2129-2170 (2008).

[0212] 6. Löhle, M. et al. Validation of the PD home diary for assessment of motor fluctuations in advanced Parkinson’s disease. Npj Park. Dis.8, 69 (2022).

[0213] 7. Zolfaghari, S. et al. Self-Report versus Clinician Examination in Early Parkinson’s Disease. Mov. Disord. J. Mov. Disord. Soc.37, 585-597 (2022).

[0214] 8. Huckvale, K., Venkatesh, S. & Christensen, H. Toward clinical digital phenotyping: a timely opportunity to consider purpose, quality, and safety. Npj Digit. Med.2, 88 (2019).

[0215] 9. Matias, R., Paixão, V., Bouça, R. & Ferreira, J. J. A Perspective on Wearable Sensor Measurements and Data Science for Parkinson’s Disease. Front. Neurol.8, 677 (2017).

[0216] 10. Navani, D., Block, V., Cree, B. & Abbasi-Asl, R. Diurnal Step Count Patterns in Progressive Multiple Sclerosis (P3-3.011). Neurology 100, 3320 (2023).

[0217] 11. Habets, J. G. V. et al. Rapid Dynamic Naturalistic Monitoring of Bradykinesia in Parkinson’s Disease Using a Wrist-Worn Accelerometer. Sensors 21, 7876 (2021).

[0218] 12. Pang, Y. et al. Automatic detection and quantification of hand movements toward development of an objective assessment of tremor and bradykinesia in Parkinson’s disease. J. Neurosci. Methods 333, 108576 (2020).

[0219] 13. Khodakarami, H. et al. Prediction of the Levodopa Challenge Test in Parkinson’s Disease Using Data from a Wrist-Worn Sensor. Sensors 19, 5153 (2019).

[0220] 14. Lugaresi, C. et al. MediaPipe: A Framework for Building Perception Pipelines. doi.org / 10.48550 / ARXIV.1906.08172. (2019)Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0

[0221] 15. Mathis, A. et al. DeepLabCut: markerless pose estimation of user- defined body parts with deep learning. Nat. Neurosci.21, 1281-1289 (2018).

[0222] 16. Graving, J. M. et al. DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning. eLife 8, e47994 (2019).

[0223] 17. Cao, Z., Simon, T., Wei, S.-E. & Sheikh, Y. Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. doi.org / 10.48550 / ARXIV.1611.08050. (2016).

[0224] 18. Park, K. W. et al. Machine Learning-Based Automatic Rating for Cardinal Symptoms of Parkinson Disease. Neurology 96, e1761-e1769 (2021).

[0225] 19. Espay, A. J. et al. A roadmap for implementation of patient-centered digital outcome measures in Parkinson’s disease obtained using mobile health technologies. Mov. Disord. J. Mov. Disord. Soc.34, 657-663 (2019).

[0226] 20. Sibley, K. G., Girges, C., Hoque, E. & Foltynie, T. Video-Based Analyses of Parkinson’s Disease Severity: A Brief Review. J. Park. Dis.11, S83-S93 (2021).

[0227] 21. Van Kersbergen, J. et al. Camera-based objective measures of Parkinson’s disease gait features. BMC Res. Notes 14, 329 (2021).

[0228] 22. Morinan, G. et al. Computer vision quantification of whole-body Parkinsonian bradykinesia using a large multi-site population. Npj Park. Dis.9, 10 (2023).

[0229] 23. Baltrusaitis, T., Ahuja, C. & Morency, L.-P. Multimodal Machine Learning: A Survey and Taxonomy. IEEE Trans. Pattern Anal. Mach. Intell.41, 423-443 (2019).

[0230] 24. Lim, W. S. et al. An integrated biometric voice and facial features for early detection of Parkinson’s disease. Npj Park. Dis.8, 145 (2022).

[0231] 25. Makarious, M. B. et al. Multi-modality machine learning predicting Parkinson’s disease. Npj Park. Dis.8, 35 (2022).

[0232] 26. Rupprechter, S. et al. A Clinically Interpretable Computer-Vision Based Method for Quantifying Gait in Parkinson’s Disease. Sensors 21, 5437 (2021).

[0233] 27. Sarapata, G. et al. Video-Based Activity Recognition for Automated Motor Assessment of Parkinson’s Disease. IEEE J. Biomed. Health Inform.27, 5032-5041 (2023).

[0234] 28. Lonini, L. et al. Wearable sensors for Parkinson’s disease: which data are worth collecting for training symptom detection models. Npj Digit. Med.1, 64 (2018).

[0235] 29. Sica, M. et al. Continuous home monitoring of Parkinson’s disease using inertial sensors: A systematic review. PLoS One 16, e0246528 (2021).Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0

[0236] 30. Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R. & Yu, B. Definitions, methods, and applications in interpretable machine learning. Proc. Natl Acad. Sci. USA 116, 22071-22080 (2019).

[0237] 31. Abbasi-Asl, R. & Yu, B. Structural Compression of Convolutional Neural Networks with Applications in Interpretability. Front Big Data 4, 704182 (2021).

[0238] 32. Exploiting the ensemble paradigm for stable feature selection: A case study on high- dimensional genomic data. Inf. Fusion 35, 132-147 (2017).

[0239] 33. Tolosi, L. & Lengauer, T. Classification with correlated features: unreliability of feature ranking and solutions. Bioinformatics 27, 1986-1994 (2011).

[0240] 34. Nogueira, S., Sechidis, K. & Brown, G. On the stability of feature selection algorithms. J. Mach. Learn Res.18, 6345-6398 (2017).

[0241] 35. Lundberg, S. M. et al. From Local Explanations to Global Understanding with Explainable AI for Trees. Nat. Mach. Intell.2, 56-67 (2020).

[0242] 36. Gunduz, H. Deep Learning-Based Parkinson’s Disease Classification Using Vocal Feature Sets. IEEE Access 7, 115540-115551 (2019).

[0243] 37. Jha, A. et al. The CloudUPDRS smartphone software in Parkinson’s study: cross- validation against blinded human raters. NPJ. Park. Dis.6, 36 (2020).

[0244] 38. Zitser, J. et al. Parkinsonism of uncertain clinical significance (PUCS): A proposed new diagnostic entity. J. Neurol. Sci.451, 120696 (2023).

[0245] 39. Martínez-Martín, P. et al. Parkinson’s disease severity levels and MDS-Unified Parkinson’s Disease Rating Scale. Park. Relat. Disord.21, 50-54 (2015).

[0246] 40. Egerton, T., Williams, D. R. & Iansek, R. Comparison of gait in progressive supranuclear palsy, Parkinson’s disease and healthy older adults. BMC Neurol.12, 116 (2012).

[0247] 41. Cordato, N. J., Halliday, G. M., Caine, D. & Morris, J. G. L. Comparison of motor, cognitive, and behavioral features in progressive supranuclear palsy and Parkinson’s disease. Mov. Disord.21, 632-638 (2006).

[0248] 42. Ling, H., Massey, L. A., Lees, A. J., Brown, P. & Day, B. L. Hypokinesia without decrement distinguishes progressive supranuclear palsy from Parkinson’s disease. Brain 135, 1141- 1153 (2012).

[0249] 43. Wang, X., Garg, S., Tran, S. N., Bai, Q. & Alty, J. Hand tremor detection in videos with cluttered background using neural network based approaches. Health Inf. Sci. Syst.9, 30 (2021).

[0250] 44. Tibshirani, R. Regression Shrinkage and Selection via the Lasso. J. R. Stat. Soc. Ser. B Stat. Methodol.58, 267-288 (1996).Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0

[0251] 45. Hajian-Tilaki, K. Receiver Operating Characteristic (ROC) Curve Analysis for Medical Diagnostic Test Evaluation. Casp. J. Intern. Med.4, 627-635 (2013).

[0252] 46. Mei, J., Desrosiers, C. & Frasnelli, J. Machine Learning for the Diagnosis of Parkinson’s Disease: A Review of Literature. Front Aging Neurosci.13, 633752 (2021).

[0253] 47. Seethapathi, N., Wang, S., Saluja, R., Blohm, G. & Kording, K. P. Movement science needs different pose tracking algorithms. doi.org / 10.48550 / ARXIV.1907.10226. (2019)

[0254] 48. Friedrich, M. et al. Visual perceptive deep learning for smartphone video-based tremor analysis: VIPER-Tremor. Preprint at doi. org / 10.21203 / rs.3.rs-3692906 / v1 (2023).

[0255] 49. Friedrich, M. U. et al. Smartphone video nystagmography using convolutional neural networks: ConVNG. J. Neurol.270, 2518-2530 (2023).

[0256] 50. Kour, N., Sunanda & Arora, S. Computer-Vision Based Diagnosis of

[0257] Parkinson’s Disease via Gait: A Survey. IEEE Access 7, 156620-156645 (2019).

[0258] 51. Evers, L. J. W., Krijthe, J. H., Meinders, M. J., Bloem, B. R. & Heskes, T.

[0259] M. Measuring Parkinson’s disease over time: The real‐world within subject reliability of the MDS‐UPDRS. Mov. Disord.34, 1480-1487 (2019).

[0260] 52. Lang, A. E. et al. Deep brain stimulation: Preoperative issues. Mov. Disord.21, S171- S196 (2006).

[0261] 53. Faletto, G. & Bien, J. Cluster Stability Selection. doi.org / 10.48550 / ARXIV.2201.00494 (2022).

[0262] 54. Ghosal, G. & Abbasi-Asl, R. Multi-modal prototype learning for interpretable multivariable time series classification. ArXiv Prepr. ArXiv210609636 (2021).

[0263] 55. Grover, S., Bhartia, S., Akshama, Yadav, A. & Seeja, K. R. Predicting Severity Of Parkinson’s Disease Using Deep Learning. Procedia Comput Sci.132, 1788-1794 (2018).

[0264] 56. Bazarevsky, V. et al. BlazePose: On-device Real-time Body Pose tracking. doi.org / 10.48550 / ARXIV.2006.10204 (2020).

[0265] 57. Lin, T.-Y. et al. Feature pyramid networks for object detection. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2117-2125 (IEEE, Honolulu, HI, 2017).

[0266] 58. Li, M. H., Mestre, T. A., Fox, S. H. & Taati, B. Vision-based assessment of parkinsonism and levodopa-induced dyskinesia with pose estimation. J. Neuroeng. Rehabil.15, 97 (2018).

[0267] 59. Güney, G. et al. Video-Based Hand Movement Analysis of Parkinson Patients before and after Medication Using High-Frame-Rate Videos and MediaPipe. Sensors 22, 7992 (2022).Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0

[0268] 60. Dentamaro, V., Impedovo, D. & Pirlo, G. Gait Analysis for Early Neurodegenerative Diseases Classification Through the Kinematic Theory of Rapid Human Movements. IEEE Access 8, 193966-193980 (2020).

[0269] 61. McKinney, W. Data Structures for Statistical Computing in Python. In Proc. of the 9th Python in Science Conference (SciPy, Austin, Texas, 2010).

[0270] 62. Tharwat, A., Gaber, T., Ibrahim, A. & Hassanien, A. E. Linear discriminant analysis: A detailed tutorial. AI Commun.30, 169-190 (2017).

[0271] 63. Schober, P. & Vetter, T. R. Logistic Regression in Medical Research. Anesth. Analg. 132, 365-366 (2021).

[0272] 64. Hearst, M. A., Dumais, S. T., Osuna, E., Platt, J. & Scholkopf, B. Support vector machines. IEEE Intell. Syst. Their Appl.13, 18-28 (1998).

[0273] 65. Breiman, L. Random Forests. Mach. Learn 45, 5-32 (2001).

[0274] 66. Freund, Y., Schapire, R. E. & AT&T Labs. A Short Introduction to Boosting. J. Japanese Soc. Artificial Intell.14, 771-780 (1999).

[0275] 67. Taunk, K., De, S., Verma, S. & Swetapadma, A. A Brief Review of Nearest Neighbor Algorithm for Learning and Classification. In 2019 International Conference on Intelligent Computing and Control Systems (ICCS) 1255-1260 (2019).

[0276] 68. Hand, D. J. & Yu, K. Idiot’s Bayes? Not so stupid after all? Int Stat. Rev.69, 385-398 (2001).Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 Table 1: Summary of clinical and demographic characteristics in patient records dichotomized by MDS-UPDRS Part III motor score for each video clip. Low Motor High Motor p-value Impairment Impairment (n = 33) (n = 25) Disease Type PD=33, PSP=0 PD=17, PSP=7, - PUCS=1 Age (yrs) 67.1 ± 7.4 71.4 ± 6.2 0.02*Sex M=20, F=13 M=15, F=10 0.96 Handedness R=28, L=3, A=2 R=21, L=3, A=1 0.89 Education (yrs) 18.3 ± 2.5 17.5 ± 2.3 0.22∗Disease duration (yrs) 7.4 ± 4.0 5.9 ± 3.0 0.14∗Medication status On=18, Off=15 On=9, Off=16 0.16 MDS-UPDRS, Part I score 9.4 ± 5.4 10.4 ± 6.3 0.55∗MDS-UPDRS, Part II score 8.8 ± 7.5 18.4 ± 13.7 0.01#MDS-UPDRS, Part III 18.5 ± 10.3 43.4 ± 8.2 0.00*score Hoehn and Yahr scale 1.9 ± 0.8 3.0 ± 1.1 0.00#MoCA score 26.5 ± 3.2 23.5 ± 5.0 0.01#PD = Parkinson’s disease; PSP = Progressive supranuclear palsy; MDS-UPDRS = Movement Disorder Society-Sponsored Revision of the Unified Parkinson’s Disease Rating Scale; MDS-UPDRS, Part I = non-motor experiences of daily life; MDS-UPDRS, Part II = motor experiences of daily life; MDSUPDRS, Part III = motor examination; MoCA = Montreal Cognitive Assessment. ∗ Independent 2-sample t-test † Mann Whitney U-test Fisher’s exact test Table 2: Effect Sizes of Selected Features in Best Performing Model (LR). Feature Coefficient () Odds Ratio (OR) Pinky Mvmt. Speed [Rel. Pwr. >6 Hz] 1.39 ± 0.78 4.02 ± 2.18 Ankle Separation [Rel. Pwr in 1-2 Hz] 0.80 ± 0.33 2.22 ± 1.39 Ankle Separation [Rel. Pwr. in 0.5-1 Hz] 0.73 ± 0.33 2.07 ± 1.39 Finger Tapping Speed [Rel. Pwr in 4-6 Hz] 0.61 ± 0.37 1.83 ± 1.45 Arm-Body Ang. Vel. [Rel. Pwr in 0.5-1 Hz] 0.57 ± 0.35 1.77 ± 1.42 Pinky Mvmt. Speed [Half Pwr. Freq.] -0.34 ± 0.74 0.71 ± 2.10 Δ Ankle-Hip Dist. [Rel. Pwr. in 2-4 Hz] -0.64 ± 0.33 0.53 ± 1.39 Finger Tapping Amp. [MAD] -0.73 ± 0.33 0.48 ± 1.39 Δ Ankle Separation [Rel. Pwr. in 4-6 Hz] -0.93 ± 0.33 0.40 ± 1.39Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 Arm-Body Ang. Vel. [Rel. Pwr in 4-6 Hz] -1.16 ± 0.35 0.31 ± 1.42 Finger Tapping Speed [Rel. Pwr. in 2-4 Hz] -1.23 ± 0.39 0.29 ± 1.48 Ankle-Hip Dist. [STD] -1.55 ± 0.43 0.21 ± 1.54 Neck Angle [Rel. Pwr. in 2-4 Hz] -2.07 ± 0.59 0.13 ± 1.80 LR = Logistic Regression. OR = ^! represent the effect sizes of features. For each unit of increase in the corresponding standardized feature, OR is the multiplicative factor to the odds of the subject having higher motor deficits. ^ and OR are presented as the mean estimate ± 95% confidence interval. Table 3: Temporal and spectral metrics used to characterize movement measurements extracted from video recordings.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 Example 2 Extension of Video-Based Tracking for Capture of Motor and Non-Motor Symptom Severity

[0277] Our initial work validated an interpretable machine learning pipeline on clinical videos of hand movements and walking to classify parkinsonian severity. We are extending this work by incorporating: 1) at home evaluations using smartphone cameras with videos captured by patients and their caregivers, as opposed to a clinical team; 2) non-motor (mood) and motor evaluations from facial emotion evaluations; and 3) passive data collection through running video capture and pose estimation passively on the phone (with the patient’s consent) whenever they are using their phone.

[0278] These extensions of video-based tracking have the potential to result in a daily capture of motor and non-motor symptom severity from passive phone usage. This could have the potential to track patients' expanded range of motor and non-motor severity continuously in a naturalistic environment using pre-existing technologies. These data can also be analyzed using interpretable machine learning pipelines, expanded from the analysis protocol described in Example 1.

Claims

Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 What is claimed is:

1. A computer implemented method for determining symptom severity of a subject having a movement disorder, the computer performing steps comprising: receiving a first video of the subject’s entire body while the subject is walking and / or receiving a second video of a hand of the subject while the subject is finger-tapping; segmenting the first video and / or the second video to generate a plurality of video segments; identifying body landmarks and / or hand landmarks in the plurality of video segments; using the body landmarks and / or hand landmarks to extract a set of body movement features and / or hand movement features from the plurality of video segments based on relative movement between the landmarks with time; using a machine learning algorithm to determine symptom severity of the subject based on classification of the set of body movement features and / or hand movement features; and displaying information regarding the symptom severity of the subject.

2. The computer implemented method of claim 1, wherein the body landmarks comprise a nose, a right shoulder, a left shoulder, a right elbow, a left elbow, a right wrist, a left wrist, a right hip, a left hip, a right knee, a left knee, a right ankle, and a left ankle.

3. The computer implemented method of claim 1 or 2, wherein the hand landmarks comprise a wrist, a thumb, an index finger, a middle finger, a ring finger, a pinky finger, base of the index finger, and base of the pinky finger.

4. The computer implemented method of any one of claims 1-3, wherein the body movement features comprise: neck angle; arm-body angle; distance between left wrist and left shoulder and / or distance between right wrist and right shoulder; distance between left ankle and hip; horizontal displacement between ankles; horizontal displacement between knees; horizontal displacement between mid-shoulder and mid-hip;Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 vertical displacement between ankles; vertical displacement between shoulders; and vertical displacement between hips.

5. The computer implemented method any one of claims 1-4, wherein the hand movement features comprise: distance between thumb and index finger of active hand that is finger tapping; distance between middle finger and wrist of active hand that is finger tapping; distance between ring finger and wrist of active hand that is finger tapping; and distance between pinky finger and wrist of active hand that is finger tapping.

6. The computer implemented method of any one of claims 1-5, wherein the body movement features further comprise relative power in 0.5 Hz to 1 Hz of ankle separation, standard deviation (STD) of ankle-hip distance, relative power in 2 Hz to 4 Hz of neck angle, relative power in 4 Hz to 6 Hz of arm-body angular velocity, and mean of neck angle.

7. The computer implemented method of any one of claims 1-6, wherein the hand movement features further comprise half-power frequency in pinky finger movement speed, relative power in 2 Hz to 4 Hz of finger tapping speed, and median absolute deviation (MAD) of finger tapping amplitude.

8. The computer implemented method of any one of claims 1-7, wherein the set of body movement features and / or hand movement features are identified by performing sparsity-inducing feature selection on the plurality of video segments.

9. The computer implemented method of any one of claims 1-8, wherein the movement disorder is Parkinson’s disease, progressive supranuclear palsy, or multiple systems atrophy.

10. The computer implemented method of any one of claims 1-9, wherein the machine learning algorithm uses a logistic regression (LR) classifier, a support vector machine (SVM) classifier, or a linear discriminant analysis (LDA) classifier.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 11. The computer implemented method of any one of claims 1-10, wherein the machine learning algorithm uses an artificial neural network.

12. The computer implemented method of any one of claims 1-11, wherein the machine learning algorithm uses a deep learning algorithm.

13. The computer implemented method of claim 12, wherein the deep learning algorithm uses a convolutional neural network, a deep neural network, a recurrent neural network, a deep residual neural network, a long short-term memory network, a deep belief network, a multilayer perceptron, or deep reinforcement learning.

14. The computer implemented method of any one of claims 1-13, wherein the machine learning algorithm is supervised, semi-supervised, or unsupervised.

15. The computer implemented method of any one of claims 1-14, wherein each video segment shows multiple strides during the walking or multiple finger-tapping cycles during the finger- tapping.

16. The computer implemented method of any one of claims 1-15, wherein each video segment is 3 to 8 seconds in length.

17. The computer implemented method of any one of claims 1-16, wherein any video segment less than 3 seconds in length is discarded.

18. The computer implemented method of any one of claims 1-17, wherein any video segment greater than 8 seconds in length is further segmented.

19. The computer implemented method of any one of claims 1-18, further comprising correlating the symptom severity of the subject based on classification of the set of body movement features and / or hand movement features with a Movement Disorder Society-Sponsored Revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS) or a Hoehn and Yahr (HnY) scale, a Parkinson’s Disease Composite Scale (PDCS), or a Schwab and England Activities of Daily Living (ADL) Scale.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 20. The computer implemented method of any one of claims 1-19, further comprising comparing movement metrics for single joint or multi-joint movements of the subject to reference value ranges for a normal age-matched control subject.

21. The computer implemented method of claim 20, wherein the movement metrics are for finger-tapping or gait speed.

22. The computer implemented method of any one of claims 1-21, further comprising: receiving a third video of the subject’s face; segmenting the third video to generate a plurality of video segments; identifying face landmarks in the plurality of video segments; using the face landmarks to extract a set of face movement features from the plurality of video segments based on relative movement between the face landmarks with time; using a machine learning algorithm to determine symptom severity of the subject based on classification of the set of face movement features in combination with the set of body movement features and / or hand movement features; and displaying information regarding the symptom severity of the subject.

23. The computer implemented method of claim 22, wherein the face landmarks comprise a nose, a right eye, a left eye, a right eyelid, a left eyelid, a right eyebrow, a left eyebrow, eye corners, a mouth, an upper lip, a lower lip, a right cheek, a left cheek, a forehead, tongue, jaw, and a chin.

24. The computer implemented method of claim 22 or 23, wherein the set of face movement features is used to detect an abnormal facial movement or an involuntary facial movement associated with a movement disorder.

25. The computer implemented method of claim 24, wherein the abnormal facial movement or the involuntary facial movement is rapid eye blinking, face twitching, face tremor, jaw tremor, lip tremor, lip-smacking, tongue twisting, tongue curling, tongue thrusting, lip puckering, facial drooping, or grimacing.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 26. The computer implemented method of any one of claims 22-25, wherein the movement disorder is Tardive dyskinesia, facial dystonia, hemifacial spasm, peripheral facial palsy, facial myokymia, facial bradykinesia, blepharospasm, Tourette syndrome, orofacial dyskinesia, brainstem tumor, peripheral neuropathy, chorea, multiple sclerosis, Guillain-Barré syndrome, Parkinson’s disease, congenital facial nerve palsy, acquired facial nerve palsy, Moebius syndrome, Poland syndrome, CHARGE syndrome, Hemifacial microsomia, 22q11.2 deletion syndrome, facial paralysis or paresis, or a psychogenic movement disorder.

27. The computer implemented method of any one of claims 22-26, further comprising using the face landmarks for facial recognition, head pose estimation, facial expression detection, or facial emotion recognition.

28. The computer implemented method of claim 27, wherein the facial emotion recognition is used to detect depression, psychosis, fatigue, fear, anger, sadness, or joy.

29. A computer implemented method for determining symptom severity of a subject having a facial movement disorder, the computer performing steps comprising: receiving a video of the subject’s face; segmenting the video to generate a plurality of video segments; identifying face landmarks in the plurality of video segments; using the face landmarks to extract a set of face movement features from the plurality of video segments based on relative movement between the face landmarks with time; using a machine learning algorithm to determine symptom severity of the subject based on classification of the set of face movement features; and displaying information regarding the symptom severity of the subject.

30. The computer implemented method of claim 29, wherein the face landmarks comprise a nose, a right eye, a left eye, a right eyelid, a left eyelid, a right eyebrow, a left eyebrow, eye corners, a mouth, an upper lip, a lower lip, a right cheek, a left cheek, a forehead, tongue, jaw, and a chin.

31. The computer implemented method of claim 29 or 30, wherein the set of face movement features is used to detect an abnormal facial movement or an involuntary facial movement associated with the facial movement disorder.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 32. The computer implemented method of claim 31, wherein the abnormal facial movement or the involuntary facial movement is rapid eye blinking, face twitching, face tremor, jaw tremor, lip tremor, lip-smacking, tongue twisting, tongue curling, tongue thrusting, lip puckering, or grimacing.

33. The computer implemented method of any one of claims 29-32, wherein the facial movement disorder is Tardive dyskinesia, facial dystonia, hemifacial spasm, peripheral facial palsy, facial myokymia, facial bradykinesia, blepharospasm, Tourette syndrome, orofacial dyskinesia, brainstem tumor, peripheral neuropathy, chorea, multiple sclerosis, Guillain-Barré syndrome, Parkinson’s disease, congenital facial nerve palsy, acquired facial nerve palsy, Moebius syndrome, Poland syndrome, CHARGE syndrome, Hemifacial microsomia, 22q11.2 deletion syndrome, facial paralysis or paresis, or a psychogenic movement disorder.

34. The computer implemented method of any one of claims 29-33, further comprising using the face landmarks for facial recognition, head pose estimation, facial expression detection, or facial emotion recognition.

35. The computer implemented method of claim 34, wherein the facial emotion recognition is used to detect depression, psychosis, fatigue, fear, anger, sadness, or joy.

36. A non-transitory computer-readable medium comprising program instructions that, when executed by a processor in a computer, causes the processor to perform the method of any one of claims 1-35.

37. A kit comprising the non-transitory computer-readable medium of claim 36 and instructions for determining symptom severity of a subject having a movement disorder.

38. A system comprising: a video recording device; a processor, wherein the processor is programmed to determine symptom severity of a subject having a movement disorder according to the computer implemented method of any one of claims 1-35; andAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 a display component that displays information regarding the symptom severity of the subject.

39. The system of claim 38, wherein the video recording device is provided by a digital camera, a smartphone, a tablet, a laptop, or a camcorder.

40. The system of claims 38 or 39, further comprising a tripod, wherein the video recording device is mounted on the tripod.

41. The system of any one of claims 38-40, further comprising one or more additional recording devices, wherein each additional video recording device records videos of the subject from a different angle.

42. The system of any one of claims 38-41, wherein the video recording device comprises a laser imaging, detection, and ranging (LIDAR) sensor.

43. The system of any one of claims 38-42, wherein the display further displays information comparing movement metrics for single joint or multi-joint movements of the subject to reference value ranges for a normal age-matched control subject.

44. The system of claim 43, wherein the movement metrics are for finger-tapping or gait speed.

45. A method of using the system of any one of claims 38-44 for determining symptom severity of a subject having a movement disorder, the method comprising: recording a first video of the subject’s entire body while the subject is walking and / or a second video of a hand of the subject while the subject is finger-tapping using the video recording device; and determining symptom severity of the subject having the movement disorder according to the computer implemented method.

46. The method of claim 45, wherein only one hand of the subject is actively performing finger-tapping and the subject’s other hand is resting.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 47. The method of claim 45 or 46, wherein the body landmarks comprise a nose, a right shoulder, a left shoulder, a right elbow, a left elbow, a right wrist, a left wrist, a right hip, a left hip, a right knee, a left knee, a right ankle, and a left ankle.

48. The method of any one of claims 45-47, wherein the hand landmarks comprise a wrist, a thumb, an index finger, a middle finger, a ring finger, a pinky finger, base of the index finger, and base of the pinky finger.

49. The method of any one of claims 45-48, wherein the body movement features comprise: neck angle; arm-body angle; distance between left wrist and left shoulder and / or distance between right wrist and right shoulder; distance between left ankle and hip; horizontal displacement between ankles; horizontal displacement between knees; horizontal displacement between mid-shoulder and mid-hip; vertical displacement between ankles; vertical displacement between shoulders; and vertical displacement between hips.

50. The method of any one of claims 45-49, wherein the hand movement features comprise: distance between thumb and index finger of active hand that is finger tapping; distance between middle finger and wrist of active hand that is finger tapping; distance between ring finger and wrist of active hand that is finger tapping; and distance between pinky finger and wrist of active hand that is finger tapping.

51. The method of any one of claims 45-50, wherein the body movement features comprise relative power in 0.5 Hz to 1 Hz of ankle separation, standard deviation (STD) of ankle-hip distance, relative power in 2 Hz to 4 Hz of neck angle, relative power in 4 Hz to 6 Hz of arm-body angular velocity, and mean of neck angle.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 52. The method of any one of claims 45-51, wherein the hand movement features comprise half-power frequency in pinky finger movement speed, relative power in 2 Hz to 4 Hz of finger tapping speed, and median absolute deviation (MAD) of finger tapping amplitude.

53. The method of any one of claims 45-52, wherein the set of body movement features and / or hand movement features are identified by performing sparsity-inducing feature selection on the plurality of video segments.

54. The method of any one of claims 45-53, wherein the movement disorder is Parkinson’s disease, progressive supranuclear palsy, or multiple systems atrophy.

55. The method of any one of claims 45-54, wherein the machine learning algorithm uses a logistic regression (LR) classifier, a support vector machine (SVM) classifier, or a linear discriminant analysis (LDA) classifier.

56. The method of any one of claims 45-55, wherein the machine learning algorithm uses an artificial neural network.

57. The method of any one of claims 45-56, wherein the machine learning algorithm uses a deep learning algorithm.

58. The method of claim 57, wherein the deep learning algorithm uses a convolutional neural network, a deep neural network, a recurrent neural network, a deep residual neural network, a long short-term memory network, a deep belief network, a multilayer perceptron, or deep reinforcement learning.

59. The method of any one of claims 45-58, wherein the machine learning algorithm is supervised, semi-supervised, or unsupervised.

60. The method of any one of claims 45-59, wherein each video segment shows multiple strides during the walking or multiple finger-tapping cycles during the finger-tapping.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 61. The method of any one of claims 45-60, wherein each video segment is 3 to 8 seconds in length.

62. The method of any one of claims 45-61, wherein any video segment less than 3 seconds in length is discarded.

63. The method of any one of claims 45-62, wherein any video segment greater than 8 seconds in length is further segmented.

64. The method of any one of claims 45-63, further comprising correlating the symptom severity of the subject based on classification of the set of body movement features and / or hand movement features with a Movement Disorder Society-Sponsored Revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS) or a Hoehn and Yahr (HnY) scale, a Parkinson’s Disease Composite Scale (PDCS), or a Schwab and England Activities of Daily Living (ADL) Scale.

65. The method of any one of claims 45-64, further comprising assessing one or more motor and non-motor symptoms of the subject using a Movement Disorder Society-Sponsored Revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS), a Hoehn and Yahr (HnY) scale, or a Montreal Cognitive Assessment (MoCA) scale.

66. The method of any one of claims 45-65, further comprising comparing movement metrics for single joint or multi-joint movements of the subject to reference value ranges for a normal age-matched control subject.

67. The method of claim 66, wherein the movement metrics are for finger-tapping or gait speed.

68. The method of any one of claims 45-67, further comprising correlating the symptom severity of the subject based on classification of the set of body movement features and / or hand movement features with a Progressive Supranuclear Palsy Clinical Deficits Scale (PSP-CDS), a Psychogenic Movement Disorders (PMD) scale, a Unified Huntington's Disease Rating Scale-Total Motor Score (UHDRS-TMS).Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 69. The method of any one of claims 45-68, wherein each image in the first video or the second video has at least 1920 pixels horizontally and at least 1080 pixels vertically.

70. The method of any one of claims 45-69, wherein the video recording device shoots at least 30 frames per second.

71. The method of any one of claims 45-70, wherein the subject walks toward the video recording device and away from the video recording device for a minimum distance of 10 meters in each direction during said recording of the first video.

72. The method of any one of claims 45-71, wherein the first video and the second video are single-view videos shot with a single video recording device.

73. The method of any one of claims 45-72, wherein the first video and the second video are multi-view videos shot with at least two video recording devices.

74. The method of any one of claims 45-73, wherein the method is performed before and after administering a treatment to the subject for the movement disorder to evaluate efficacy of the treatment.

75. The method of any one of claims 45-74, further comprising altering the treatment of the subject for the movement disorder if the subject shows increased symptom severity based on said classification of the set of body movement features and / or hand movement features.

76. The method of any one of claims 45-75, wherein the method is performed when the subject is at home or when the subject is at a hospital.

77. The method of any one of claims 45-76, further comprising placing fiducial markers on the subject for said identifying body landmarks and / or hand landmarks.

78. The method of any one of claims 45-77, further comprising: recording a third video of the subject’s face; segmenting the third video to generate a plurality of video segments;Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 identifying face landmarks in the plurality of video segments; using the face landmarks to extract a set of face movement features from the plurality of video segments based on relative movement between the face landmarks with time; and using a machine learning algorithm to determine symptom severity of the subject based on classification of the set of face movement features in combination with the set of body movement features and / or hand movement features.

79. The method of claim 78, wherein the face landmarks comprise a nose, a right eye, a left eye, a right eyelid, a left eyelid, a right eyebrow, a left eyebrow, eye corners, a mouth, an upper lip, a lower lip, a right cheek, a left cheek, a forehead, tongue, jaw, and a chin.

80. The method of claim 78 or 79, wherein the set of face movement features is used to detect an abnormal facial movement or an involuntary facial movement associated with a movement disorder.

81. The method of claim 80, wherein the abnormal facial movement or the involuntary facial movement is rapid eye blinking, face twitching, face tremor, jaw tremor, lip tremor, lip- smacking, tongue twisting, tongue curling, tongue thrusting, lip puckering, facial drooping, or grimacing.

82. The method of any one of claims 78-81, wherein the movement disorder is Tardive dyskinesia, facial dystonia, hemifacial spasm, peripheral facial palsy, facial myokymia, facial bradykinesia, blepharospasm, Tourette syndrome, orofacial dyskinesia, brainstem tumor, peripheral neuropathy, chorea, multiple sclerosis, Guillain-Barré syndrome, Parkinson’s disease, congenital facial nerve palsy, acquired facial nerve palsy, Moebius syndrome, Poland syndrome, CHARGE syndrome, Hemifacial microsomia, 22q11.2 deletion syndrome, facial paralysis or paresis, or a psychogenic movement disorder.

83. The method of any one of claims 78-82, further comprising using the face landmarks for facial recognition, head pose estimation, facial expression detection, or facial emotion recognition.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 84. The method of claim 83, wherein facial emotion recognition is used to detect depression, psychosis, fatigue, fear, anger, sadness, or joy.

85. The method of any one of claims 45-84, wherein said recording the first video, the second video, or the third video is performed by the subject, a caregiver, or a physician.

86. The method of any one of claims 45-85, wherein the video recording device is a digital camera, a smartphone, a tablet, a laptop, or a camcorder.

87. The method of claim 86, wherein said recording the first video, the second video, or the third video is performed by the smartphone automatically whenever the subject uses the smartphone.

88. A computer implemented method for diagnosing a movement disorder in a subject, the computer performing steps comprising: receiving a first video of the subject’s entire body while the subject is walking and / or receiving a second video of a hand of the subject while the subject is finger-tapping; segmenting the first video and / or the second video to generate a plurality of video segments; identifying body landmarks and / or hand landmarks in the plurality of video segments; using the body landmarks and / or hand landmarks to extract a set of body movement features and / or hand movement features from the plurality of video segments based on relative movement between the landmarks with time; using a machine learning algorithm to detect a symptom of the movement disorder, if present, based on classification of the set of body movement features and / or hand movement features; and displaying information regarding the diagnosis of the subject based on whether the symptom of the movement disorder is detected.

89. The computer implemented method of claim 88, wherein the body landmarks comprise a nose, a right shoulder, a left shoulder, a right elbow, a left elbow, a right wrist, a left wrist, a right hip, a left hip, a right knee, a left knee, a right ankle, and a left ankle.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 90. The computer implemented method of claim 88 or 89, wherein the hand landmarks comprise a wrist, a thumb, an index finger, a middle finger, a ring finger, a pinky finger, base of the index finger, and base of the pinky finger.

91. The computer implemented method of any one of claims 88-90, wherein the body movement features comprise: neck angle; arm-body angle; distance between left wrist and left shoulder and / or distance between right wrist and right shoulder; distance between left ankle and hip; horizontal displacement between ankles; horizontal displacement between knees; horizontal displacement between mid-shoulder and mid-hip; vertical displacement between ankles; vertical displacement between shoulders; and vertical displacement between hips.

92. The computer implemented method any one of claims 88-91, wherein the hand movement features comprise: distance between thumb and index finger of active hand that is finger tapping; distance between middle finger and wrist of active hand that is finger tapping; distance between ring finger and wrist of active hand that is finger tapping; and distance between pinky finger and wrist of active hand that is finger tapping.

93. The computer implemented method of any one of claims 88-92, wherein the display further displays body movement features comprising relative power in 0.5 Hz to 1 Hz of ankle separation, standard deviation (STD) of ankle-hip distance, relative power in 2 Hz to 4 Hz of neck angle, relative power in 4 Hz to 6 Hz of arm-body angular velocity, and mean of neck angle.

94. The computer implemented method of any one of claims 88-93, wherein the display further displays hand movement features comprising half-power frequency in pinky finger movementAtty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 speed, relative power in 2 Hz to 4 Hz of finger tapping speed, and median absolute deviation (MAD) of finger tapping amplitude.

95. The computer implemented method of any one of claims 88-94, wherein the set of body movement features and / or hand movement features are identified by performing sparsity- inducing feature selection on the plurality of video segments.

96. The computer implemented method of any one of claims 88-95, wherein the movement disorder is Parkinson’s disease, progressive supranuclear palsy, or multiple systems atrophy.

97. The computer implemented method of any one of claims 88-96, wherein the machine learning algorithm uses a logistic regression (LR) classifier, a support vector machine (SVM) classifier, or a linear discriminant analysis (LDA) classifier.

98. The computer implemented method of any one of claims 88-97, wherein the machine learning algorithm uses an artificial neural network.

99. The computer implemented method of any one of claims 88-98, wherein the machine learning algorithm uses a deep learning algorithm.

100. The computer implemented method of claim 99, wherein the deep learning algorithm uses a convolutional neural network, a deep neural network, a recurrent neural network, a deep residual neural network, a long short-term memory network, a deep belief network, a multilayer perceptron, or deep reinforcement learning.

101. The computer implemented method of any one of claims 88-100, wherein the machine learning algorithm is supervised, semi-supervised, or unsupervised.

102. The computer implemented method of any one of claims 88-101, wherein each video segment shows multiple strides during the walking or multiple finger-tapping cycles during the finger- tapping.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 103. The computer implemented method of any one of claims 88-102, wherein each video segment is 3 to 8 seconds in length.

104. The computer implemented method of any one of claims 88-103, wherein any video segment less than 3 seconds in length is discarded.

105. The computer implemented method of any one of claims 88-104, wherein any video segment greater than 8 seconds in length is further segmented.

106. A non-transitory computer-readable medium comprising program instructions that, when executed by a processor in a computer, causes the processor to perform the method of any one of claims 88-105.

107. A kit comprising the non-transitory computer-readable medium of claim 106 and instructions for diagnosing a movement disorder.

108. A system comprising: a video recording device; a processor, wherein the processor is programmed to determine symptom severity of a subject having a movement disorder according to the computer implemented method of any one of claims 88-105; and a display component that displays information regarding the diagnosis of the subject.

109. The system of claim 108, wherein the video recording device is provided by a digital camera, a smartphone, a tablet, a laptop, or a camcorder.

110. The system of claim 108 or 109, further comprising a tripod, wherein the video recording device is mounted on the tripod.

111. The system of any one of claims 108-110, further comprising one or more additional recording devices, wherein each additional video recording device records videos of the subject from a different angle.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 112. The system of any one of claims 108-111, wherein the video recording device comprises a laser imaging, detection, and ranging (LIDAR) sensor.

113. The system of any one of claims 108-112, wherein the display further displays information comparing movement metrics for single joint or multi-joint movements of the subject to reference value ranges for a normal age-matched control subject.

114. The system of claim 113, wherein the movement metrics are for finger-tapping or gait speed.

115. A method of using the system of any one of claims 108-114 for diagnosing a movement disorder in a subject, the method comprising: recording a first video of the subject’s entire body while the subject is walking and / or a second video of a hand of the subject while the subject is finger-tapping using the video recording device; and detecting whether the subject has a symptom of the movement disorder according to the computer implemented method; and diagnosing the subject with the movement disorder if the symptom of the movement disorder is detected.

116. The method of claim 115, further comprising treating the subject for the movement disorder if the subject has a positive diagnosis for the movement disorder based on detection of the symptom.

117. The method of claim 115 or 116, wherein only one hand of the subject is actively performing finger-tapping and the subject’s other hand is resting.

118. The method of any one of claims 115-117, wherein the body landmarks comprise a nose, a right shoulder, a left shoulder, a right elbow, a left elbow, a right wrist, a left wrist, a right hip, a left hip, a right knee, a left knee, a right ankle, and a left ankle.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 119. The method of any one of claims 115-118, wherein the hand landmarks comprise a wrist, a thumb, an index finger, a middle finger, a ring finger, a pinky finger, base of the index finger, and base of the pinky finger.

120. The method of any one of claims 115-119, wherein the body movement features comprise: neck angle; arm-body angle; distance between left wrist and left shoulder and / or distance between right wrist and right shoulder; distance between left ankle and hip; horizontal displacement between ankles; horizontal displacement between knees; horizontal displacement between mid-shoulder and mid-hip; vertical displacement between ankles; vertical displacement between shoulders; and vertical displacement between hips.

121. The method of any one of claims 115-120, wherein the hand movement features comprise: distance between thumb and index finger of active hand that is finger tapping; distance between middle finger and wrist of active hand that is finger tapping; distance between ring finger and wrist of active hand that is finger tapping; and distance between pinky finger and wrist of active hand that is finger tapping.

122. The method of any one of claims 115-121, wherein the body movement features comprise relative power in 0.5 Hz to 1 Hz of ankle separation, standard deviation (STD) of ankle-hip distance, relative power in 2 Hz to 4 Hz of neck angle, relative power in 4 Hz to 6 Hz of arm-body angular velocity, and mean of neck angle.

123. The method of any one of claims 115-122, wherein the hand movement features comprise half-power frequency in pinky finger movement speed, relative power in 2 Hz to 4 Hz of finger tapping speed, and median absolute deviation (MAD) of finger tapping amplitude.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 124. The method of any one of claims 115-123, wherein the set of body movement features and / or hand movement features are identified by performing sparsity-inducing feature selection on the plurality of video segments.

125. The method of any one of claims 115-124, wherein the movement disorder is Parkinson’s disease, progressive supranuclear palsy, or multiple systems atrophy.

126. The method of any one of claims 115-125, wherein the machine learning algorithm uses a logistic regression (LR) classifier, a support vector machine (SVM) classifier, or a linear discriminant analysis (LDA) classifier.

127. The method of any one of claims 115-126, wherein the machine learning algorithm uses an artificial neural network.

128. The method of any one of claims 115-127, wherein the machine learning algorithm uses a deep learning algorithm.

129. The method of claim 128, wherein the deep learning algorithm uses a convolutional neural network, a deep neural network, a recurrent neural network, a deep residual neural network, a long short-term memory network, a deep belief network, a multilayer perceptron, or deep reinforcement learning.

130. The method of any one of claims 115-129, wherein the machine learning algorithm is supervised, semi-supervised, or unsupervised.

131. The method of any one of claims 115-130, wherein each video segment shows multiple strides during the walking or multiple finger-tapping cycles during the finger-tapping.

132. The method of any one of claims 115-131, wherein each video segment is 3 to 8 seconds in length.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 133. The method of any one of claims 115-132, wherein any video segment less than 3 seconds in length is discarded.

134. The method of any one of claims 115-133, wherein any video segment greater than 8 seconds in length is further segmented.

135. The method of any one of claims 115-134, further comprising correlating the symptom severity of the subject based on classification of the set of body movement features and / or hand movement features with a Movement Disorder Society-Sponsored Revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS) or a Hoehn and Yahr (HnY) scale, a Parkinson’s Disease Composite Scale (PDCS), or a Schwab and England Activities of Daily Living (ADL) Scale.

136. The method of any one of claims 115-135, further comprising assessing one or more motor and non-motor symptoms of the subject using a Movement Disorder Society-Sponsored Revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS), a Hoehn and Yahr (HnY) scale, or a Montreal Cognitive Assessment (MoCA) scale.

137. The method of any one of claims 115-136, further comprising correlating the symptom severity of the subject based on classification of the set of body movement features and / or hand movement features with a Progressive Supranuclear Palsy Clinical Deficits Scale (PSP-CDS), a Psychogenic Movement Disorders (PMD) scale, a Unified Huntington's Disease Rating Scale-Total Motor Score (UHDRS-TMS) 138. The method of any one of claims 115-137, wherein each image in the first video or the second video has at least 1920 pixels horizontally and at least 1080 pixels vertically.

139. The method of any one of claims 115-138, wherein the video recording device shoots at least 30 frames per second.

140. The method of any one of claims 115-139, wherein the subject walks toward the video recording device and away from the video recording device for a minimum distance of 10 meters in each direction during said recording of the first video.Atty. Dckt.: UCSF-735WO Client Ref.: SF-2023-181-2-PCT-0 141. The method of any one of claims 115-140, wherein the first video and the second video are single-view videos shot with a single video recording device.

142. The method of any one of claims 115-141, wherein the first video and the second video are multi-view videos shot with at least two video recording devices.

143. The method of any one of claims 115-142, wherein the method is performed when the subject is at home or when the subject is at a hospital.

144. The method of any one of claims 115-143, further comprising placing fiducial markers on the subject for said identifying body landmarks and / or hand landmarks.

Images