Image-based motor function assessment

Abstract

Methods and systems are provided to assess physical movements of a patient based on a video or consecutive images taken of the patient performing a particular physical activity. The assessment can be used for determining a medical disorder or a stage of a medical disorder that the patient suffers.

Description

IMAGE-BASED MOTOR FUNCTION ASSESSMENTBACKGROUND

[0001] Assessing a patient’s motor functions is a critical aspect of diagnosing and monitoring various neurological and musculoskeletal disorders. The severity of such disorders are generally assessed in clinic at intermittent times, requiring the patient to commute to a hospital / healthcare facility each time. Generally, gait and physical activity are assessed by asking the patient to perform repetitive physical activities while a physician evaluates the patient’s performance. The patient’s motor functions depend on the severity of the disorder. For example, at an early stage of a particular disorder, a patient may still be able to walk without support, but experience instability. At a more advanced stage of the disorder the same patient may lose the ability to walk, but be able to crawl or roll until possibly losing these functions at an even more advanced stage of the disorder. Similarly, a patient may gradually lose their ability to talk, eat, or use other facial muscles as a disorder advances. An early and correct diagnosis of a patient’s disorder or the disorder’s stage is critical in starting treatments that are appropriate for the disorder and for that particular stage. The more the treatment is delayed, the faster the disorder advances, which can lead to limiting or disabling the patient in performing their daily routines.SUMMARY

[0002] Implementations of the present disclosure include methods and systems for detecting and / or monitoring biomarkers (i.e., biological characteristics) of a patient through analyzing videos (or consecutive images) of the patient performing a particular physical activity. The patient may suffer from a neurological or musculoskeletal disorder such as Multiple Sclerosis (MS), Pompe, Metachromatic leukodystrophy (MLD), Parkinson’s Disease, etc. The implementations detect or monitor the progress of the disorder by analyzing the biomarkers of the patient performing the particular activity (or a set of particular tasks). The analysis can be performed by a machine learning algorithm that is trained based on biomarkers data (e.g., muscle joint, and / or bone movements) in patients suffering from a previously diagnosed disorder(s) when performed the same task.

[0003] Some implementations include a computer-implemented method executable by a computing system. The method includes: receiving a plurality of consecutive images taken froma patient while performing a physical activity; transforming the images to respective anonymized representations of a patient body of the patient, each anonymized representation including a plurality of nodes that each represents a particular body portion of the patient body, each node being identifiable by a respective spatial coordinate assigned to the node; identifying from the plurality of nodes a set of target nodes representing one or more targeted body parts of the patient; determining a set of coordinate sequences that represent movements of the one or more targeted body parts during the physical activity, each coordinate sequence in the set including spatial coordinates of a respective target node associated with a target body portion as depicted in the consecutive images; analyzing the set of coordinate sequences to determine a stage of a medical disorder that the patient suffers, the medical disorder being a neurological or musculoskeletal disorder; and sending the determined stage to a user interface for presentation.

[0004] In some implementations, the physical activity is at least one of movement of a head of the patient, talking, chewing, or swallowing, and the one or more targeted body parts includes at least one of the head, mouth, lips, or neck of the patient.

[0005] In some implementations, the physical activity includes walking. The one or more targeted body parts can include at least portions of legs of the patient. The set of target nodes can include a first leg node representing a portion of a first leg, and a second leg node representing a portion of a second leg of the patient. Analyzing the set of coordinate sequences can include: comparing a first coordinate sequence of the first leg node to a second coordinate sequence of the second leg node to obtain a signal representing gait cycles during the walking, each gait cycle starting from a respective starting point when the first leg passes the second leg along a walking direction; determining, based on the gait cycles, step parameters for the patient, the step parameters including one or more of step duration, step count, step frequency, or cadence; and determining the stage of the medical disorder based on the step parameters.

[0006] The step parameters can include the step frequency or cadence. The stage of the medical disorder can be determined based on a predetermined association between different stages of the disorder and different step frequencies or cadences.

[0007] In some implementations, each gait cycle ends at a respective point when the second leg passes the first leg along the walking direction.

[0008] In some implementations, a gait cycle can start from a point when the first leg passes the second leg along the walking direction, and end the next time the first leg passes the second leg.

[0009] The method can further include determining a frontal plane of the patient body, wherein the first leg passes the second leg when a first coordinate of the first leg point changes from a negative value to zero or a positive value in a direction perpendicular to the frontal plane. The method can further include determining that the physical activity is walking by determining that the first coordinate changes between a negative value and a positive value alternatively and periodically.

[0010] In some implementations, the first leg node represents at least part of the first leg between a hip and an ankle of the first leg. The first leg node can represent a knee, an ankle, a shin, or a thigh of the first leg.

[0011] In some implementations, the physical activity includes walking, and the one or more target body parts includes at least a portion of an arm of the patient. The set of target nodes can include an elbow node representing an elbow of the arm, an upper-arm node representing a body portion between the elbow and a shoulder of the arm, and a lower-arm node representing a finger of the arm or a section between the finger and the elbow. Analyzing the set of coordinate sequences can include determining a sequence of arm angles each formed between the upper-arm node and the lower-arm node with respect to the elbow node at a respective image of the walk. The stage of the medical disorder can be determined based on an analysis of the sequence of arm angles.

[0012] The analysis of the sequence of arm angles can include determining a variance in the arm angle based on a change in the angles in the sequence of arm angles. The stage of the medical disorder can be determined by comparing the variance to a predetermined association between arm angle variances and different stages of the medical disorder.

[0013] The analysis of the sequence of arm angles can include: determining a signal representing gait cycles by determining repetitions in changes in the arm angles in consecutive periods of time during the walking; and determining, based on the gait cycles, step parameters for the patient, the step parameters including at least one of step duration, step count, or step frequency. The stage of the medical disorder can be determined based on the step parameters.

[0014] The analysis of the sequence of arm angles can include determining a minimum arm angle in the sequence of arm angles. The stage of the medical disorder can be determined based on the minimum arm angle.

[0015] The stage of the medical disorder can be determined based on a progress or a regress of the disorder over a period of time. The progress or the regress being determined by: obtaining, from a medical history of the patient, an old set of coordinate sequences that was determined for a prior physical activity that the patient performed at a past time before performing the physical activity, the past time indicating a beginning of the period of time; and comparing the set of coordinate sequences to the old set of coordinate sequences to determine an improvement or a deterioration of a motor function of the patient, wherein the improvement is associated with the regress of the disorder, and the deterioration is associated with the progress of the disorder.

[0016] The method can include determining a treatment efficacy of a treatment that the patient undergoes based on the determined stage and a previously determined stage of the disorder, and based on a time passed from the previously determined stage.

[0017] The method can include: determining a hip node representing a hip of the patient, and setting a spatial coordinate of the hip node as (0,0,0). The spatial coordinates of other nodes are determined relative to the hip node.

[0018] The anonymized representation can be a 3-dimensional representation of the patient body.

[0019] The computing system can be a mobile device or a smart phone. The mobile device or the smart phone includes a camera used to capture the plurality of consecutive images.

[0020] The method can include causing the user interface to present instructions for performing the physical activity. The method can further include: comparing the movements of the one or more targeted body parts to the instructions; and in response to determining that the movements do not comply the instructions, sending further instructions to the user interface to guide the patient redo the physical activity, wherein the further instructions include suggesting particular actions to correct the movements.

[0021] The method can include causing the user interface to present real-time feedback about the patient’s performance of the physical activity as the patient performs the physical activity.

[0022] The present disclosure further provides a system for implementing the methods provided herein. The system can include one or more computers, and one or more computer- readable storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations to perform operations in accordance with implementations of the methods provided herein.

[0023] The present disclosure also provides one or more non-transitory computer-readable storage medium coupled to one or more processors and having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations in accordance with implementations of the methods provided herein.

[0024] Among other advantages, implementations can provide one or more of the following benefits. First, some implementations provide an objective evaluation of a patient’s biomarkers, and as a result, can deliver a more reliable determination of the disorder, the stage of the disorder, or a prediction of the progress of the disorder over a period of time. Conventionally, a clinician reviews the patient’s biomarkers to identify potential disorders and / or the stages of the disorders. For example, a clinician observes the gait posture of a patient in an in-clinic visit to diagnose MLD disorder or to determine the severity of the disorder. Such diagnosis and determination is highly subjective. As a result, one clinician may identify the disorder on a particular patient as severe and prescribe a high dosage of a medicament, while another clinician may identify the same disorder on the same patient as mild and prescribe a low dosage of the medicament or other treatment methods. Implementations of the present disclosure reduce subjectivity in the monitoring and diagnosis by using a system that objectively monitors the patient’s biomarkers extracted from videos (or images) of the patient through signal processing and machine learning methods. The implementations can also provide an objective analysis of past or future progress of the disorder by objectively monitoring and evaluating biomarkers of the patient over a period of time.

[0025] Second, certain implementations provide a more convenient and accessible solution for the patient as compared to the conventional clinical evaluations, where the patient would have to make advanced appointments and travel to a clinic. This is because patients using the technology described in this disclosure can use their personal device to get an evaluation of their condition anywhere (e.g., at home) and at any time. The accessibility of the patient’s personal device to provide the evaluations at any time that the patient desires can also lead to obtainingmore data points about the patient’s biomarkers. More data points can result in a more reliable diagnosis or monitoring of the disorder. The data points can possibly be obtained at smaller intervals compared to consecutive clinical visits, which can lead to a quick determination of existence of an alarming condition that may need an immediate attention.

[0026] Third, the system may be capable of suggesting particular treatments, or sending the results of its diagnosis to a doctor to prescribe treatments based on the system’s diagnosis. Because of the improved reliability, treatments suggested for the patient as a result of the diagnosis can be formed specific to the patient with a higher level of accuracy as compared to generalized treatment plans suggested for patients suffering from a particular disorder or a particular stage of a disorder. Further, a patient can enter other biomarkers (e.g., age, gender, race, chronic disease, etc.) that the system can consider when proposing a particular treatment, or when transferring the diagnosis data to a doctor. Accordingly, the treatments can be personalized based on the particularities of the patient’s body or life style.

[0027] Fourth, particular techniques presented in this disclosure can reduce the amount of data received from the patient’s video (or consecutive images). Such data reduction improves processing speed as compared to analyzing the raw videos of the patient. More specifically, some implementations transform the body image of a patient depicted in a video, to an animated body representing only particular (or target) parts of the patient’s body rather than the whole body. Those particular body parts are represented by a limited number of data points, where each data point is assigned to a respective spatial coordinates. Accordingly, the implementations review only a limited amount of data associated with particular body parts of interest (that correspond to the target biomarkers); and even those particular body parts are represented by only a few data points. Such a procedure reduces the amount of data analysis needed to study the biomarkers (e.g., movements of particular muscles), increases the processing speed, and even reduces the network traffic if the data needs to be transmitted to an external device at any time during or after the data analysis process.

[0028] Fifth, in some implementations, the system can provide results in real-time. Because of a reduction in the amount of data and an increased efficiency in data processing, the present implementations are capable of reducing the data analysis time, and even providing results in real-time. A patient can simply use their personal device to take a video of themselves, and receives the result (e.g., an identified disorder or disorder stage) in a short period of time after thevideo has been recorded (e.g., within seconds or minutes). This is much more convenient for patients, and reduces stress from waiting to receive the results as compared to other methods that do not benefit from the proposed data reduction. The immediate feedback that the implementations provide encourages the patients to use the device more frequently, which can lead to obtaining more data points, and thus, more reliable results, as explained above.

[0029] Sixth, the hassle of attaching sensors to patients may be eliminated in some implementations. Since the implementations presented in this disclosure use videos or images of the patient to evaluate the biomarkers, the patient does not have to wear any sensors. Wearing sensors can be (mentally or physically) annoying, limit movements of the patient, and be prone to errors if a sensor is placed in a wrong part of the body. The present implementations eliminate such drawbacks, and rather, give the advantage of letting the patients move as freely as they would in their daily lives without being concerned about the placement of the sensors.

[0030] Seventh, implementations described herein provide scalability. While the implementations provide personalized diagnosis (and event personalized treatments), there are little to no limits on the number of patients they can serve. The artificial intelligence models used in the implementations enables the assessment of large patient cohorts efficiently, thereby improving clinical workflow and reducing assessment time.

[0031] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other features and advantages of the present disclosure will be apparent from the description and drawings, and from the claims.DESCRIPTION OF DRAWINGS

[0032] FIG. 1 depicts an example environment that can be used to execute implementations of the present disclosure.

[0033] FIG. 2 depicts example components of a system capable of analyzing raw images or videos of a patient to determine a disorder or its stage.

[0034] FIG. 3 depicts an example “gait cycle” as defined in the present disclosure.

[0035] FIG. 4 depicts an example process that can be executed in accordance with implementations of the present disclosure.

[0036] FIG. 5 shows a schematic diagram of an example computing system that can perform the methods described in the present disclosure.

[0037] FIGs. 6A and 6B show an example image of a patient while walking, and two anime of the patient’s body with different numbers of head nodes for the example image.

[0038] FIGs. 7A-7C show example images of a patient standing up and sitting down.

[0039] FIGs. 8A-8D show example images of a patient’s head movements.

[0040] FIGs. 9 A and 9B show example images of a patient tapping fingers.

[0041] Like-labeled components in the figures refer to the same elements or steps.DETAILED DESCRIPTION

[0042] Implementations of the present disclosure provide techniques for assessing physical movements (i.e., motor skills) of a patient based on a video of the patient performing a particular physical activity. The techniques are performed by a computing system. The system transforms the images in the video into an anonymized representation of the patient’s body. The anonymized representation can be a graphical representation including nodes that represent particular body parts. Each node is connected to one or more adjacent nodes by respective lines.

[0043] To reduce the amount of data, the system selects anonymized representations of only particular body parts of interest. Alternatively, the system can anonymize only the particular body parts, and skip the rest of the body’s image, or represent the rest with only a few (e.g., one) node.

[0044] The selection or creation of anime for the particular body parts depends on the physical activity that the patient is asked to perform. For example, in FIG. 6A, where the patient’s walking abilities is of interest, no node is assigned to facial muscles such as lips, eyes, nose, etc. If the ability to eat is of interest, those muscles would be represented with corresponding nodes, and leg muscles, for example, would be skipped in the representation.

[0045] The particular physical activity may differ based on what muscles or joints are being studied. For example, if the patient’s ability to roam is of interest, the particular activity can be walking, or more specifically, walking in a particular pattern, for a specific number of steps or distance. The pattern can include a straight line, zigzag, hops, turns, etc.

[0046] When the patient’s ability to roam is being studied, the system aims for determining gait biomarkers for the patient. As example gait biomarkers, the system can look for the number of steps, speed or frequency of taking steps, etc. to determine the patient’s motor skills, e.g.,walking abilities. To do so, the system determines a gait cycle for the patient, where each complete step would represent one gait cycle.[00471 To determine the gait cycle and extract gait biomarkers, the system can study movements of the legs and / or arms of the patient. For the legs, the system makes a correlation between the gait cycle and when one of the patient’s legs passes the other of the legs, and / or when the other of the legs passes the first leg. For the arms, the system makes a correlation between the gait cycle and an angle (or a change of the angle) at the patient’s elbows while the patient walks. The system can study or use one or both of the legs and arms movements to determine the gait biomarkers.

[0048] As another example, the system may study patient’s ability to eat, talk, or perform facial impressions by studying a physical activity corresponding to movement of the patient’s facial muscles. Depending on the muscles of interest, the system may request the patient to perform a particular physical activity, for example, by presenting the details of that physical activity on a screen (e.g., on a user interface) of the system. The system then takes a video of the patient while performing that physical activity, and analyzes that video to determine a disorder, or an stage of a disorder that the patient suffers.

[0049] FIG. 1 depicts an example environment 100 that can be used to execute implementations of the present disclosure. As presented, computing device 104 takes a video or consecutive images / video 106 (referred to as “video” herein) of patient 102 while the patient is performing a particular physical activity. As discussed further below, device 104 can perform a partial or a complete analysis of the received video, where the analysis leads to an objective diagnosis of patient 102. The device 104 can optionally send the video or a formatted version of the video 108b to a physician’s device 122 for a subjective evaluation.

[0050] While the present disclosure refers to the input data (106) as “video,” the input can be a set of consecutive images taken of the patient (e.g., every second) while performing the physical activity. In some implementation, device 104 fragments the input video into a set of images. For example, the video can be fragmented into seconds (or a few seconds), and one image may be taken from each fragment. Using the images instead of the video leads to a reduction in the amount of data needed to be processed, and thus, increases the processing speed. In some implementations, the anonymized data for each node (representing a respective body part) is created and / or stored as a sequence of a specific number of coordinates over time, forexample, a sequence of thirty to sixty coordinates per second in a study period (which is a period of time during which the patient performs the physical activity).[00511 Device 104 transforms images of the patient’s body (depicted in video 106) to respective anonymized representations (anime 108a) of the patient’s body. Device 104 can further analyze the anime 108a to determine the patient’s disorder or disorder stage, or can send the anime 108a (or at least a portion of it) to an external backend system 112, for further analysis.

[0052] FIG. 2 depicts example components of a system 200 capable of analyzing raw images or videos of a patient to diagnose a disorder or determine a stage of the disorder of the patient. In some implementations, each of the components shown in FIG. 2 is part of a single computing device such as device 104 shown in FIG. 1. In some implementations, the system includes multiple computing devices or sub-systems that are in communication with each other to perform the tasks described below. For example, device 104 can include receiver module 212 and transformation module 214, and can send the anonymized representation of the patient’s body (e.g., anime 108a or portion(s) of it 224a / 224b) to another computing system, e.g., backend system 112, to determine the disorder or its stage.

[0053] System 200 includes receiver module 212 that receives the video or images of the patient (e.g., 102). The images or video are captured by a camera, e.g., camera 202. Camera 202 can be part of the same system 200, or external to the system and in communication with the receiver module 212. For example, both camera 202 and receiver module 212 can be parts of the patient’s personal device, e.g., device 104 shown in FIG. 1.

[0054] Receiver module 212 can store or format the video / images in preparation for the video / images being analyzed by the rest of the system. For example, the receiver module can clean up the video to remove noises or to crop and save only the parts that depict the patient performing the particular activity. Receiver module 212 sends the received, stored, and / or formatted video / images to transformation module 214.

[0055] Transformation module 214 detects the patient’s body in the video / images 106, and anonymizes the body into anime 108a. In some implementations, transformation module 214 can extract the image of the patient’s body from the video / images by removing the background or any items depicted in each image (of the video). Thus, the amine includes only a representation of the patient’s body moving in an empty space. By doing so, the transformationmodule reduces the amount of data needed to process in each image (of the fragments of the video).[00561 Transformation module 214 creates the anime in form of a graph including nodes that are connected to their adjacent node(s) through edges. Each node represents a particular part or limb of the patient’s body. Depending on how much detail may be needed, the nodes can be closer or further apart from each other. For example, for an analysis of the facial muscles, the lower lip can be assigned to multiple nodes that are close to each other. But for analysis of legs during a walk, each legs can be assigned to respective nodes that are distanced from each other much further than the distance used for the lower lip nodes in the facial muscles analysis. While the higher number of nodes provide more details about granular movement of the patient’s specific muscles, the lower the number of nodes used, the less amount of data needs to be analyzed for disorder / stage determination. For walk, for example, as few as one node per leg can be assigned to each leg to determine the gait cycle introduced in this disclosure.

[0057] Transformation module 214 assigns a respective coordinate to each node in the anime. In some implementations, transformation module 214 sets a point on a particular body portion (e.g., a point on hip) to an origin (i.e., (0,0,0) coordinate), and assigns respective coordinates to the other nodes of the anime 108a based on their respective positions with respect to the origin. During the patient’s movement, the origin’s coordinate stays the same (0,0,0) while the other points can change their coordinates as they move with respect to the origin. By having the origin fixed to a particular part of the patient’s body, transformation module 214 creates the anime (108a) independent of the patient’s background or surroundings that are depicted in the images / video 106.

[0058] Transformation module 214 sends anime 108a to target node identifier 122. Target node identifier 122, coordinate sequence identifier 226, and disorder determiner 228 can be parts of a machine learning module 240, and together perform a machine learning model on the anime(s) 108a received from transformation module 214. The machine learning model evaluates the anime(s) to determine a disorder, or a stage of a disorder from which the patient suffers.

[0059] Target node identifier 122 identifies from the anime 108a part(s) of anime 108a (e.g., 224a, 224b) that are of interest for determining the disorder or its stage. For example, if the particular physical activity is roaming (e.g., walk), body parts such as legs and / or arms thatperiodically move during roaming are be the target body parts. Accordingly, transformation module 214 identifies nodes associated to the patient’s legs (e.g., anime segment 224a) and / or nodes associated to the patient’s arms (e.g., anime segment 224b) from the overall body anime 108a. By removing the rest of the anime 108a, target nodes identifier 222 helps in further reduction of the amount of data to be analyzed, and thus, improves the processing speed.

[0060] Target nodes identifier 222 can create a respective anime segment for each body part, or can include multiple body segments into the same anime segment. For example, while different anime segments 224a, 224b are shown in FIG. 2 for arms and legs, anime of both arms and legs can be included in a single anime segment (not shown). The separation of the anime segments for each identified body part allows a separate analysis of each body part. However, a similar separation can be performed later, e.g., by coordinate sequence identifier 226 or disorder determiner 228, to separately study coordinates of the nodes that are associated with different body parts (e.g., legs versus arms).

[0061] Target node identifier 122 can identify the target body portion based on a default instruction previously provided to the identifier, or provided to a storage device from which the identifier retrieves instructions to operate. For example, for a patient suspected of a particular disorder, disorder determiner 228 can send the target body portions whose muscle strength and ability to move are indicative of different stages of the particular disorder. As another example, the storage device may have pre-stored correlations between the degree of freedom to move in different body parts and respective disorders, and can provide all or part of that information to the target node identifier 122. The information may be sent to the target node identifier based on the disorder from which the patient suffers, or is suspected to suffer. The target node identifier 122 uses that information to select target nodes (or anime segments) associated with the target body portions included in the information. Alternatively, or in addition, target node identifier 122 can communicate with disorder determiner 228 to receive information about what body portions or muscles are being used to analyze the patient’s motor functions.

[0062] Target node identifier 122 sends the identified target anime segments (e.g., 224a, 224b) to coordinate sequence identifier 226. Coordinate sequence identifier 226 determines a set of coordinate sequences that represent movements of the one or more targeted body parts during the physical activity. Each coordinate sequence in the set includes coordinates of a respective target node in the images (of the video) received from the receiver module 212. For example, asequence associated with a particular node can be presented as a respective N x 3 matrix, where each row in the matrix shows the coordinates of the particular node in a respective image, where N consecutive images (of the video) are being studied. Target node identifier 122 sends the identified (or determined) sequences to disorder determiner 228.

[0063] Disorder determiner 228 analyzes the set of coordinate sequences to determine a disorder (or its stage) from which the patient suffers. A disorder determiner module can include multiple sub-modules that each can analyze movements associated with a respective physical activity. For example, disorder determiner 228 shown in FIG. 2 has sub-modules roaming analyzer 230, and facial movement analyzer 238. Roaming analyzer 230 analyzes images (or videos) associated with roaming, e.g., walking, running, hopping, and facial movement analyzer 238 analyzes images associated with movements of facial muscles, e.g., talking, eating, or expressing emotions such as laugh, frown, winkle, etc.

[0064] Based on the target body parts, the respective sub-module(s) associated with those body parts analyze the coordinate sequences associated those body parts from among all sequences received from coordinate sequence identifier 226. If multiple body parts are being studied, one or multiple sub-modules associated with those body parts can operate, e.g., in parallel. For example, if both patient’s ability to roam (e.g., to walk) and ability to talk are being studied, each of sub-modules roaming analyzer 230 and facial movement analyzer 238 analyzes the coordinate sequences associated with the target body part(s) that the sub-module evaluates; e g., legs and / or arm for roaming analyzer 230, and jaws and / or lips for facial movement analyzer 238. As noted above, the target body parts are identified based on the disorder that is being checked.

[0065] A roaming analyzer can include further sub-modules that each reviews particular subset of coordinate sequences associated with a particular target body portion(s). For example, roaming analyzer 230 includes hip position detector 232, leg position detector 234, and arm position detector 236. Hip position detector 232 studies coordinate sequences associated with patient’s hip. Leg position detector 234 analyzes coordinate sequences associated with patient’s leg anime segment 224a. Arm position detector 236 studies coordinate sequences associated with patient’s arm(s) segment 224b. Roaming analyzer 230 can use all, or only a few of these sub-modules to determine gait cycles for the patient. For example, roaming analyzer 230 may determine the gait cycle only based on the patient’s legs movements (i.e., use only leg positiondetector 234), or based on a coordination between the patient’s legs movements and arms movements (i.e., using both leg position detector 234 and arm position detector 236 in coordination).

[0066] To determine the gait cycle, leg position detector 234 identifies at least a first leg node assigned to a part of a first leg of the patient, and at least a second leg node assigned to a part of a second leg of the patient. Each of the first and the second leg nodes can be assigned to any part of the respective leg from patient’s hip to the toes of that leg. For example, a first leg node can be assigned to any or some of the patient’s left thigh, knee (e.g., node 244), calf, shin, ankle (e.g., node 246), heel, toe, etc., and a second leg node can be assigned to any or some of the patient’s right thigh, knee (e.g., node 248), calf, shin, ankle, heel, toes, etc. In some embodiments, the first and the second leg nodes are assigned to similar leg parts of the respective legs; for example, both are assigned to the respective knees, or both are assigned to the respective angles. But assigning to similar leg portions is no a requirement.

[0067] In some implementations, a gait cycle is step-specific, and is defined as the period of time between when the first leg passes the second leg along the roaming direction (e.g., the direction of the walk), and ending at an ending point when the second leg passes the first leg along the roaming direction. FIG. 3 shows an example step-specific gait cycle 306. As can be seen, the cycle starts when a first leg passes the other leg (at point 302), and ends when the other leg passes the first leg (at point 304).

[0068] In some implementations, the gait cycle is defined as the time between two consecutive passing of the legs. For example, FIG. 3 shows gait cycle 308 as used in such implementations. Compared to step-specific gait cycle 306, the bi-step gait cycle 308 accounts for the asymmetrical movement of the legs, and is more likely to provide a repetitive step pattern. For example, if a patient drags one foot more than the other, or is slower in moving one foot as compared to the other foot, gait cycle 308 includes two consecutive steps taken by both feet in one cycle while gait cycle 306 provides a respective cycle for each of the steps / feet.

[0069] Both step-specific and bi-step gait cycle account for a wider range of feet / legs positions during roaming compared a conventionally defined gait cycle, which is defined as the time period between consecutive times that the heels touch the ground. For example, if a patient drags one or both feet on the ground (which is not uncommon in patients suffering from neurological or musculoskeletal disorders), the heel(s) would be on the ground constantly, or fora significant (e g., a major) part of a step. But the position of the heels with respect to the ground would not limit or introduce error to identifying the gait cycles in the particular implementations noted discussed here.

[0070] Going back to FIG. 2, leg position detector 234 determines the gait cycle(s) by comparing the first coordinate sequence of the first leg node to the second coordinate sequence of the second leg node to determine when one leg passes the other in the patient’s roaming. Leg position detector 234 can determine a first step gait cycle by assigning a first point in time as the starting point of the cycle, and by assigning a second point in time as the ending point of the cycle. For a cycle similar to the step-specific gait cycle 306 shown in FIG. 3, the first point in time is when the first leg passes the second leg along a direction of the walk, and the second point in time is when the second leg passes the first leg along the direction of the walk. Leg position detector 234 can similarly create a second step gait cycle for the time period between when the second leg passes the first leg and consecutively, the first leg passes the second.

[0071] Alternatively, leg position detector 234 can determine and use a bi-step gait cycle, similar to gait cycle 308 in FIG. 3. In such scenarios, a gait cycle would be the time period between (i) when the first leg passes the second leg, and (ii) the next time that the patient’s first leg passes the second leg.

[0072] Leg position detector 234 determines the passing points (i.e., points in time when one of the leg passes the other leg) by comparing the first coordinate sequence(s) associate with one (or more) first leg node(s) that represent the first leg, to second coordinate(s) associated with (respective) one (or more) second leg node(s) that represent the second leg. As discussed earlier, each coordinate sequence is associated with a respective node, and each coordinate in a particular node’s coordinate sequence represents a respective spatial position of the particular node obtained from a respective image (of the video) taken from the patient.

[0073] Leg position detector 234 determines as a passing point the time when a first coordinate (associated with a first leg node) changes from a negative value (or coordinate) to a positive value (or coordinate) in the roaming direction and relative to a second coordinate (associated with a second node). The detector 234 identifies the change from the respective coordinate sequences of the first and the second nodes as presented in multiple consecutive images (e.g., of the video). For example, assuming that the roaming direction is assigned to direction X in an XYZ space, when the first coordinate changing from (2,5,0) to (6,5,0) in twoconsecutive images, it changes from a negative to a positive value (or coordinate) in the roaming direction X and with respect to a second coordinate that is fixed at (4,2,0) in the two consecutive images.

[0074] In some implementations, a relative change from a non-positive to a positive value (or coordinate) in the roaming direction is considered as the time when one leg passes the other. For example, when a first coordinate changes from (2,5,0) to (4,5,0) in two consecutive images, the first coordinate changes from a non-positive to a positive value (or coordinate) in the roaming direction X and with respect to a second coordinate that is fixed at (2,2,0) in the two consecutive images. These implementations account for gait cycles associated with step patterns where one of the legs is not positions behind another of the legs, e.g., due the patient’s difficulty in moving the latter leg.

[0075] In some implementations, leg position detector 234 determines gait cycles based on a frontal plane of the patient’s body. Leg position detector 234 can determine a frontal plane for the body of the patient, and determines each leg’s passing point relative to the other leg based on the legs’ positions relative to the frontal plane. Leg position detector 234 can identify the frontal plane as a plane that is perpendicular to the ground and divides the patient body to a posterior portion and an anterior portion. The Posterior portion includes the back of the patient, and the posterior portion includes the chest of the patient. In these implementations, leg position detector 234 determines as a passing point the time when the first leg passes the second leg or when the second leg passes the first leg in a direction perpendicular to the frontal plane; meaning when a first coordinate of a first leg node changes from a negative value to zero or a positive value in the direction perpendicular to the frontal plane, or when a second coordinate of a second leg node changes from a negative value or zero to a positive value in the direction perpendicular to the frontal plane. In some implementations, the leg position detector 234 determines an offset of the right leg to the left leg, and defines the frontal plane such that the offset is at the origin or on a line that passes the origin of the plane.

[0076] Once the gait cycle is determined, roaming analyzer 230 divides the images to sets of gait-specific images, and analyzes the patient’s step parameters based on the sets. Step parameters can be step-specific parameters, which would be extracted from the images in individual sets, or can be general step parameters, which can be determined from studying multiple (e.g., all) of the sets. Step-specific parameters can be step-specific. Examples of step-specific parameters include the movement of one leg relative to the other, step width for each leg, mobility of each leg, stability of each leg, step duration, etc. General step parameters can be associated with the overall roaming of the patient. Examples of general step parameters include step count, step frequency, roaming speed, etc.

[0077] As discussed earlier, the physical activity used for the disorder (stage) determination can be pre-defined based on the disorder, and disorder determiner 228 can use the submodule (e.g., 230) associated with that activity to analyze the patient’s motor skills. However, the disorder may not be pre-defined. In such implementations, disorder determiner 228 may need to analyze the nodes to determine what activity the patient is performing before focusing on the details of that activity.

[0078] In some implementations, disorder determiner 228 uses the positions of the leg nodes relative to the frontal plane to determine that roaming is the activity of interest. For example, disorder determiner 228 can determining that the particular physical activity is “walk” by determining that a first coordinate associated with a first leg node (of a first leg) and a second coordinate associated with a second leg node (of a second leg) change alternatively and periodically with respect to and perpendicular to the frontal plane.

[0079] In addition or alternative to using leg nodes, roaming determiner 228 can use arm nodes associated with one or both arms of the patient to determine the gait cycle and / or to determine the disorder or its stage. Roaming determiner 228 uses arm position detector 236 to make such determination.

[0080] Arm position detector 236 receives coordinate sequences associated with arm nodes from coordinate identifier 226. Coordinate sequence identifier 226 receives anime segments (e.g., 224b) representing spatial positions of one or both arms of the patient represented by respective arm nodes included in the anime segments. Coordinate sequence identifier 226 generates for those arm nodes respective coordinate sequence sets representing the nodes in consecutive images.

[0081] An anime segment of an arm can have any number of arm nodes. In a preferred embodiment, the anime segment would include at least three arm nodes: (i) an elbow node representing the elbow of the arm, (ii) an upper-arm node representing a body portion higher than the elbow, e.g., a shoulder or a section between the elbow and the shoulder on the arm, and(iii) a lower-arm node representing a finger of the arm or a section between the finger and the elbow.[00821 Arm position detector 236 analyzes the set of coordinate sequences it has received from coordinate sequence identifier 226 to determine a sequence of arm angles (for the respective arm) formed during the roaming activity. Each arm angle is formed between an upper-arm node and a lower-arm node with respect to the elbow node at a respective time during the activity. Roaming determiner 228 (or arm position detector 236) determines the disorder or its stage based on the analysis of the sequence of the arm angles.

[0083] The roaming determiner 228 (or arm position detector 236) can determine the disorder or a stage of the disorder based on the minimum arm angle happened during the roaming. There is a correlation between difficulty to roam (e.g., to walk) and the arm angles. A patient with difficulties to have a stable walk would generally tighten their arm muscles in positions with lower angles than a person that has no difficulties in walking stably.

[0084] FIGs. 6A and 6B show an example image (i.e., snap shot) 602 of a patient suffering from MLD during the patient’s walk. As shown, the patient bent his arms to create stability in his body posture during the walk.

[0085] FIG. 6A shows the anime 604 of the patient’s body for image 602. Arm angles 606 and 608 are depicted in the anime. Positions of other body parts can be studied in addition or as an alternative to the arm angle discussed above. For example, one or both shoulder joint angles 610, 612 can be studies in addition to the arm angles 606, 608. A shoulder joint angle is an angle created between an arm and the torso of the patient. The torso can be represented by as few as two nodes representing respective shoulders, and two nodes representing hip sides, and by connecting the respective nodes to form the sides of the torso.

[0086] As another example, a change of the head angle relative to the shoulder can be studied as a representative of the patient’s motor function ability. In an embodiment where roaming is analyzed, the head can be represented by as few as one or two nodes, and the head angle can be determined based on the position of those node(s) relative to the shoulder node, or relative to the torso. Of course a more sophisticated study of the head is also possible, for example, where the head is represented by a more number of nodes (e.g., more than 100 nodes), which can lead to a three-dimensional (3D) representation of the head, which can include details of some parts of the head such as jaws, lips, eyebrows, etc.

[0087] In FIG. 6A, the head is represented by two lines that each connects (i) a jaw (or an ear) node representing a respective jaw (or ear), and (ii) a neck point representing a neck of the patient. As the patient tilts their head during the walk, the head angle 614 changes. Studying of the head angle can give information about the patient’s ability to stabilize their head during the walk, or can indicate the patient’s confidence in walking in a relaxed or a tensed manner.

[0088] FIG. 6B shows anime 620 of the patient’s body for the same image 602. In FIG. 6B, the head is represented by a mesh formed of more than 400 nodes. This allows the 3D representation 622 of the patient’s head during the walk. The 3D representation provides more data about the patient’s head angle (e.g., 624) in the 3D space as compared to the tilt representation (e.g., angle 614) shown in FIG. 6 A.

[0089] As noted above, roaming determiner 228 (or arm position detector 236) determines the disorder or its stage based on the analysis of the sequence of the arm angles. In some implementations, the analysis of the sequence of arm angles includes determining a minimum arm angle in the sequence of arm angles, and making a conclusion on the stage of a particular order based on the minimum arm angle. In some implementations, arm position detector 236 uses as the minimum arm angle an angle that consistently happens during the roaming activity and discards even lower arm angles that happen less frequent, e.g., less than in one fourth of the steps taken during the activity. For example, the analysis includes determining the instances that the minimum arm angle happens, and if the minimum arm angle happens sporadically (e.g., in less than one third of the total number of steps) throughout the roaming activity, searching for the next minimum arm angle that happens more frequently (or consistently) throughout the activity, and using that consistently-happening angle as the minimum arm angle.

[0090] In some implementations, the analysis of the sequence of arm angles includes determining a variance in the arm angle based on the changes in the angles during the roaming activity. For example, roaming determiner 228 can determine the disorder or its stage by comparing the variance to a predetermined association between the arm angle variances and different stages of the disorder. The more variance in an arm’s angle during a roaming indicates a better stability of the patient. Indeed, there is a correlation between the arm angle variance and the patient’s ability to take wider steps; and wider steps indicate more control of the muscles involved in roaming, which can imply a lower stage of the a particular disorder, or non-existence of that disorder for the patient.

[0091] While the arm angle is described in detail in this disclosure, other portions of the arm, or the patient’s body in general, can be studied in a similar manner. For example, disorder determiner 228 can include other position detector submodules (such as a wrist position detector, a finger position detector, a head position detector, a knee position detector, a hip position detector, etc.) that can operate in addition to or as part of arm position detector 236 to determine the position of the respective body portions (e.g., wrist, fingers, head, knee, hip, etc.) at different stages of a study, e.g., as governed by a protocol of the study. Similar to the arm angle described above, each of those submodules can determine the position of the respective body portion based on a respective angle (e.g., a wrist angle or fingers angle) of that body portion during the patient’s performance of the study protocol.

[0092] FIGs. 7A-7C show another example where the activity of interest includes standing up from a sitting position, and sitting back down from a standing position. The system (e.g., system 200 in FIG. 2) can identify a patient’s stage of a particular medical disorder based on the sequence of standing up and sitting down movements; for example, based on the time it takes the patient to stand up, the time it takes the patient to sit down, or both. The system can also combine the sitting / standing activity with other activities, e.g., the walking activity to take into consideration the movement and dexterity of more body parts. For example, the system can study the duration (e.g., Timed Up and Go (TUG)) that it takes the patient to get up from a sitting position, stand, walk certain number of (e.g., ten) steps, turn, walk back, stand, and sit down where the patient had started the activity.

[0093] In the example depicted in FIGs. 7A-7C, the system detects the patient’s sitting position based on the angles of the patient’s joins, for example, hip angle, knee(s) angle(s), arm(s) angle(s), etc. A hip angle is created between a first vector (e.g., 702) connecting a hip node to a shoulder node, and a second vector (e g., 704) connecting the hip node to a knee node. A knee angle is created between the second vector (e.g., 704) connecting a knee node to the hip node, and a third vector (e.g., 706) connecting the knee node to a foot node, e.g., an ankle node. The system can assign a respective hip angle and a respective knee angle to each side (i.e., a left side and a right side) of the patient.

[0094] In some embodiments, when at least one hip angle and at least one knee angle are approximately 90 degrees, the system identifies the sitting position. The angular proximities can be determined based on particular angle ranges pre-set for respective angles for a given study(e g., according to the study’s protocol). For example, hip angles between 60 and 120 degrees are considered as approximately 90 degrees, while knee angles between 70 and 110 degrees are considered as approximately 90 degrees for the purpose of sitting determination.

[0095] Similarly, in some embodiments, when at least one hip angle and at least one knee angle are approximately 180 degrees, the system identifies the standing position. Again, the angular proximities can be associated with respective angles, for example, within 30 degrees for the hip angle, and within 20 degrees for the knee angle.

[0096] In some examples, when at least one hip angle is smaller than a prespecified sitting hip angle, the patient is considered sitting. In some examples, at least one knee angle also needs to be smaller than a prespecified sitting knee angle to identify the patient as sitting. Sitting hip angle and sitting knee angles can be different from each other. For example, the sitting hip angle can be pre-set to 120 degrees, while sitting knee angle is pre-set to 110 degrees.

[0097] Similarly, in some embodiments, when at least one hip angle is larger than a prespecified standing hip angle, the system identifies the standing position. In some examples, at least one knee angle also needs to be greater than a prespecified standing knee angle to identify the patient as standing. Standing hip angle and standing knee angles can be different from each other. For example, the standing hip angle can be pre-set to 150 degrees, while the standing knee angle is pre-set to 170 degrees.

[0098] FIG. 7A shows a patient in an initial sitting position. The figure also depicts hip signal 708 that tracts the hip angle(s), and knee signal 710 that tracts the knee angle(s). FIG. 7B shows the patient standing. Hip signal 708 tracts the changes in the hip angle, and shows that at the moment depicted in FIG. 7B, the hip angle is close to (e.g., within 30 degrees from) 180 degrees. Knee signal 710 tracts the changes in the knee angle, and shows that at the moment depicted in FIG. 7B, the knee angle is close to (e.g., within 10 degrees from) 180 degrees.

[0099] In some embodiments, the system identifies as the standing point in time when the hip angle is at its maximum value in the hip signal 708. In some embodiments, the system identifies as standing point in time when the patient has reached a hip angle proximate to 180 degrees, and has been maintaining the hip angle within a certain angular margin (e.g., within 10 degrees) for a pre-specified period of time. In such embodiments, the patient has to stand for the pre-specified period of time (e.g., for at least 5 seconds) before moving to a next activity (e.g., walking or sitting down) so that the system can detect a proper standing time and position.

[0100] FIG. 7C. shows the patient sitting back down. As depicted, hip signal 708 and knee signal 710 show decreases in the hip angle and the knee angle, respectively, as the patient moved from the standing position to the sitting position.

[0101] Depending on the angle of the camera taking the patient’s images, other sides of patient’s body or other body portions can be used for this study. For example, while FIGs. 7A- 7C focus on the right side of the patient (due to the angle of the camera), measurements on the left side of the patient’s body can also be used, for example, as a confirmation for the identified bodily position. For example, in FIG. 7C the camera was able to depict the left side of the patient’s body, and the system was able to create the corresponding left hip and knee signals. Other body parts such as arm angles can be used as other indicators of patient’s sitting and / or standing positions.

[0102] FIGs. 8A-8D show another example, where the activity of interest is head movement. The system can determine the head control ability of the patient based on predefined head movement activities that the patient is instructed to perform. Such predefined head movement activities can include (but is not limited to) left / right turning, up / down nodding, and head tilting. Based on the extent of the movement activities that the patient can perform, the system can determine the stage of patient’s particular disorder. The extent of the movement activities can include (but is not limited to) the range of head movement angles, the speed of head movement, the continuity or variability (e.g., smooth vs. interrupted) of the movement, etc.

[0103] In FIG. 8A, the image of the patient’s face (802) is anonymized (804), and respective nodes are selected from a larger number of nodes that were included in the anonymized representation of the head. The system uses the selected nodes to measure respective angles of movements and / or statistics associated with the head movement.

[0104] Image 806 in FIG. 8A includes three measurement indicators (812, 814, 816) that each represents the head position with respect to a particular head movement. Measurement indicator 812 is a representative of the head’s left / right turning. Measurement indicator 814 is a representative of the head’s up / down nodding. Measurement indicator 816 is a representative of the head’s tilting.

[0105] FIG. 8B shows the change in measurement indicator 812 as the patient turns his head to left and to right. The measurement indicator 812 indicates how well the patient can turn his head with respect to a yaw (or vertical) axis passing through the patient’s body, e g., from thehead to the spine. The 1 eft / right range (which is -61 to 56 in this example) depicted by the measurement indicator 812 indicates an extent of the patient’s ability to turn his head left and right. In some examples, the system can study each of the left and right movements separately to identify the patient’s potential disability in turning in a particular direction (e.g., disability to turn to left).

[0106] FIG. 8C shows the change in measurement indicator 814 as the patient nodes his head up and down. The measurement indicator 814 indicates how well the patient can move his head up and down with respect to a pitch (or frontal) axis passing through the head, e.g., from one ear to another ear of the patient. The up / down range (here -33 to 13) depicted by the measurement indicator 814 indicates an extent of the patient’s ability to move his head up and down. In some examples, the system can study each of the up and down movements separately to identify the patient’s potential disability in nodding in a particular direction (e.g., disability to move head down).

[0107] FIG. 8D shows the change in measurement indicator 816 as the patient tilts his head. The measurement indicator 816 indicates how well the patient can tilt his head left and right with respect to a roll axis that is perpendicular to the yaw and pitch axes noted earlier. The tilt range (here -16 to 15) depicted by the measurement indicator 816 indicates an extent of the patient’s ability to tilt his head. In some examples, the system can study each of the left and right tilts separately to identify the patient’s potential disability in tilting in a particular direction (e.g., disability to tilt his chin left).

[0108] The system determines the stage of the patient’s disorder based on the measured head control ability as represented by the measurement indicators 812, 814, 816. In some implementations, the system compares the measurements to a history of the patient’s measurements to determine the progress or improvement of the patient’s disorder over time and / or to determine the efficacy of a treatment that the patient used between multiple (e.g., consecutive) measurements.

[0109] In some implementations, the system compares the patient’s measurements to cohorts of patients that each is associated with a respective stage of the patient’s disorder in order to determine the stage of the disorder on the patient. In an example, each cohort can be associated with a respective metric, e.g., a widely accepted metric of Expanded Disability Status Scale (EDSS). For example, an EDSS ranging from zero to ten, with zero indicating no disability andten indicating the highest disability, may be divided to four cohorts. The system can compare the patient’s measurements with the EDSS ranges in each cohort, and indicate how sever or progressed the patient’s disorder is as compared to the disorder of the patients in that cohort. In some examples, one or more of the cohorts also refer to respective treatments (e.g., respective study treatments). The system can identify those treatments for the patient based on the comparison.

[0110] FIGs. 9A-9B show another example, where the activity of interest is finger tapping. Measuring the patient’s finger tapping abilities is useful in diagnosing and determining stage of particular disorders such as Parkinson’s disease.

[0111] The system can study either or both hands abilities in finger tapping. In the depicted example of FIG. 9A, the system has anonymized patient’s left hand, and in FIG. 9B the system has anonymized patient’s right hand. The system has also identified particular nodes associated with the index and thumb fingers of each hand in the respective figure. The system uses the identified nodes to track the patient’s finger tapping abilities. To do so, the system tracks the distance between a particular index node (e.g., a node representing the top of the index finger) and a particular thumb node (e.g., a node representing the top of the thumb finger) over time and as the patient continues to tap his index finger to his thumb.

[0112] The system uses the statistics on the tapping time to determine the patient’s finger tapping abilities. For example, the system can use statistics such as mean value, standard deviation, etc. of the time passed between two consecutive tapping or of the time passed between the minimum distance between the two fingers (which represents a tap) and the maximum distance between the two fingers (which indicates the farthest position of the two fingers with respect to each other before the next tapping happens).

[0113] The system can study each hand separately, or can compare the two hands abilities, e.g., to normalize the ability based on the patient’s personal coordination. Similar to the other examples discussed above, the system can compare the patient’s tapping measurements to the patient’s history of tapping measurements, or to cohorts of patients that are at different stages of the disorder.

[0114] After the system determines the disorder (and / or its stage), the system sends the results to transmission / presentation module 242 for presentation to the patient and / or for presentation to the patient’s health care provider. Transmission / presentation module 242 can beassociated with a display that displays the result. For example, the module can include a screen, or can transmit the result to a device that has a screen for data presentation.

[0115] As noted above, system 200 can reside on device 104 (FIG. 1), or can be partly on device 104 and partly on backend system 112. For example, receiver module 212 and transformation module 214 can be on device 104, and ML module 240 can run on backend system 112. In such scenarios, backend system 112 can transmit the results back to the user device 104, e.g., to transmission / presentation module 242, for presentation to the patient 102. In general, data can be transmitted between device 104 and backend system 242 through a wired or wireless communication link between device 104 and backend system 112. To protect patient’s privacy, in some implementations, the data transfer includes only the anime 108a without specifying any identification of the patient.

[0116] In some implementations, device 104 analyzes the video and determines the stage of the disorder independently of the backend system 112. In some implementations, device 104 confirms its determination with the backend system 112. Alternatively or in addition, device 104 can periodically receive updates of the model or algorithm from system 112, or system 112 can send such updates, when available, to device 104.

[0117] Since backend system 112 supports multiple devices like device 104 associated with different patients, device 104 can benefit from the improvements that system 112 has done on the model or algorithm (that each device can use to determine the disorder / disorder stage) based on the data that the system dynamically receives from those devices over time. For example, backend system 112 may change or update biomarkers used to determine the disorder / stage, based on the data backend system 112 receives from those devices, and communications with physician devices (e.g., 122) confirming or making a change into the determined disorder / stage.

[0118] In some implementations, device 104 sends video 108b (with or without the disorder stage that system 200 determined) to physician device 122. The physician can use the video to make their own independent determination of the disorder stage. Backend system 112 can receive from physician’s device 122 the physician’s diagnosis, and use such information to modify its disorder determiner model or algorithm.

[0119] In some implementations, device 104 sends the system’s determined disorder to the physician’s device 122 to help the physician in making their disorder determination or in prescribing treatments or medication for the patient. The physician (or health care provide, ingeneral) can approve, modify, or deny system’s determination. Physician’s device sends such information to system 200, and system 200 uses this information to improve its model or algorithm.

[0120] In some implementations, the analysis of the sequence of arm angles include determining a signal representing gait cycles by determining repetitions in the arm angles in consecutive periods of time during the walk, and determining, based on the gait cycles, step parameters for the patient. Disorder determiner 228, or either of leg position detector 234 and arm position detector 236 alone or in correlation with each other, can determine the disorder or its stage based on the step parameters obtained from studying the gait cycles. For example, disorder determiner 228 can analyze the patient’s ability to move any of the muscles (e.g., either of arms, either of legs, hip muscles, etc.) involved in a walk during each of the gait cycles to determine a persistent muscle failure or limitation happening repetitively during the walk, and make a conclusion on the stage of a particular disorder based on that failure or limitation. The step parameters can include step duration, step count, step frequency, cadence, step width, particular join bent (based on the respective angle it creates), etc.

[0121] In some implementations, disorder determiner 228 determines the stage of the disorder based on a predetermined associations between different stages of the disorder and different values of one or more particular step parameters. For example, disorder determiner 228 can retrieve the predetermined associations (e.g., from a storage device), and compare the determined step parameter of the patient to the predetermined associations to indicate the stage of disorder. The predetermined associations can be in form of a table that maps each disorder stage to a range of the step parameters (e.g., a range of step frequencies) for patients with biological biomarkers (e.g., age, gender, weight, height, other chronic diseases, etc.) similar to the biological biomarkers of the patient. In this way, the step parameters (e.g., step frequencies, cadence) can help in quantifying the disorder stage.

[0122] In some implementations, disorder determiner 228 determines the stage of the medical disorder based on a progress or a regress of the disorder according to a medical history of the patient. In other words, disorder determiner 228 can determine the disorder stage based on a progress or a regress of the patient in one or more step parameters over time. For example, if the patient was previously diagnosed with a particular stage of the disorder (either by system 200 or by a medical practitioner), disorder determiner 228 can use that history to determine howmuch faster or slower the patient walks now as compared to the time of that previous diagnosis, and provide an estimate of the progress of the disorder or an improvement of the patient’s condition.

[0123] The amount of time that has passed can be a factor in these determinations. For example, the disorder determiner 228 may determine a less or slower improvement for a first patient whose previous diagnosis was done five years ago and whose step frequency has improved by one-tenth since then, as compared to a second patient whose previous diagnosis was done six months ago and shows an improved step frequency of one-tenth.

[0124] In some implementations, disorder determiner 228 can determine a treatment efficacy for a treatment that the patient undergoes, based on the disorder stage(s) determined at different times. For example, based on the stages of the disorder determined at multiple (e.g., two) rounds of diagnosis and the time period passed between the multiple diagnoses, disorder determiner 228 can determine how effective or ineffective the treatment has been in improving the patient’s condition or in stalling a progress of the disorder. Disorder determiner 228 can, for example, compare the progress of the patient’s disorder to an average progress of the disorder on (other) patients suffering from the same disorder over a similar period of time to determine the treatment efficacy on the patient. Two periods of time are “similar” if they differ by a specific amount, e.g., by one-tenth of the overall time periods. For example, a nine-months time period is similar to a ten-months time period when the specific amount of time difference is set as one-tenth, which would indicate one-tenth of ten months.

[0125] As discussed above, the coordinates of each node can be determined with respect to a particular node representing a particular part of the patient’s body. The particular part can be patient’s hip or chest, and the particular node can be a hip node or a chest node assigned to the hip or the chest, respectively. For example, transformation module 214 can set the hip node to coordinates (0,0,0), and set the coordinates of every other node based on the node’s spatial position with respect to the hip node. In some examples, the particular node can the center of mass of the patient’s body.

[0126] The anonymized presentation (e.g., anime 108a) disclosed in this disclosure can be a three dimensional (3D) representation of the patient’s body. A 3D representation provides more degrees of freedom on the camera angle with respect to the patient. Accordingly, the patient hasmore freedom to roam or move their muscles in different directions without having to be located right in front of the camera (which is a limitation for 2D representations).

[0127] As discussed above, the implementations provided in this disclosure provide the patients the convenience of using their personal devices to evaluate their health. For example, in FIG. 1, patient 102 can use a camera on their smart phone or mobile device (e.g., device 104) to capture a video or a set of consecutive images of the patient while the patient is walking.

[0128] Software or an application can be downloaded on the personal device to analyze the video according to the description discussed above. Alternatively, or in addition, the device can send the image or video to an external device, e.g., to system 112, to perform the analysis, to confirm the output of the personal device’s analysis, and / or to update the model or algorithm used to perform the analysis.

[0129] In some implementations, the personal device can also provide instructions regarding the physical activity to the patient. For example, an application may have been downloaded on the personal device that facilitates communications with an external server or system (e.g., 112) to provide the patient with instructions on what activities to perform for different disorders. The instructions are presented on a user interface of the personal device.

[0130] The application can also specify the type of the activity based on the patient’s health or medical history, e.g., based on prior diagnoses on the patient. For example, for a patient suffering from an early stage (e.g., stage 1) of MLD, the application may identify “walk” as the activity to be performed, while for a patient suffering from a more advanced stage (e g., stage 4) of MLD, the application may instruct the patient to perform an “eat” or “speak” based activity.

[0131] The application can also provide details for the activity to be performed. For a walk activity, for example, the application can specify number of steps, direction of walk, duration of the walk, etc. For a speech activity, for example, the application can specify particular words or phrases to be spoken, how fast to speak, etc.

[0132] The application can also guide the patient to correct a performed activity if the patient does not comply with the instructions that the application provides. For example, the personal device can compare the movements of the one or more targeted body parts (e.g., legs in a walk activity) to the instructions, and in response to determining that the movements do not comply the instructions, and present further instructions to the patient to guide the patient to try the activity again through the suggested correction. The further instructions can include suggestingparticular actions to correct the movements. For example, it can include increasing the number of steps if the patient did not walk long enough, or the speed of the walk, if the patient walked slower than expected, or reiterating a particular phrase, of the patient missed or did not speak part of the phrase, etc.

[0133] In some implementations, the application (e.g., smartphone application) has the primary responsibility of acquiring video and anonymizing the video. In some implementations, the application can act as an enforcer and / or as a real-time (e.g., at home) guide of a study protocol designed for the patient. A study protocol includes a set of rules designed for the patient to perform so that the stage of the patient’s disorder can be diagnosed. A study protocol can be designed by a medical practitioner or by system 200. For example, system 200 can identify / determine from a set of predetermined study protocols stored at (or accessible by) system 200, a particular study protocol for the patient. The study protocol for a patient is determined based on the biomedical biomarkers and / or a medical history of the patient.

[0134] In an example, the application initially conducts a real-time analysis of the patient’s biomarkers, e.g., by analyzing the motor functions of the patient, and then guides the patient to perform tasks included in a particular study protocol determined for the patient. The application can determine the particular study protocol based on the real-time analysis of the video.

[0135] The application can provide a step by step (or task by task) guidance for the patient to perform the study protocol, e.g., through audio or a display of text, images, or movie clip. An example of the application’s step by step guide of a particular protocol can include the following steps (where the application is running on “the phone”):Please put the phone on the provided tripod;Please flip the phone so that the front / selfie camera and phone screen face you / the patient;Raise the phone a little higher / lower;Rotate the phone left / right to make the subject at the center of the screen.

[0136] In an example of a study protocol where a patient’s head mobility is to be studied, for example, the patient’s ability to rotate or tilt head to left and right, to up and down can be tested. In this example, the application guides the patient to turn head left, right, etc. as the patient progresses through the study protocol. The application can also provide a feedback to the patient for each step or task along the study protocol before moving to the next step / task. For example,the application may indicate that the patient has turned their head for a good degree, and now it is time to move to the next step to tilt the head to the right.

[0137] Another example of a study protocol is to study the gait of the patient while walking. The application can provide details of each or multiple steps of the walk. For example, the application may indicate to the patient to walk forward for ten steps, then stop, then turn 180 degrees, and then return. As noted above, the application may give the guidance to the patient after each step or task (e.g., after 10 steps) and / or give the patient the overall protocol to perform. In an example task by task (or step by step) scenario, the application can output an audio stating: start walking. After the patient takes each step of the 10 steps, the application can provide a count of the steps taken properly and ask the patient to redo the steps that were not taken properly. After ten steps, the application can ask the patient to turn. At the end, the application can indicate an end of the gait study to the patient.

[0138] As discussed earlier, there are other study protocols that can be used to study the patient’s mobility with respect to different parts of the body and depending on the target disorder. Examples of such study protocols include: sitting upright with tasks including lift one foot to a maximum knee angle, repeat for the other foot; or sitting to standing, e.g., for 10 times, with tasks including (timed up and go) performance of sitting to standing, then walking for 25 feet, return by walking back 25 feet, and sit down.

[0139] In addition or alternative to giving the real time feedback to the patient about the patient’s performance during a protocol study, the application can provide to the patient a selfreflection summary of the patient’s performance based on a history of the patient’s past use of the application. For example, the application can indicate that the patient has walked 10% faster than last week, or 10% faster than usual (usual being an average of the walk speed over a specific period of time, e.g., a month).

[0140] As noted above, the application can guide the patient about and through performing a study protocol or a particular task. The personal device can present the guide to the patient as audio, for example spoken words played through a speaker of the personal device. Alternatively, or in addition, the personal device can present the guide (at least partly) as a visual cue through a screen (e.g., phone screen) of the personal device. For example, the guide can include images or video clips of a correct way of performing the physical activity / study protocol or a particular task within the study protocol. The visual cue can include particular graphical user interface elementssuch as arrows, highlighted areas of a particular image, etc., to emphasize on a particular movement of a body portion during a specific task.

[0141] The application can provide the guide on an ongoing task, for example, in real-time, and as the patient is performing the task. For example, upon detecting an error or a deviation of the patient in performing a particular task in a study protocol, the application can warn the patient about the error, and guide the patient to correct their error through the audio or visual presentations discussed above. Depending on the severity of the error and how much it affects the overall study, the application can request the patient to redo just that particular task or the whole study (which includes multiple consecutive tasks.

[0142] FIG. 4 depicts an example process 400 that can be executed in accordance with implementations of the present disclosure. Process 400 is performed by a computer system, e.g., system 200 depicted in FIG. 2.

[0143] In process 400, the computing system receives (402) consecutive images (or a video) depicting a patient performing a particular physical activity. The system transforms (404) the images into anonymized representations. Each anonymized representation is an anime of the patient’s body as depicted in an image in the consecutive images. Each anime includes multiple nodes, that each represents a particular body part of the patient in a respective image represented by the anime. In an anime, each node can be connected to one or mode adjacent nodes by a line. Each node is identifiable by a spatial coordinate assigned to the node.

[0144] Depending on what particular body parts are of interest, the system identifies (406) in the anime target nodes associated to those body parts. For example, if the movements of legs is of interest, the system identifies in the anime an anime segment that represents legs through target nodes associated to the legs.

[0145] The system determines (408) respective coordinate sequences representing the target nodes’ movements as depicted in the consecutive images. Each coordinate sequence is associated to one node, and includes spatial coordinates of that node as they change in the consecutive images.

[0146] The system analyses (410) the coordinate sequences to determine a disorder of a stage of a disorder of the patient. Depending on the physical activity, one or multiple modules and sub-modules of the system can be involved in such analysis. For example, for a walk activity, sub-modules that analyze arm node and / or leg node (e g., 234, 236) can get involved, while foran eat activity, one or more sub-modules of facial movement analyzer 238 can get involved. To determine a particular disorder or a stage of a particular disorder, the system can compare the movements of the respective body parts to movement abilities that are typical for cohorts of patients suffering that particular disorder, and output the disorder or its stage based on the closest match — i.e., the cohort whose patients have the closest abilities or movement limitations to the patient’s abilities and movement limitations.

[0147] The system transmits (412) the determined disorder or stage for presentation. For example, a processor that has made the determination can transmit the result to a display component of the system. Both the processor and the display can reside on the same device (e.g., 104 in FIG. 1) or on separate devices, or on separate devices (e.g., one on backend system 112 and the other on device 104).

[0148] FIG. 5 shows an example of a computing device 500 and an example of a mobile computing device that can be used to implement the techniques described here. For example, the system 200 depicted in FIG. 2 can be in the form of the computing device 500, the mobile computing device 550, or a combination of them. The computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The mobile computing device is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and / or claimed in this document.

[0149] The computing device 500 includes a processor 502, a memory 504, a storage device 506, a high-speed interface 508 connecting to the memory 504 and multiple high-speed expansion ports 510, and a low-speed interface 512 connecting to a low-speed expansion port 514 and the storage device 506. Each of the processor 502, the memory 505, the storage device 506, the high-speed interface 508, the high-speed expansion ports 510, and the low-speed interface 512, are interconnected using various busses, and can be mounted on a common motherboard or in other manners as appropriate. The processor 502 can process instructions for execution within the computing device 500, including instructions stored in the memory 504 or on the storage device 506 to display graphical information for a GUI on an external input / outputdevice, such as a display 516 coupled to the high-speed interface 508. In other implementations, multiple processors and / or multiple buses can be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices can be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

[0150] The memory 504 stores information within the computing device 500. In some implementations, the memory 504 is a volatile memory unit or units. In some implementations, the memory 504 is a non-volatile memory unit or units. The memory 504 can also be another form of computer-readable medium, such as a magnetic or optical disk.

[0151] The storage device 506 is capable of providing mass storage for the computing device 500. In some implementations, the storage device 506 can be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. A computer program product can be tangibly embodied in an information carrier. The computer program product can also contain instructions that, when executed, perform one or more methods, such as those described above. The computer program product can also be tangibly embodied in a computer- or machine-readable medium, such as the memory 505, the storage device 506, or memory on the processor 502.

[0152] The high-speed interface 508 manages bandwidth-intensive operations for the computing device 500, while the low-speed interface 512 manages lower bandwidth-intensive operations. Such allocation of functions is exemplary only. In some implementations, the highspeed interface 508 is coupled to the memory 505, the display 516 (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports 510, which can accept various expansion cards (not shown). In the implementation, the low-speed interface 512 is coupled to the storage device 506 and the low-speed expansion port 515. The low-speed expansion port 515, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

[0153] The computing device 500 can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a standard server 520, or multiple times in a group of such servers. In addition, it can be implemented in a personal computer such as a laptop computer 522. It can also be implemented as part of a rack server system 525. Alternatively, components from the computing device 500 can be combined with other components in a mobile device (not shown), such as a mobile computing device 550. Each of such devices can contain one or more of the computing device 500 and the mobile computing device 550, and an entire system can be made up of multiple computing devices communicating with each other.

[0154] The mobile computing device 550 includes a processor 552, a memory 565, an input / output device such as a display 555, a communication interface 566, and a transceiver 568, among other components. The mobile computing device 550 can also be provided with a storage device, such as a micro-drive or other device, to provide additional storage. Each of the processor 552, the memory 565, the display 555, the communication interface 566, and the transceiver 568, are interconnected using various buses, and several of the components can be mounted on a common motherboard or in other manners as appropriate.

[0155] The processor 552 can execute instructions within the mobile computing device 550, including instructions stored in the memory 565. The processor 552 can be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor 552 can provide, for example, for coordination of the other components of the mobile computing device 550, such as control of patient interfaces, applications run by the mobile computing device 550, and wireless communication by the mobile computing device 550.

[0156] The processor 552 can communicate with a patient through a control interface 558 and a display interface 556 coupled to the display 555. The display 554 can be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface 556 can comprise appropriate circuitry for driving the display 554 to present graphical and other information to a patient. The control interface 558 can receive commands from a patient and convert them for submission to the processor 552. In addition, an external interface 562 can provide communication with the processor 552, so as to enable near area communication of the mobile computing device 550 with other devices. The external interface 562 can provide, for example,for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces can also be used.

[0157] The memory 564 stores information within the mobile computing device 550. The memory 564 can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. An expansion memory 574 can also be provided and connected to the mobile computing device 550 through an expansion interface 572, which can include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory 574 can provide extra storage space for the mobile computing device 550, or can also store applications or other information for the mobile computing device 550. Specifically, the expansion memory 574 can include instructions to carry out or supplement the processes described above, and can include secure information also. Thus, for example, the expansion memory 574 can be provide as a security module for the mobile computing device 550, and can be programmed with instructions that permit secure use of the mobile computing device 550. In addition, secure applications can be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

[0158] The memory can include, for example, flash memory and / or NVRAM memory (nonvolatile random access memory), as discussed below. In some implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The computer program product can be a computer- or machine-readable medium, such as the memory 565, the expansion memory 575, or memory on the processor 552. In some implementations, the computer program product can be received in a propagated signal, for example, over the transceiver 568 or the external interface 562.

[0159] The mobile computing device 550 can communicate wirelessly through the communication interface 566, which can include digital signal processing circuitry where necessary. The communication interface 566 can provide for communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access),CDMA2000, or GPRS (General Packet Radio Service), among others. Such communication can occur, for example, through the transceiver 568 using a radio-frequency. In addition, short-range communication can occur, such as using a Bluetooth, WiFi, or other such transceiver (not shown). In addition, a GPS (Global Positioning System) receiver module 570 can provide additional navigation- and location-related wireless data to the mobile computing device 550, which can be used as appropriate by applications running on the mobile computing device 550.

[0160] The mobile computing device 550 can also communicate audibly using an audio codec 560, which can receive spoken information from a patient and convert it to usable digital information. The audio codec 560 can likewise generate audible sound for a patient, such as through a speaker, e.g., in a handset of the mobile computing device 550. Such sound can include sound from voice telephone calls, can include recorded sound (e.g., voice messages, music files, etc.) and can also include sound generated by applications operating on the mobile computing device 550.

[0161] The mobile computing device 550 can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a cellular telephone 580. It can also be implemented as part of a smart-phone 582, personal digital assistant, or other similar mobile device.

[0162] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0163] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms machine-readable medium and computer-readable medium refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used toprovide machine instructions and / or data to a programmable processor, including a machine- readable medium that receives machine instructions as a machine-readable signal. The term machine-readable signal refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0164] To provide for interaction with a patient, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the patient and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the patient can provide input to the computer. Other kinds of devices can be used to provide for interaction with a patient as well; for example, feedback provided to the patient can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the patient can be received in any form, including acoustic, speech, or tactile input.

[0165] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical patient interface or a Web browser through which a patient can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0166] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0167] A number of implementations of the present disclosure have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims.

Claims

WHAT IS CLAIMED IS:

1. A method executed by a computing system, the method comprising: receiving a plurality of consecutive images taken of a patient while performing a physical activity; transforming the images to respective anonymized representations of a patient body of the patient, each anonymized representation including a plurality of nodes that each represents a particular body portion of the patient body, each node being identifiable by a respective spatial coordinate assigned to the node; identifying from the plurality of nodes a set of target nodes representing one or more targeted body parts of the patient; determining a set of coordinate sequences that represent movements of the one or more targeted body parts during the physical activity, each coordinate sequence in the set including spatial coordinates of a respective target node associated with a target body portion as depicted in the consecutive images; analyzing the set of coordinate sequences to determine a stage of a medical disorder that the patient suffers, the medical disorder being a neurological or musculoskeletal disorder; and sending the determined stage to a user interface for presentation.

2. The method of claim 1, wherein the physical activity is at least one of movement of a head of the patient, talking, chewing, or swallowing, and the one or more targeted body parts includes at least one of the head, mouth, lips, or neck of the patient.

3. The method of any of claims 1 or 2, wherein the physical activity includes walking, wherein the one or more targeted body parts include at least portions of legs of the patient, and the set of target nodes includes a first leg node representing a portion of a first leg, and a second leg node representing a portion of a second leg of the patient, and wherein analyzing the set of coordinate sequences comprises: comparing a first coordinate sequence of the first leg node to a second coordinate sequence of the second leg node to obtain a signal representing gait cycles during the walking, each gait cycle starting from a respective starting point when the first leg passes the second leg along a walking direction,determining, based on the gait cycles, step parameters for the patient, the step parameters including one or more of step duration, step count, step frequency, or cadence, and determining the stage of the medical disorder based on the step parameters.

4. The method of claim 3, wherein each gait cycle ends at a respective point when the second leg passes the first leg along the walking direction.

5. The method of any of claims 3 or 4, wherein a gait cycle starts from a point when the first leg passes the second leg along the walking direction, and ends the next time the first leg passes the second leg.

6. The method of any of claims 3-5, further comprising determining a frontal plane of the patient body, wherein the first leg passes the second leg when a first coordinate of the first leg point changes from a negative value to zero or a positive value in a direction perpendicular to the frontal plane.

7. The method of claim 6, further comprising determining that the physical activity is walking by determining that the first coordinate changes between a negative value and a positive value alternatively and periodically.

8. The method of any of claims 3-7, wherein the first leg node represents at least part of the first leg between a hip and an ankle of the first leg.

9. The method of claim 8, wherein the first leg node represents a knee, an ankle, a shin, or a thigh of the first leg.

10. The method of any of claims 3-9, wherein the step parameters include the step frequency or cadence, and wherein the stage of the medical disorder is determined based on a predetermined association between different stages of the disorder and different step frequencies or cadences.

11. The method of any of claims 1-10, wherein the physical activity includes walking, wherein the one or more target body parts includes at least a portion of an arm of the patient, and the set of target nodes includes an elbow node representing an elbow of the arm, an upper-arm node representing a body portion between the elbow and a shoulder of the arm, and a lower-arm node representing a finger of the arm or a section between the finger and the elbow, wherein analyzing the set of coordinate sequences comprises determining a sequence of arm angles each formed between the upper-arm node and the lower-arm node with respect to the elbow node at a respective image of the walk, and wherein the stage of the medical disorder is determined based on an analysis of the sequence of arm angles.

12. The method of claim 11, wherein the analysis of the sequence of arm angles comprises determining a variance in the arm angle based on a change in the angles in the sequence of arm angles, and wherein the stage of the medical disorder is determined by comparing the variance to a predetermined association between arm angle variances and different stages of the medical disorder.

13. The method of any of claims 11 or 12, wherein the analysis of the sequence of arm angles comprises: determining a signal representing gait cycles by determining repetitions in changes in the arm angles in consecutive periods of time during the walking, and determining, based on the gait cycles, step parameters for the patient, the step parameters including at least one of step duration, step count, or step frequency, wherein the stage of the medical disorder is determined based on the step parameters.

14. The method of any of claims 11-13, wherein the analysis of the sequence of arm angles includes determining a minimum arm angle in the sequence of arm angles, wherein the stage of the medical disorder is determined based on the minimum arm angle.

15. The method of any of claims 1-14, further comprising: determining a hip node representing a hip of the patient, and setting a spatial coordinate of the hip node as (0,0,0), wherein spatial coordinates of other nodes are determined relative to the hip node.

16. The method of any of claims 1-15, wherein the anonymized representation is a 3- dimensional representation of the patient body.

17. The method of any of claims 1-16, wherein the computing system is a mobile device or a smart phone.

18. The method of claim 17, wherein the mobile device or the smart phone includes a camera used to capture the plurality of consecutive images.

19. The method of any of claims 1-18, further comprising causing the user interface to present instructions for performing the physical activity.

20. The method of claim 19, further comprising: comparing the movements of the one or more targeted body parts to the instructions; and in response to determining that the movements do not comply the instructions, sending further instructions to the user interface to guide the patient redo the physical activity, wherein the further instructions include suggesting particular actions to correct the movements.

21. The method of any of claims 1-20, wherein the stage of the medical disorder is determined based on a progress or a regress of the disorder over a period of time, the progress or the regress being determined by obtaining, from a medical history of the patient, an old set of coordinate sequences that was determined for a prior physical activity that the patient performed at a past time before performing the physical activity, the past time indicating a beginning of the period of time, andcomparing the set of coordinate sequences to the old set of coordinate sequences to determine an improvement or a deterioration of a motor function of the patient, wherein the improvement is associated with the regress of the disorder, and the deterioration is associated with the progress of the disorder.

22. The method of any of claims 1-21, further comprising determining a treatment efficacy of a treatment that the patient undergoes based on the determined stage and a previously determined stage of the disorder, and based on a time passed from the previously determined stage.

23. The method of any of claims 1-22, further comprising causing the user interface to present real-time feedback about the patient’s performance of the physical activity as the patient performs the physical activity.

24. A system comprising: one or more computers; and one or more computer-readable storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving a plurality of consecutive images taken from a patient while performing a physical activity; transforming the images to respective anonymized representations of a patient body of the patient, each anonymized representation including a plurality of nodes that each represents a particular body portion of the patient body, each node being identifiable by a respective spatial coordinate assigned to the node; identifying from the plurality of nodes a set of target nodes representing one or more targeted body parts of the patient; determining a set of coordinate sequences that represent movements of the one or more targeted body parts during the physical activity, each coordinate sequence in the set including spatial coordinates of a respective target node associated with a target body portion as depicted in the consecutive images;analyzing the set of coordinate sequences to determine a stage of a medical disorder that the patient suffers, the medical disorder being a neurological or musculoskeletal disorder; and sending the determined stage to a user interface for presentation.

25. The system of claim 24, wherein the physical activity is at least one of movement of a head of the patient, talking, chewing, or swallowing, and the one or more targeted body parts includes at least one of the head, mouth, lips, or neck of the patient.

26. The system of any of claims 24 or 25, wherein the physical activity includes walking, wherein the one or more targeted body parts include at least portions of legs of the patient, and the set of target nodes includes a first leg node representing a portion of a first leg, and a second leg node representing a portion of a second leg of the patient, and wherein analyzing the set of coordinate sequences comprises: comparing a first coordinate sequence of the first leg node to a second coordinate sequence of the second leg node to obtain a signal representing gait cycles during the walking, each gait cycle starting from a respective starting point when the first leg passes the second leg along a walking direction, determining, based on the gait cycles, step parameters for the patient, the step parameters including one or more of step duration, step count, or step frequency, and determining the stage of the medical disorder based on the step parameters.

27. The system of claim 26, wherein each gait cycle ends at a respective point when the second leg passes the first leg along the walking direction.

28. The system of any of claims 26 or 27, wherein herein a gait cycle starts from a point when the first leg passes the second leg along the walking direction, and ends the next time the first leg passes the second leg.

29. The system of any of claims 26-28, wherein the operations further comprise determining a frontal plane of the patient body, wherein the first leg passes the second leg when a first coordinate of the first leg pointchanges from a negative value to zero or a positive value in a direction perpendicular to the frontal plane.

30. The system of claim 29, wherein the operations further comprise determining that the physical activity is walking by determining that the first coordinate changes between a negative value and a positive value alternatively and periodically.

31. The system of any of claims 26-30, wherein the first leg node represents at least part of the first leg between a hip and an ankle of the first leg.

32. The system of claim 31, wherein the first leg node represents a knee, an ankle, a shin, or a thigh of the first leg.

33. The system of any of claims 26-32, wherein the step parameters include the step frequency or cadence, and wherein the stage of the medical disorder is determined based on a predetermined association between different stages of the disorder and different step frequencies or cadences.

34. The system of any of claims 24-33, wherein the physical activity includes walking, wherein the one or more target body parts includes at least a portion of an arm of the patient, and the set of target nodes includes an elbow node representing an elbow of the arm, an upper-arm node representing a body portion between the elbow and a shoulder of the arm, and a lower-arm node representing a finger of the arm or a section between the finger and the elbow, wherein analyzing the set of coordinate sequences comprises determining a sequence of arm angles each formed between the upper-arm node and the lower-arm node with respect to the elbow node at a respective image of the walk, and wherein the stage of the medical disorder is determined based on an analysis of the sequence of arm angles.

35. The system of claim 34, wherein the analysis of the sequence of arm angles comprises determining a variance in the arm angle based on a change in the angles in the sequence of arm angles, and wherein the stage of the medical disorder is determined by comparing the variance to a predetermined association between arm angle variances and different stages of the medical disorder.

36. The system of any of claims 34 or 35, wherein the analysis of the sequence of arm angles comprises: determining a signal representing gait cycles by determining repetitions in changes in the arm angles in consecutive periods of time during the walking, and determining, based on the gait cycles, step parameters for the patient, the step parameters including at least one of step duration, step count, or step frequency, wherein the stage of the medical disorder is determined based on the step parameters.

37. The system of any of claims 34-36, wherein the analysis of the sequence of arm angles includes determining a minimum arm angle in the sequence of arm angles, wherein the stage of the medical disorder is determined based on the minimum arm angle.

38. The system of any of claims 24-37, wherein the operations further comprise: determining a hip node representing a hip of the patient, and setting a spatial coordinate of the hip node as (0,0,0), wherein spatial coordinates of other nodes are determined relative to the hip node.

39. The system of any of claims 24-38, wherein the anonymized representation is a 3- dimensional representation of the patient body.

40. The system of any of claims 24-39, wherein the one or more computers include a mobile device or a smart phone.41 . The system of claim 40, wherein the mobile device or the smart phone includes a camera used to capture the plurality of consecutive images.

42. The system of any of claims 24-41, wherein the operations further comprise causing the user interface to present instructions for performing the physical activity.

43. The system of claim 42, wherein the operations further comprise: comparing the movements of the one or more targeted body parts to the instructions; and in response to determining that the movements do not comply the instructions, sending further instructions to the user interface to guide the patient redo the physical activity, wherein the further instructions include suggesting particular actions to correct the movements.

44. The system of any of claims 24-43, wherein the stage of the medical disorder is determined based on a progress or a regress of the disorder over a period of time, the progress or the regress being determined by obtaining, from a medical history of the patient, an old set of coordinate sequences that was determined for a prior physical activity that the patient performed at a past time before performing the physical activity, the past time indicating a beginning of the period of time, and comparing the set of coordinate sequences to the old set of coordinate sequences to determine an improvement or a deterioration of a motor function of the patient, wherein the improvement is associated with the regress of the disorder, and the deterioration is associated with the progress of the disorder.

45. The system of any of claims 24-44, wherein the operations further comprise determining a treatment efficacy of a treatment that the patient undergoes based on the determined stage and a previously determined stage of the disorder, and based on a time passed from the previously determined stage.

46. The system of any of claims 24-45, wherein the operations further comprise causing the user interface to present real-time feedback about the patient’s performance of the physical activity as the patient performs the physical activity.

47. A non-transitory, computer-readable medium storing one or more instructions that when executed by a computing system cause the computing system to perform operations comprising: receiving a plurality of consecutive images taken from a patient while performing a physical activity; transforming the images to respective anonymized representations of a patient body of the patient, each anonymized representation including a plurality of nodes that each represents a particular body portion of the patient body, each node being identifiable by a respective spatial coordinate assigned to the node; identifying from the plurality of nodes a set of target nodes representing one or more targeted body parts of the patient; determining a set of coordinate sequences that represent movements of the one or more targeted body parts during the physical activity, each coordinate sequence in the set including spatial coordinates of a respective target node associated with a target body portion as depicted in the consecutive images; analyzing the set of coordinate sequences to determine a stage of a medical disorder that the patient suffers, the medical disorder being a neurological or musculoskeletal disorder; and sending the determined stage to a user interface for presentation.

48. The non-transitory, computer-readable medium of claim 47, wherein the physical activity is at least one of movement of a head of the patient, talking, chewing, or swallowing, and the one or more targeted body parts includes at least one of the head, mouth, lips, or neck of the patient.

49. The non-transitory, computer-readable medium of any of claims 47 or 48, wherein the physical activity includes walking, wherein the one or more targeted body parts include at least portions of legs of the patient, and the set of target nodes includes a first leg node representing a portion of a first leg, and a second leg node representing a portion of a second leg of the patient, and wherein analyzing the set of coordinate sequences comprises:comparing a first coordinate sequence of the first leg node to a second coordinate sequence of the second leg node to obtain a signal representing gait cycles during the walking, each gait cycle starting from a respective starting point when the first leg passes the second leg along a walking direction, determining, based on the gait cycles, step parameters for the patient, the step parameters including one or more of step duration, step count, or step frequency, and determining the stage of the medical disorder based on the step parameters.

50. The non-transitory, computer-readable medium of claim 49, wherein each gait cycle ends at a respective point when the second leg passes the first leg along the walking direction.

51. The non-transitory, computer-readable medium of any of claims 49 or 50, wherein herein a gait cycle starts from a point when the first leg passes the second leg along the walking direction, and ends the next time the first leg passes the second leg.

52. The non-transitory, computer-readable medium of any of claims 49-51, wherein the operations further comprise determining a frontal plane of the patient body, wherein the first leg passes the second leg when a first coordinate of the first leg point changes from a negative value to zero or a positive value in a direction perpendicular to the frontal plane.

53. The non-transitory, computer-readable medium of claim 52, wherein the operations further comprise determining that the physical activity is walking by determining that the first coordinate changes between a negative value and a positive value alternatively and periodically.

54. The non-transitory, computer-readable medium of any of claims 49-53, wherein the first leg node represents at least part of the first leg between a hip and an ankle of the first leg.

55. The non-transitory, computer-readable medium of claim 54, wherein the first leg node represents a knee, an ankle, a shin, or a thigh of the first leg.

56. The non-transitory, computer-readable medium of any of claims 49-55, wherein the step parameters include the step frequency or cadence, and wherein the stage of the medical disorder is determined based on a predetermined association between different stages of the disorder and different step frequencies or cadences.

57. The non-transitory, computer-readable medium of any of claims 47-56, wherein the physical activity includes walking, wherein the one or more target body parts includes at least a portion of an arm of the patient, and the set of target nodes includes an elbow node representing an elbow of the arm, an upper-arm node representing a body portion between the elbow and a shoulder of the arm, and a lower-arm node representing a finger of the arm or a section between the finger and the elbow, wherein analyzing the set of coordinate sequences comprises determining a sequence of arm angles each formed between the upper-arm node and the lower-arm node with respect to the elbow node at a respective image of the walk, and wherein the stage of the medical disorder is determined based on an analysis of the sequence of arm angles.

58. The non-transitory, computer-readable medium of claim 57, wherein the analysis of the sequence of arm angles comprises determining a variance in the arm angle based on a change in the angles in the sequence of arm angles, and wherein the stage of the medical disorder is determined by comparing the variance to a predetermined association between arm angle variances and different stages of the medical disorder.

59. The non-transitory, computer-readable medium of any of claims 57 or 58, wherein the analysis of the sequence of arm angles comprises: determining a signal representing gait cycles by determining repetitions in changes in the arm angles in consecutive periods of time during the walking, and determining, based on the gait cycles, step parameters for the patient, the step parameters including at least one of step duration, step count, or step frequency, wherein the stage of the medical disorder is determined based on the step parameters.

60. The non-transitory, computer-readable medium of any of claims 57-59, wherein the analysis of the sequence of arm angles includes determining a minimum arm angle in the sequence of arm angles, wherein the stage of the medical disorder is determined based on the minimum arm angle.

61. The non-transitory, computer-readable medium of any of claims 47-60, wherein the operations further comprise: determining a hip node representing a hip of the patient, and setting a spatial coordinate of the hip node as (0,0,0), wherein spatial coordinates of other nodes are determined relative to the hip node.

62. The non-transitory, computer-readable medium of claims 47-61, wherein the anonymized representation is a 3-dimensional representation of the patient body.

63. The non-transitory, computer-readable medium of claims 47-62, wherein the one or more computers include a mobile device or a smart phone.

64. The non-transitory, computer-readable medium of claim 63, wherein the mobile device or the smart phone includes a camera used to capture the plurality of consecutive images.

65. The non-transitory, computer-readable medium of claims 47-64, wherein the operations further comprise causing the user interface to present instructions for performing the physical activity.

66. The non-transitory, computer-readable medium of claim 65, wherein the operations further comprise: comparing the movements of the one or more targeted body parts to the instructions; andin response to determining that the movements do not comply the instructions, sending further instructions to the user interface to guide the patient redo the physical activity, wherein the further instructions include suggesting particular actions to correct the movements.

67. The non-transitory, computer-readable medium of claims 47-66, wherein the stage of the medical disorder is determined based on a progress or a regress of the disorder over a period of time, the progress or the regress being determined by obtaining, from a medical history of the patient, an old set of coordinate sequences that was determined for a prior physical activity that the patient performed at a past time before performing the physical activity, the past time indicating a beginning of the period of time, and comparing the set of coordinate sequences to the old set of coordinate sequences to determine an improvement or a deterioration of a motor function of the patient, wherein the improvement is associated with the regress of the disorder, and the deterioration is associated with the progress of the disorder.

68. The non-transitory, computer-readable medium of claims 47-67, wherein the operations further comprise determining a treatment efficacy of a treatment that the patient undergoes based on the determined stage and a previously determined stage of the disorder, and based on a time passed from the previously determined stage.

69. The non-transitory, computer-readable medium of claims 47-68, wherein the operations further comprise causing the user interface to present real-time feedback about the patient’s performance of the physical activity as the patient performs the physical activity.

Images