Image-based motor function assessment

Abstract

Methods and systems are provided for assessing a patient's physical movement based on video or sequential images of the patient performing specific physical activities. The assessment can be used to determine the patient's medical disorder or stage of the disorder.

Description

Background Technology

[0001] Assessing a patient's motor function is a crucial aspect of diagnosing and monitoring a variety of neuromuscular and musculoskeletal disorders. The severity of this disorder is typically assessed intermittently in the clinic, requiring the patient to commute to the hospital / medical facility each time. Gait and physical activity are usually assessed by asking the patient to perform repetitive physical activities while the physician evaluates the patient's performance. A patient's motor function depends on the severity of the disorder. For example, in the early stages of a particular disorder, a patient may still be able to walk without support but experience instability. In later stages of the disorder, the same patient may lose the ability to walk but be able to crawl or roll until these functions are lost in even later stages of the disorder. Similarly, as the disorder progresses, a patient may gradually lose the ability to speak, eat, or use other facial muscles. Early and accurate diagnosis of the patient's disorder or stage of disorder is crucial in initiating treatment appropriate for the disorder and that particular stage. The more treatment is delayed, the faster the disorder progresses, potentially leading to limitations or preventing the patient from performing daily routines. Summary of the Invention

[0002] This disclosure includes methods and systems for detecting and / or monitoring a patient's biomarkers (i.e., biometrics) by analyzing video (or sequential images) of the patient performing specific physical activities. The patient may have a neuromuscular or musculoskeletal disorder, such as multiple sclerosis (MS), Pompe disease, metachromatic leukodystrophy (MLD), Parkinson's disease, etc. The embodiments detect or monitor the progression of the disorder by analyzing the biomarkers of the patient performing a specific activity (or a specific set of tasks). This analysis can be performed using a machine learning algorithm trained on biomarker data (e.g., muscle, joint, and / or bone movement) from a patient with a previously diagnosed disorder while performing the same task.

[0003] Some implementations include a computer-implemented method executable by a computing system. The method includes: receiving multiple consecutive images of a patient performing physical activity; transforming the images into corresponding anonymized representations of the patient's body, each anonymized representation including multiple nodes, each node representing a specific body part of the patient's body, each node being identifiable by corresponding spatial coordinates assigned to it; identifying a set of target nodes representing one or more target body parts of the patient from the multiple nodes; determining a set of coordinate sequences representing the movement of the one or more target body parts during the physical activity, each coordinate sequence in the set including spatial coordinates associated with a corresponding target node as depicted in the consecutive images and associated with the target body part; analyzing the set of coordinate sequences to determine a stage of a medical disorder suffered by the patient, the medical disorder being a neuromuscular or musculoskeletal disorder; and sending the determined stage to a user interface for presentation.

[0004] In some implementations, the physical activity is at least one of the patient’s head movement, speaking, chewing, or swallowing, and the one or more target body parts include at least one of the patient’s head, mouth, lips, or neck.

[0005] In some implementations, the physical activity includes walking. The one or more target body parts may include at least multiple portions of the patient's legs. The set of target nodes may include first leg nodes representing a portion of the patient's first leg and second leg nodes representing a portion of the patient's second leg. Analyzing the set of coordinate sequences may include: comparing a first coordinate sequence of the first leg nodes with a second coordinate sequence of the second leg nodes to obtain a signal representing gait cycles during the walking process, each gait cycle starting from a corresponding starting point when the first leg passes the second leg along the walking direction; determining the patient's foot parameters based on the gait cycles, these foot parameters including one or more of foot duration, foot count, foot frequency, or gait tone; and determining the stage of the medical impairment based on the foot parameters.

[0006] These gait parameters can include gait frequency or pace. The stage of the medical disorder can be determined based on a predetermined correlation between different stages of the disorder and different gait frequencies or paces.

[0007] In some implementations, each gait cycle ends at the corresponding point when the second leg passes the first leg along the walking direction.

[0008] In some implementations, the gait cycle may begin at the point where the first leg passes the second leg along the walking direction and end when the first leg passes the second leg again.

[0009] The method may further include determining the frontal plane of the patient's body, wherein the first leg passes the second leg when the first coordinate of the first leg point changes from a negative value to zero or a positive value in a direction perpendicular to the frontal plane. The method may further include determining whether the body activity is walking by determining that the first coordinate alternately and periodically changes between negative and positive values.

[0010] In some implementations, the first leg node represents at least the portion of the first leg between the hip and ankle. The first leg node may represent the knee, ankle, tibia, or thigh of the first leg.

[0011] In some implementations, the physical activity includes walking, and the one or more target body parts include at least a portion of the patient's arm. The set of target nodes may include an elbow node representing the elbow of the arm, an upper arm node representing the body portion between the elbow and the shoulder of the arm, and a lower arm node representing a finger of the arm or a segment between the finger and the elbow. Analyzing the set of coordinate sequences may include determining a sequence of arm angles, each arm angle sequence formed relative to the elbow node between the upper arm node and the lower arm node at a corresponding image of the walking. The stage of the medical impairment may be determined based on the analysis of the arm angle sequence.

[0012] Analysis of the arm angle sequence involves determining differences in arm angles based on variations in the sequence. The stage of the medical disorder can be determined by comparing these differences with predetermined associations between arm angle variations and different stages of the disorder.

[0013] Analysis of the arm angle sequence may include: identifying signals representing gait cycles by determining the repetition of arm angle changes over consecutive time periods during the walking process; and determining the patient's gait parameters based on the gait cycles, including at least one of gait duration, gait count, or gait frequency. The stage of the medical impairment may be determined based on these gait parameters.

[0014] Analysis of this arm angle sequence may include determining the minimum arm angle within the sequence. The stage of the medical obstacle can then be determined based on this minimum arm angle.

[0015] The stage of the medical impairment can be determined based on its progression or regression over time. Progression or regression is determined by obtaining a set of old coordinate sequences from the patient's medical history, representing previous physical activities performed by the patient at a time prior to the physical activity, the past time indicating the start of that period; and comparing this set of coordinate sequences with the old set of coordinate sequences to determine whether the patient's motor function has improved or deteriorated, wherein improvement is associated with regression of the impairment and deterioration is associated with progression of the impairment.

[0016] The method may include determining the therapeutic efficacy of treatments experienced by the patient based on a determined stage of the barrier and a previously determined stage, and based on the time elapsed since the previously determined stage.

[0017] The method may include: determining a hip node representing the patient's hip, and setting the spatial coordinates of the hip node to (0, 0, 0). The spatial coordinates of other nodes are then determined relative to this hip node.

[0018] This anonymous representation could be a 3D representation of the patient's body.

[0019] The computing system can be a mobile device or a smartphone. The mobile device or smartphone includes a camera for capturing the multiple consecutive images.

[0020] The method may include causing the user interface to present instructions for performing the physical activity. The method may further include: comparing the movement of one or more target body parts with these instructions; and, in response to determining that the movements do not conform to the instructions, sending additional instructions to the user interface to guide the patient to repeat the physical activity, wherein these additional instructions include suggesting specific movements to correct the movements.

[0021] The method may include enabling the user interface to display real-time feedback on the patient’s performance on the physical activity as the patient performs the activity.

[0022] This disclosure further provides a system for implementing the methods provided herein. The system may include one or more computers and one or more computer-readable storage devices storing operable instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations described in embodiments of the methods provided herein.

[0023] This disclosure also provides one or more non-transitory computer-readable storage media coupled to one or more processors and having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations according to embodiments of the methods provided herein.

[0024] Among other advantages, implementations may also provide one or more of the following benefits. First, some implementations provide an objective assessment of a patient's biomarkers and thus provide a more reliable determination of the impairment, impairment stage, or progression of the impairment over time. Conventionally, clinicians review a patient's biomarkers to identify potential impairments and / or impairment stages. For example, a clinician observes a patient's gait during a clinical visit to diagnose MLD or determine the severity of the impairment. This diagnosis and determination is highly subjective. Thus, one clinician may label a particular patient's impairment as severe and prescribe high-dose medication, while another clinician may label the same impairment in the same patient as mild and prescribe low-dose medication or other treatments. The implementations disclosed herein reduce subjectivity in monitoring and diagnosis by using a system that objectively monitors a patient's biomarkers extracted from the patient's video (or image) using signal processing and machine learning methods. Implementations may also provide an objective analysis of the past or future progression of the impairment by objectively monitoring and assessing the patient's biomarkers over time.

[0025] Second, some implementations offer a more convenient and accessible solution for patients compared to routine clinical assessments that require advance appointments and visits to a clinic. This is because patients using the technologies described in this disclosure can obtain assessments of their condition anywhere (e.g., at home) and anytime using their personal devices. The ease of use of a patient's personal device to provide assessments whenever the patient desires allows for the acquisition of more data points about the patient's biomarkers. More data points can make the diagnosis or monitoring of impairment more reliable. Obtaining data points can likely be done at shorter intervals compared to consecutive clinical visits, which can allow for the rapid identification of any alarming conditions that may require immediate attention.

[0026] Third, the system may be able to suggest specific treatments or send its diagnostic results to a physician to prescribe treatment based on the system's diagnosis. Due to increased reliability, treatment recommendations based on diagnosis can be tailored to the patient with a higher level of accuracy compared to general treatment plans suggested for patients with a specific disorder or stage of a disorder. Furthermore, patients can input other biomarkers (e.g., age, sex, race, chronic diseases, etc.) that the system can consider when proposing specific treatments or transmitting diagnostic data to a physician. Accordingly, treatment can be personalized based on the specificities of the patient's physical condition or lifestyle.

[0027] Fourth, the specific techniques presented in this disclosure can reduce the amount of data received from a patient's video (or continuous images). This data reduction improves processing speed compared to analyzing the patient's raw video. More specifically, some embodiments transform the patient's body image depicted in the video into an animated body representing only specific (or target) parts of the patient's body, rather than the entire body. Those specific body parts are represented by a limited number of data points, where each data point is assigned a corresponding spatial coordinate. Accordingly, the embodiments examine only a limited amount of data associated with the specific body part of interest (corresponding to the target biomarker); and even those specific body parts can only be represented by a few data points. This procedure reduces the amount of data analysis required to study biomarkers (e.g., the movement of specific muscles) if data needs to be transferred to an external device during or after data analysis, improves processing speed, and even reduces network traffic.

[0028] Fifth, in some implementations, the system can provide results in real time. Due to the reduced data volume and increased data processing efficiency, embodiments of the present invention can reduce data analysis time and even provide results in real time. Patients can simply use their personal devices to record their own videos and receive results (e.g., identified obstacles or obstacle stages) within a short period after the video is recorded (e.g., within seconds or minutes). This is more convenient for patients and reduces the stress of waiting for results compared to other methods that do not benefit from the proposed data reduction. The immediate feedback provided by these implementations encourages patients to use their devices more frequently, which can lead to more data points and therefore more reliable results, as explained above.

[0029] Sixth, in some embodiments, the inconvenience of attaching sensors to a patient can be eliminated. Since the embodiments presented in this disclosure use video or images of the patient to assess biomarkers, the patient does not need to wear any sensors. Wearing sensors can be (mentally or physically) annoying, restrict the patient's movement, and is prone to errors if the sensor is placed in the wrong part of the body. Embodiments of the present invention eliminate these disadvantages, instead offering the advantage of allowing patients to move freely as they would in their daily lives without concern for sensor placement.

[0030] Seventh, the implementations described herein offer scalability. While the implementations provide personalized diagnosis (and event-specific treatment), there are virtually no limitations on the number of patients they can serve. The artificial intelligence models used in the implementations are capable of efficiently assessing large patient groups, thereby improving clinical workflows and reducing assessment time.

[0031] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the description below. Other features and advantages of this disclosure will be apparent from the specification, drawings, and claims. Attached Figure Description

[0032] Figure 1 An example environment that can be used to implement the embodiments disclosed herein is described.

[0033] Figure 2 Example components of a system capable of analyzing a patient’s original images or videos to determine a disorder or its stage are depicted.

[0034] Figure 3 An example “gait cycle” as defined in this disclosure is depicted.

[0035] Figure 4 Example processes that can be executed according to embodiments of this disclosure are described.

[0036] Figure 5 A schematic diagram of an example computing system that can perform the methods described in this disclosure is shown.

[0037] Figure 6A and Figure 6B The example image of the patient walking is shown, along with two animations of the patient's body with different numbers of head nodes for that example image.

[0038] Figures 7A to 7C Example images of patients standing up and sitting down are shown.

[0039] Figures 8A to 8D An example image of the patient's head movement is shown.

[0040] Figure 9A and Figure 9B An example image of a patient tapping their fingers is shown.

[0041] In the accompanying drawings, the same parts refer to the same elements or steps. Detailed Implementation

[0042] The embodiments disclosed herein provide techniques for assessing a patient's physical movement (i.e., motor skills) based on videos of the patient performing specific physical activities. These techniques are performed by a computing system. The system converts images from the video into an anonymous representation of the patient's body. The anonymous representation may be a graphical representation including nodes representing specific body parts. Each node is connected to one or more adjacent nodes by corresponding lines.

[0043] To reduce the amount of data, the system selects only the anonymized representation of the specific body part of interest. Alternatively, the system may anonymize only the specific body part while skipping the rest of the body image, or represent the rest with only a few (e.g., one) nodes.

[0044] The choice or creation of animation for a specific body part depends on the physical activity the patient is required to perform. For example, in Figure 6A In the context of interest in a patient's walking ability, no nodes are assigned to facial muscles such as the lips, eyes, and nose. However, if interest is focused on eating ability, these muscles are represented with corresponding nodes, and, for example, leg muscles are skipped in the representation.

[0045] Specific physical activities may vary based on the muscles or joints being studied. For example, if the interest is in a patient's ability to walk, the specific activity could be walking, or more specifically, walking a specific number of steps or distance in a specific pattern. Patterns could include straight lines, zigzags, jumping, turning, etc.

[0046] When studying a patient's walking ability, the system aims to identify gait biomarkers. As example, gait biomarkers can be found in steps, walking speed, or frequency to determine a patient's motor skills, such as walking ability. To this end, the system determines the patient's gait cycle, where each complete step represents one gait cycle.

[0047] To determine gait cycles and extract gait biomarkers, the system can study the patient's leg and / or arm movements. For the legs, the system establishes a correlation between gait cycles and when one leg passes over the other and / or when the other leg passes over the first. For the arms, the system establishes a correlation between gait cycles and the angle (or changes in angle) at the patient's elbow during walking. The system can study or use one or both of the leg and arm movements to determine gait biomarkers.

[0048] As another example, the system can study a patient's ability to eat, speak, or make facial expressions by studying physical activities corresponding to the movement of the patient's facial muscles. Depending on the muscles of interest, the system can request the patient to perform a specific physical activity, for example, by presenting details of that physical activity on the system's screen (e.g., on a user interface). The system then records a video of the patient performing that physical activity and analyzes the video to determine the impairment or stage of impairment the patient has.

[0049] Figure 1 An example environment 100 that can be used to perform the embodiments disclosed herein is depicted. As presented, computing device 104 captures video or continuous images / videos 106 (referred to herein as “videos”) of patient 102 while the patient is performing specific physical activities. As discussed further below, device 104 can perform partial or complete analysis on the received video, wherein the analysis produces an objective diagnosis of patient 102. Device 104 may optionally send the video or a formatted version of video 108b to physician device 122 for subjective evaluation.

[0050] Although this disclosure refers to the input data (106) as “video,” the input can be a collection of continuous images captured of a patient performing physical activity (e.g., per second). In some embodiments, device 104 segments the input video into a collection of images. For example, the video can be segmented into multiple seconds (or several seconds), and an image can be captured from each segment. Using images instead of video reduces the amount of data that needs to be processed and thus increases processing speed. In some embodiments, anonymized data for each node (representing a corresponding body part) is created and / or stored over time as a sequence of a specific number of coordinates, for example, a sequence of thirty to sixty coordinates per second during a study period (which is the time period during which the patient performs physical activity).

[0051] Device 104 transforms an image of the patient's body (depicted in video 106) into a corresponding anonymous representation of the patient's body (animation 108a). Device 104 may further analyze animation 108a to determine the patient's impairment or stage of impairment, or it may send animation 108a (or at least a portion thereof) to an external backend system 112 for further analysis.

[0052] Figure 2 Example components of a system 200 are depicted that are capable of analyzing raw images or videos of a patient to diagnose or determine a patient's impairment. In some embodiments, Figure 2 Each of the components shown is a single computing device (such as...) Figure 1This is part of device 104 shown. In some embodiments, the system includes multiple computing devices or subsystems that communicate with each other to perform the tasks described below. For example, device 104 may include receiver module 212 and transformation module 214, and may send an anonymous representation of the patient's body (e.g., animation 108a or one or more portions 224a / 224b thereof) to another computing system (e.g., backend system 112) to determine obstacles or their stages.

[0053] System 200 includes a receiver module 212 that receives video or images of a patient (e.g., 102). The images or video are captured by a camera (e.g., camera 202). Camera 202 may be part of the same system 200 or external to the system and in communication with receiver module 212. For example, both camera 202 and receiver module 212 may be personal devices of the patient (e.g., Figure 1 Part of the device 104 shown.

[0054] Receiver module 212 can store or format video / images in preparation for analysis by the rest of the system. For example, the receiver module can clean the video to remove noise or crop it and save only the portions depicting the patient performing specific activities. Receiver module 212 sends the received, stored, and / or formatted video / images to transformation module 214.

[0055] Transformation module 214 detects the patient's body in video / image 106 and anonymizes the body into animation 108a. In some embodiments, transformation module 214 can extract images of the patient's body from the video / image by removing the background or any depicted objects from each image (of the video). Thus, the animation only includes a representation of the patient's body moving in empty space. By doing so, the transformation module reduces the amount of data that needs to be processed in each image (of the video segment).

[0056] Transformation module 214 creates a graphical animation comprising multiple nodes connected to their neighboring nodes by edges. Each node represents a specific part or limb of the patient's body. Depending on how much detail may be required, nodes may be closer or farther apart. For example, for facial muscle analysis, the lower lip might be assigned to multiple nodes that are close to each other. However, for leg analysis during walking, each leg might be assigned to a corresponding node that is much farther apart than the node used for the lower lip in facial muscle analysis. While a greater number of nodes provides more granular detail about the movement of specific muscles in the patient, fewer nodes are used, and less data needs to be analyzed for obstacle / stage determination. For example, for walking, as few as one node might be assigned to each leg to determine the gait cycle described in this disclosure.

[0057] Transformation module 214 assigns corresponding coordinates to each node in the animation. In some implementations, transformation module 214 sets a point on a specific body part (e.g., a point on the hip) as the origin (i.e., coordinates (0, 0, 0)) and assigns corresponding coordinates to these nodes based on their respective positions relative to the origin of other nodes in animation 108a. During patient movement, the coordinates of the origin remain constant (0, 0, 0), while other points can change their coordinates as they move relative to the origin. By fixing the origin to a specific part of the patient's body, transformation module 214 creates an animation (108a) independent of the patient's background or surrounding environment depicted in image / video 106.

[0058] Transformation module 214 sends animation 108a to target node identifier 122. Target node identifier 122, coordinate sequence identifier 226, and obstacle determiner 228 may be part of machine learning module 240 and together perform a machine learning model on the animation 108a received from transformation module 214. The machine learning model evaluates the animation to determine the obstacle or stage of the obstacle suffered by the patient.

[0059] Target node identifier 122 identifies portions of animation 108a of interest for identifying an obstacle or its phase (e.g., 224a, 224b). For example, if a particular physical activity is walking (e.g., strolling), then body parts that move periodically during the walk (such as legs and / or arms) are target body parts. Therefore, transformation module 214 identifies nodes associated with the patient's legs (e.g., animation segment 224a) and / or nodes associated with the patient's arms (e.g., animation segment 224b) from the overall body animation 108a. By removing the remainder of animation 108a, target node identifier 222 helps to further reduce the amount of data to be analyzed and thus improves processing speed.

[0060] The target node identifier 222 can create a corresponding animation segment for each body part, or it can include multiple body segments into the same animation segment. For example, although in Figure 2 Different animation segments 224a and 224b for the arms and legs are shown, but the animation of the arms and legs can be included in a single animation segment (not shown). Separating the animation segment for each identified body part allows for individual analysis of each body part. However, similar separation can later be performed, for example, by coordinate sequence identifier 226 or obstacle determiner 228, to study the coordinates of nodes associated with different body parts (e.g., legs and arms) separately.

[0061] Target node identifier 122 can identify target body parts based on default instructions previously provided to the identifier or to a storage device, from which the identifier retrieves operational instructions. For example, for a patient suspected of having a specific impairment, impairment determiner 228 can send information about muscle strength and mobility indicating target body parts at different stages of the specific impairment. As another example, the storage device can pre-store correlations between the degrees of freedom of movement of different body parts and corresponding impairments, and can provide all or part of this information to target node identifier 122. Information can be sent to target node identifiers based on the impairment the patient has or is suspected of having. Target node identifier 122 uses this information to select target nodes (or animation segments) associated with the target body parts included in the information. Alternatively or additionally, target node identifier 122 can communicate with impairment determiner 228 to receive information about which body parts or muscles are being used to analyze the patient's motor function.

[0062] Target node identifier 122 sends the identified target animation segments (e.g., 224a, 224b) to coordinate sequence identifier 226. Coordinate sequence identifier 226 determines a set of coordinate sequences representing the motion of one or more target body parts during body movement. Each coordinate sequence in this set includes the coordinates of a corresponding target node in the image (of video) received from receiver module 212. For example, a sequence associated with a specific node can be presented as a corresponding... A matrix, where each row of the matrix shows the coordinates of a specific node in the corresponding image, wherein N consecutive images (of the video) are being studied. Target node identifier 122 sends the identified (or determined) sequence to obstacle determiner 228.

[0063] The obstacle determiner 228 analyzes the set of coordinate sequences to determine the obstacle (or its stage) suffered by the patient. The obstacle determiner module may include multiple sub-modules, each capable of analyzing movements associated with a corresponding physical activity. For example, Figure 2 The obstacle determiner 228 shown has sub-modules: a walk analyzer 230 and a facial motion analyzer 238. The walk analyzer 230 analyzes images (or videos) associated with walking, such as walking, running, or jumping, and the facial motion analyzer 238 analyzes images associated with facial muscle movements, such as speaking, eating, or expressing emotions, such as laughing, frowning, blinking, etc.

[0064] Based on the target body parts, the corresponding submodules associated with those body parts analyze the coordinate sequences associated with those body parts from all sequences received from the coordinate sequence identifier 226. If multiple body parts are being studied, one or more submodules associated with those body parts can, for example, operate in parallel. For example, if both a patient's gait (e.g., walking) ability and speaking ability are being studied, each of the submodules gait analyzer 230 and facial motion analyzer 238 analyzes the coordinate sequences associated with the target body parts being evaluated by the submodule (the target body parts are, for example, the legs and / or arms for gait analyzer 230, and the jaw and / or lips for facial motion analyzer 238). As described above, the target body parts are identified based on the obstacles being examined.

[0065] The gait analyzer may include additional submodules, each examining a specific subset of coordinate sequences associated with a particular target body part. For example, gait analyzer 230 includes a hip position detector 232, a leg position detector 234, and an arm position detector 236. Hip position detector 232 studies the coordinate sequence associated with the patient's hip. Leg position detector 234 analyzes the coordinate sequence associated with the patient's leg animation segment 224a. Arm position detector 236 studies the coordinate sequence associated with the patient's arm segment 224b. Gait analyzer 230 may use all or only a few of these submodules to determine the patient's gait cycle. For example, gait analyzer 230 may determine the gait cycle based solely on the patient's leg movements (i.e., using only leg position detector 234), or based on the coordination between the patient's leg and arm movements (i.e., coordinating the use of both leg position detector 234 and arm position detector 236).

[0066] To determine the gait cycle, leg position detector 234 identifies at least a first leg node assigned to a portion of the patient's first leg and at least a second leg node assigned to a portion of the patient's second leg. Each of the first and second leg nodes can be assigned to any portion of the corresponding leg from the patient's hip to the toes of that leg. For example, the first leg node can be assigned to any or some of the patient's left thigh, knee (e.g., node 244), calf, tibia, ankle (e.g., node 246), heel, toes, etc., and the second leg node can be assigned to any or some of the patient's right thigh, knee (e.g., node 248), calf, tibia, ankle, heel, toes, etc. In some embodiments, the first and second leg nodes are assigned to similar leg portions of the corresponding legs; for example, both are assigned to the corresponding knee, or both are assigned to the corresponding ankle. However, assignment to similar leg portions is not required.

[0067] In some implementations, the gait cycle is foot-specific and is defined as the time interval between the first leg passing the second leg along the walking direction (e.g., the direction of travel) and the second leg passing the first leg along the walking direction, ending at the end point. Figure 3 An example foot-specific gait cycle 306 is shown. As can be seen, the cycle begins when the first leg passes the other leg (at point 302) and ends when the other leg passes the first leg (at point 304).

[0068] In some implementations, gait cycle is defined as the time between consecutive passes of two legs. For example, Figure 3 A gait cycle 308 used in this embodiment is shown. Compared to a foot-specific gait cycle 306, a bipedal gait cycle 308 takes into account asymmetrical leg movements and is more likely to provide repetitive foot patterns. For example, if one foot drags more than the other, or one foot moves slower than the other, the gait cycle 308 includes two consecutive steps taken by both feet in one cycle, while the gait cycle 306 provides a corresponding cycle for each step / foot.

[0069] Compared to the conventionally defined gait cycle (which is defined as the time interval between consecutive moments when the heel touches the ground), both foot-specific gait cycles and bipedal gait cycles account for a wider range of foot / leg positions during walking. For example, if a patient drags one or both feet on the ground (which is not uncommon in patients with neurological or musculoskeletal disorders), the heel will remain on the ground for a large (e.g., primary) portion of the step. However, in the specific embodiments discussed here, the position of the heel relative to the ground does not limit the identification of gait cycles nor introduce errors into them.

[0070] Back Figure 2 The leg position detector 234 determines gait cycles by comparing a first coordinate sequence of a first leg node with a second coordinate sequence of a second leg node to determine when one leg passes over the other during the patient's walk. The leg position detector 234 can determine a first step gait cycle by assigning a first time point as the start of the cycle and by assigning a second time point as the end of the cycle. For similar... Figure 3 The step-specific gait cycle 306 shown has a first time point when the first leg passes the second leg along the walking direction, and a second time point when the second leg passes the first leg along the walking direction. The leg position detector 234 can similarly create a second step gait cycle during the time interval between the second leg passing the first leg and the subsequent first leg passing the second leg.

[0071] Alternatively, the leg position detector 234 can determine and use a bipedal gait cycle, similar to... Figure 3The gait cycle 308 is shown in the diagram. In such a scenario, the gait cycle would be the time interval between (i) the first leg passing the second leg and (ii) the next time the patient's first leg passes the second leg.

[0072] The leg position detector 234 determines the transit point (i.e., the time point at which one leg passes over the other) by comparing a first sequence of coordinates associated with one (or more) first leg nodes representing the first leg with second coordinates associated with one (or more) corresponding second leg nodes representing the second leg. As previously discussed, each coordinate sequence is associated with a corresponding node, and each coordinate in the coordinate sequence of a particular node represents the corresponding spatial location of the particular node obtained from the corresponding image of the patient (video).

[0073] The leg position detector 234 determines the time when a first coordinate (associated with a first leg node) changes from a negative value (or coordinate) to a positive value (or coordinate) relative to a second coordinate (associated with a second node) in the walking direction as a transit point. The detector 234 identifies changes in the corresponding coordinate sequences of the first and second nodes presented in multiple consecutive images (e.g., in a video). For example, assuming the walking direction is assigned to direction X in XYZ space, when the first coordinate changes from (2, 5, 0) to (6, 5, 0) in two consecutive images, its second coordinate changes from a negative value (or coordinate) in the walking direction X and relative to a second coordinate fixed at (4, 2, 0) in the two consecutive images.

[0074] In some implementations, the relative change from a non-positive value to a positive value (or coordinate) in the walking direction is considered as the time it takes for one leg to pass the other. For example, when a first coordinate changes from (2, 5, 0) to (4, 5, 0) in two consecutive images, the first coordinate changes from a non-positive value to a positive value (or coordinate) in the walking direction X relative to a second coordinate fixed at (2, 2, 0) in the two consecutive images. These implementations explain the gait cycle associated with footwork patterns where one leg is not behind the other, for example, due to difficulty for the patient to move the rear leg.

[0075] In some embodiments, the leg position detector 234 determines the gait cycle based on the frontal plane of the patient's body. The leg position detector 234 can determine the frontal plane of the patient's body and determine the transit points of each leg relative to the other leg based on the position of the legs relative to the frontal plane. The leg position detector 234 can define the frontal plane as a plane perpendicular to the ground that divides the patient's body into a posterior and anterior portion. The posterior portion includes the patient's back and chest. In these embodiments, the leg position detector 234 determines the transit point as the time when the first leg passes the second leg or the time when the second leg passes the first leg in a direction perpendicular to the frontal plane; meaning when the first coordinate of the first leg node changes from a negative value to zero or a positive value in a direction perpendicular to the frontal plane, or when the second coordinate of the second leg node changes from a negative value or zero to a positive value in a direction perpendicular to the frontal plane. In some embodiments, the leg position detector 234 determines the offset of the right leg relative to the left leg and defines the frontal plane such that the offset is at the origin or on a line passing through the origin of the plane.

[0076] Once the gait cycle is determined, the walk analyzer 230 divides the images into multiple gait-specific image sets and analyzes the patient's gait parameters based on these sets. Gait parameters can be gait-specific parameters, extracted from images in each set, or they can be general gait parameters, determined from studying multiple (e.g., all) sets. Gait-specific parameters can be gait-specific. Examples of gait-specific parameters include the movement of one leg relative to another, stride width of each leg, activity of each leg, stability of each leg, gait duration, etc. General gait parameters can be correlated with the patient's overall walk. Examples of general gait parameters include step count, step frequency, walk speed, etc.

[0077] As discussed earlier, the physical activity used for obstacle (stage) determination can be predefined based on the obstacle, and the obstacle determiner 228 can use sub-modules (e.g., 230) associated with that activity to analyze the patient's motor skills. However, the obstacle may not be predefined. In such an implementation, the obstacle determiner 228 may need to analyze nodes to determine what activity the patient is performing before focusing on the details of that activity.

[0078] In some implementations, obstacle determiner 228 uses the position of the leg nodes relative to the frontal plane to determine that walking is the activity of interest. For example, obstacle determiner 228 can determine that a particular physical activity is “walking” by determining that a first coordinate associated with the first leg node (of the first leg) and a second coordinate associated with the second leg node (of the second leg) change alternately and periodically relative to and perpendicular to the frontal plane.

[0079] In addition to or as an alternative to using leg nodes, the gait determiner 228 may use arm nodes associated with one or both of the patient's arms to determine gait cycles and / or identify obstacles or their stages. The gait determiner 228 uses an arm position detector 236 to make this determination.

[0080] Arm position detector 236 receives coordinate sequences associated with arm nodes from coordinate identifier 226. Coordinate sequence identifier 226 receives animation segments (e.g., 224b) that represent the spatial positions of one or both arms of the patient represented by corresponding arm nodes included in the animation segments. Coordinate sequence identifier 226 generates a set of corresponding coordinate sequences representing nodes in a series of images for those arm nodes.

[0081] An animation segment of the arm can have any number of supporting arm nodes. In a preferred embodiment, the animation segment will include at least three arm nodes: (i) an elbow node representing the elbow of the arm, (ii) an upper arm node representing the upper part of the body above the elbow, such as the shoulder or the segment between the elbow and the shoulder on the arm, and (iii) a lower arm node representing the fingers of the arm or the segment between the fingers and the elbow.

[0082] Arm position detector 236 analyzes the set of coordinate sequences it has received from coordinate sequence identifier 226 to determine the sequence of arm angles formed (for the corresponding arm) during the walk activity. Each arm angle is formed between the upper arm node and the lower arm node at the corresponding time during the activity relative to the elbow node. Walk determiner 228 (or arm position detector 236) determines obstacles or their phases based on the analysis of the arm angle sequence.

[0083] The gait determiner 228 (or arm position detector 236) can determine the obstacle or stage of the obstacle based on the minimum arm angle that occurs during gait. There is a correlation between gait (e.g., walking) difficulty and arm angle. Patients with difficulty walking smoothly typically tense their arm muscles at smaller angles compared to people who have no difficulty walking smoothly.

[0084] Figure 6A and Figure 6B An example image (i.e., snapshot) 602 of a patient with MLD walking is shown. As shown, the patient bends his arm to generate stability in his body posture during walking.

[0085] Figure 6AAnimation 604 shows the patient's body as depicted in image 602. Arm angles 606 and 608 are depicted in the animation. As a supplement to or alternative to the arm angles discussed above, the positions of other body parts can be investigated. For example, in addition to arm angles 606 and 608, one or two shoulder joint angles 610 and 612 can be investigated. Shoulder joint angles are the angles formed between the patient's arm and torso. The torso can be represented by as few as two nodes representing the corresponding shoulder and two nodes representing the hip side, and is further represented by connecting the corresponding nodes to form the sides of the torso.

[0086] As another example, changes in head angle relative to the shoulders can be studied as a representation of a patient's motor function. In an embodiment analyzing walking, the head can be represented by as few as one or two nodes, and the head angle can be determined based on the position of those nodes relative to shoulder nodes or relative to the torso. Of course, more complex studies of the head are also possible, for example, where the head is represented by a larger number of nodes (e.g., more than 100 nodes), which can produce a three-dimensional (3D) representation of the head that can include details of some parts of the head, such as the jaw, lips, eyebrows, etc.

[0087] exist Figure 6A In the diagram, the head is represented by two lines, each connecting (i) the jaw (or ear) node and (ii) the neck point of the patient's neck. The head angle 614 changes as the patient tilts their head during walking. Studies of head angle can provide information about a patient's ability to stabilize their head during walking, or can indicate a patient's confidence in walking in a relaxed or tense manner.

[0088] Figure 6B An animation 620 shows the patient's body in the same image 602. Figure 6B In the image, the head is represented by a mesh of over 400 nodes. This allows for a 3D representation of the patient's head during walking. Figure 6A Compared to the tilt representation shown (e.g., angle 614), the 3D representation provides more data about the angle of the patient's head in 3D space (e.g., 624).

[0089] As described above, the walk determiner 228 (or arm position detector 236) determines obstacles or their phases based on the analysis of an arm angle sequence. In some embodiments, the analysis of the arm angle sequence includes determining the minimum arm angle in the sequence and drawing conclusions about phases in a specific sequence based on the minimum arm angle. In some embodiments, the arm position detector 236 uses the angle that always occurs during the walk activity as the minimum arm angle and discards arm angles that occur less frequently (e.g., less than a quarter of the steps taken during the activity) or even smaller. For example, the analysis includes determining when the minimum arm angle occurs, and if the minimum arm angle occurs sporadically throughout the walk activity (e.g., within a third of the total steps), searching for the next minimum arm angle that occurs more frequently (or always) throughout the activity and using that always-occurring angle as the minimum arm angle.

[0090] In some implementations, the analysis of the arm angle sequence includes determining differences in arm angles based on angle changes during a walking activity. For example, a walking determiner 228 can determine an obstacle or its stage by comparing this difference with a predetermined correlation between arm angle differences and different stages of an obstacle. A greater difference in arm angles during a walking activity indicates better patient stability. In fact, there is a correlation between arm angle differences and a patient's ability to take larger steps; and larger steps indicate more control over the muscles involved in the walking activity, which may imply a lower stage of a particular obstacle or the absence of that obstacle in the patient.

[0091] While arm angles are described in detail in this disclosure, other parts of the arm or the patient's body in general can be studied in a similar manner. For example, obstacle determiner 228 may include other position detector submodules (such as wrist position detectors, finger position detectors, head position detectors, knee position detectors, hip position detectors, etc.) that can operate as a complement to or part of arm position detector 236 to determine the position of corresponding body parts (e.g., wrist, fingers, head, knee, hip, etc.) at different stages of the study (e.g., governed by the study protocol). Similar to arm angles described above, each of these submodules can determine the position of a corresponding body part based on a corresponding angle (e.g., wrist angle or finger angle) as the patient performs the study protocol.

[0092] Figures 7A to 7C Another example is shown where the activities of interest include standing up from a sitting position and sitting down from a standing position. The system (e.g., Figure 2The system 200 can identify stages of a patient's specific medical impairment based on sequences of standing and sitting movements, such as the time it takes for the patient to stand up, the time it takes for the patient to sit down, or both. The system can also combine sitting / standing activities with other activities (e.g., walking activities) to account for the movement and dexterity of more body parts. For example, the system can study the time it takes for a patient to rise from a sitting position, stand, walk a certain number of steps (e.g., ten steps), turn around, walk backward, stand again, and sit down where the patient began the activity (e.g., timed stand-up walking (TUG)).

[0093] exist Figures 7A to 7C In the illustrated example, the system detects a patient's sitting posture based on the angles of the patient's joints (e.g., hip angle, knee angle, arm angle, etc.). The hip angle is formed between a first vector (e.g., 702) connecting the hip node to the shoulder node and a second vector (e.g., 704) connecting the hip node to the knee node. The knee angle is formed between the second vector (e.g., 704) connecting the knee node to the hip node and a third vector (e.g., 706) connecting the knee node to the foot node (e.g., ankle node). The system can assign a corresponding hip angle and a corresponding knee angle to each side of the patient (i.e., left and right sides).

[0094] In some embodiments, the system identifies a sitting posture when at least one hip angle and at least one knee angle are approximately 90 degrees. Angle proximity can be determined based on a specific angle range preset for the corresponding angles in a given study (e.g., depending on the study protocol). For example, for the purpose of posture determination, a hip angle between 60 and 120 degrees is considered approximately 90 degrees, while a knee angle between 70 and 110 degrees is considered approximately 90 degrees.

[0095] Similarly, in some embodiments, the system identifies a standing posture when at least one hip angle and at least one knee angle are approximately 180 degrees. Likewise, angular proximity can be associated with the corresponding angles, for example, a hip angle within 30 degrees and a knee angle within 20 degrees.

[0096] In some examples, a patient is considered sitting when at least one hip angle is less than a pre-specified sitting hip angle. In some examples, at least one knee angle also needs to be less than a pre-specified sitting knee angle to identify the patient as sitting. The sitting hip angle and sitting knee angle can be different from each other. For example, the sitting hip angle can be preset to 120 degrees, while the sitting knee angle can be preset to 110 degrees.

[0097] Similarly, in some embodiments, the system identifies a standing posture when at least one hip angle is greater than a pre-specified standing hip angle. In some examples, at least one knee angle also needs to be greater than a pre-specified standing knee angle to identify the patient as standing. The standing hip angle and the standing knee angle can be different from each other. For example, the standing hip angle can be preset to 150 degrees, while the standing knee angle can be preset to 170 degrees.

[0098] Figure 7A The figure shows a patient in an initial sitting position. It also depicts a hip signal 708 tracking the hip angle and a knee signal 710 tracking the knee angle. Figure 7B The image shows the patient standing. The hip signal 708 tracks changes in the hip angle and is shown in... Figure 7B The moment depicted is when the hip angle is close to 180 degrees (e.g., within 30 degrees of difference). The knee signal 710 tracks changes in the knee angle and shows... Figure 7B At the moment depicted, the knee angle is close to 180 degrees (e.g., within 10 degrees of difference).

[0099] In some embodiments, the system identifies a standing time point when the hip angle is at its maximum value in the hip signal 708. In some embodiments, the system identifies a standing time point when the patient has reached a hip angle close to 180 degrees and has maintained the hip angle within a certain angular margin (e.g., within 10 degrees) for a pre-specified period of time. In such embodiments, the patient must stand for a pre-specified period of time (e.g., at least 5 seconds) before proceeding to the next activity (e.g., walking or sitting), allowing the system to detect appropriate standing time and posture.

[0100] Figure 7C The image shows the patient sitting down. As depicted, hip signal 708 and knee signal 710 show the decrease in hip and knee angles, respectively, as the patient moves from a standing to a sitting position.

[0101] Depending on the angle at which the camera captured the patient's image, other sides of the patient's body or other body parts can be used in this study. For example, Figures 7A to 7C Focusing on the patient's right side (due to the camera angle), measurements from the patient's left side can also be used, for example, to confirm the identified body position. For instance, in Figure 7C During the test, the camera can depict the left side of the patient's body, and the system can generate corresponding signals for the left hip and left knee. Angles of other body parts (such as the arms) can be used as additional indicators of the patient's sitting and / or standing posture.

[0102] Figures 8A to 8DAnother example is shown where the activity of interest is head movement. The system can determine a patient's head control ability based on predefined head movement activities instructing the patient to perform. These predefined head movement activities may include (but are not limited to) left / right turns, up / down nodding, and head tilting. Based on the extent of the movement activities the patient can perform, the system can determine the stage of a specific impairment in the patient. The extent of movement activities may include (but is not limited to) the range of head movement angles, head movement speed, continuity or variability of movement (e.g., smoothness versus interruption), etc.

[0103] exist Figure 8A In this process, the image of the patient's face (802) is anonymized (804), and a corresponding node is selected from a larger number of nodes included in the anonymized representation of the head. The system uses the selected node to measure the corresponding motion angles and / or statistics associated with head movements.

[0104] Figure 8A Image 806 includes three measurement indicators (812, 814, 816), each indicating head position with respect to a specific head movement. Measurement indicator 812 indicates left / right head turn. Measurement indicator 814 indicates head nodding up / down. Measurement indicator 816 indicates head tilt.

[0105] Figure 8B The measurement indicator 812 is shown to change as the patient turns his head to the left and right. The measurement indicator 812 indicates the extent to which the patient can turn his head about (e.g., from the head to the spine) a yaw (or vertical) axis across the patient's body. The left / right range depicted by the measurement indicator 812 (-61 to 56 in this example) indicates the extent of the patient's ability to turn his head to the left and right. In some examples, the system may study each of the leftward and rightward movements separately to identify areas where the patient may be unable to turn (e.g., unable to turn to the left).

[0106] Figure 8C The diagram illustrates the change in measurement indicator 814 as the patient nods up and down. Measurement indicator 814 indicates the extent to which the patient can move their head up and down about (e.g., from one ear to the other) a pitch (or forehead) axis across the head. The up / down range depicted by measurement indicator 814 (here -33 to 13) indicates the extent to which the patient can move their head up and down. In some examples, the system may study each of the up and down movements individually to identify situations where the patient may be unable to nod in a particular direction (e.g., unable to move their head down).

[0107] Figure 8DThe change in measurement indicator 816 is shown when the patient tilts their head. Measurement indicator 816 indicates the degree to which the patient can tilt their head left and right about a roll axis perpendicular to the aforementioned yaw and pitch axes. The tilt range depicted by measurement indicator 816 (here -16 to 15) indicates the degree of the patient's ability to tilt their head. In some examples, the system may study each of left and right tilts separately to identify areas where the patient may be unable to tilt in a particular direction (e.g., unable to tilt their chin to the left).

[0108] The system determines the patient's stage of impairment based on the measured head control ability indicated by measurement indicators 812, 814, and 816. In some implementations, the system compares the measurements with a history of the patient's measurements to determine the progression or improvement of the patient's impairment over time and / or to determine the efficacy of treatments used by the patient across multiple (e.g., consecutive) measurements.

[0109] In some implementations, the system compares a patient's measurements to patient groups, each group associated with a corresponding stage of disability for the patient, to determine the patient's stage of disability. In examples, each group may be associated with a corresponding metric, such as a widely accepted metric from the Extended Disability Status Scale (EDSS). For example, an EDSS ranging from zero to ten (where zero indicates no disability and ten indicates the highest level of disability) could be divided into four groups. The system can compare a patient's measurements to the EDSS range within each group and indicate how severe or progressive the patient's disability is compared to the disabilities of other patients in that group. In some examples, one or more groups also refer to corresponding treatments (e.g., corresponding research treatments). The system can identify those treatments for the patient based on the comparison.

[0110] Figure 9A and Figure 9B Another example is shown, where the activity of interest is finger tapping. Measuring a patient's finger tapping ability is useful in the diagnosis and identification of specific disorders, such as Parkinson's disease.

[0111] This system can study the finger-tapping ability of any one hand or both hands. In the depicted... Figure 9A In the example, the system has the left hand of an anonymous patient, and Figure 9B In the system, the patient's right hand is anonymized. The system also identifies specific nodes associated with the index finger and thumb of each hand in the corresponding diagram. The system uses the identified nodes to track the patient's finger-tapping ability. To do this, over time and as the patient continues to tap his index finger against his thumb, the system tracks the distance between specific index finger nodes (e.g., nodes representing the tip of the index finger) and specific thumb nodes (e.g., nodes representing the tip of the thumb).

[0112] The system uses statistics on tapping time to determine a patient’s finger tapping ability. For example, the system can use statistics such as mean, standard deviation, etc., on the time elapsed between two consecutive taps or between the minimum distance between two fingers (which represents a tap) and the maximum distance between two fingers (which indicates the farthest position of the two fingers relative to each other before the next tap occurs).

[0113] The system can study each hand individually or compare the abilities of both hands, for example, by normalizing abilities based on the patient's individual coordination. Similar to other examples discussed above, the system can compare a patient's tapping measurements with the patient's tapping measurement history or with patient groups at different stages of impairment.

[0114] After the system identifies the obstacle (and / or its stage), the system sends the results to the transmission / presentation module 242 for presentation to the patient and / or to the patient's healthcare provider. The transmission / presentation module 242 may be associated with a display that shows the results. For example, the module may include a screen, or the results may be transmitted to a device having a screen for data presentation.

[0115] As described above, system 200 can be located in device 104 ( Figure 1 The data may be located on device 104 and partially reside on backend system 112. For example, receiver module 212 and transformation module 214 may be on device 104, and ML module 240 may run on backend system 112. In such a scenario, backend system 112 may transmit the results back to user device 104, for example, to transmission / presentation module 242 for presentation to patient 102. Generally, data can be transmitted between device 104 and backend system 242 via a wired or wireless communication link. To protect patient privacy, in some embodiments, data transmission includes only animation 108a without specifying any patient identifier.

[0116] In some implementations, device 104 analyzes video and determines obstacle stages independently of backend system 112. In other implementations, device 104 confirms its determination through backend system 112. Alternatively or additionally, device 104 may periodically receive updates to the model or algorithm from system 112, or system 112 may send such updates to device 104 when available.

[0117] Because backend system 112 supports multiple devices (such as device 104 associated with different patients), device 104 can benefit from improvements made by system 112 to models or algorithms (which each device can use to determine barriers / stages) based on data dynamically received from those devices over time. For example, backend system 112 can modify or update biomarkers used to determine barriers / stages based on data received from those devices and communications with physician devices (e.g., 122) that confirm or modify the determined barriers / stages.

[0118] In some implementations, device 104 sends video 108b (indicating whether system 200 has determined or not determined a barrier stage) to physician device 122. Physicians can use this video to make their own independent determination of the barrier stage. Backend system 112 can receive the physician's diagnosis from physician device 122 and use this information to modify its barrier determiner model or algorithm.

[0119] In some implementations, device 104 sends identified obstacles from the system to physician device 122 to help a physician identify the obstacles or prescribe treatment or medication for a patient. The physician (or generally, a healthcare provider) can approve, modify, or reject the system's determination. The physician device sends this information to system 200, and system 200 uses this information to improve its models or algorithms.

[0120] In some implementations, the analysis of the arm angle sequence includes determining signals representing gait cycles by identifying repetitions of arm angles over consecutive time periods during walking, and determining the patient's foot parameters based on the gait cycles. The obstacle determiner 228, or any of the leg position detector 234 and arm position detector 236, individually or in relation to each other, can determine an obstacle or its stage based on foot parameters obtained from studying the gait cycles. For example, the obstacle determiner 228 can analyze the patient's ability to move any muscles involved in walking (e.g., either arm, either leg, hip muscles, etc.) during each gait cycle to determine recurring, persistent muscle exhaustion or limitation during walking, and draw conclusions about the stage of a particular obstacle based on that exhaustion or limitation. Foot parameters may include foot duration, foot count, foot frequency, gait, stride length, specific joint flexion (based on the resulting angle), etc.

[0121] In some implementations, obstacle determiner 228 determines obstacle stages based on predetermined associations between different obstacle stages and different values ​​of one or more specific gait parameters. For example, obstacle determiner 228 may retrieve predetermined associations (e.g., from a storage device) and compare the determined patient's gait parameters to the predetermined associations to indicate obstacle stages. The predetermined associations may be in tabular form, mapping each obstacle stage to gait parameters (e.g., gait frequency range) of patients with biological biomarkers similar to the patient's biological biomarkers (e.g., age, sex, weight, height, other chronic diseases, etc.). In this way, gait parameters (e.g., gait frequency, gait tone) can help quantify obstacle stages.

[0122] In some implementations, the obstacle determiner 228 determines the stage of a medical obstacle based on the progression or regression of the obstacle according to the patient's medical history. In other words, the obstacle determiner 228 may determine the stage of an obstacle based on the progression or regression of the patient's obstacle over time in one or more walking parameters. For example, if the patient was previously diagnosed (by system 200 or by a medical practitioner) with a specific stage of the obstacle, the obstacle determiner 228 can use that history to determine how fast or slow the patient is now walking compared to the time of the previous diagnosis, and to provide an estimate of obstacle progression or improvement in the patient's obstacle.

[0123] The amount of time elapsed can be a factor in these determinations. For example, compared to a second patient whose previous diagnosis was made six months ago and who showed a one-tenth increase in gait frequency, obstacle determiner 228 can determine that a first patient whose previous diagnosis was made five years ago and whose gait frequency has increased by one-tenth since then has improved less or more slowly.

[0124] In some implementations, the obstacle determiner 228 can determine the therapeutic efficacy of treatment experienced by the patient based on obstacle stages determined at different times. For example, based on obstacle stages determined in multiple rounds (e.g., two rounds) of diagnoses and the time periods elapsed between multiple diagnoses, the obstacle determiner 228 can determine the effectiveness or ineffectiveness of treatment in improving the patient's obstacle or halting obstacle progression. The obstacle determiner 228 can, for example, compare the patient's obstacle progression to the average obstacle progression of (other) patients with the same obstacle over similar time periods to determine the patient's therapeutic efficacy. Two time periods are considered "similar" if they differ by a specific amount, such as one-tenth of the total time period. For example, when the specific time difference is set to one-tenth, this indicates that one-tenth of ten months, a nine-month time period, is similar to a ten-month time period.

[0125] As discussed above, the coordinates of each node can be determined relative to a specific node representing a specific part of the patient's body. This specific part could be the patient's hip or chest, and the specific node could be a hip node or a chest node assigned to the hip or chest, respectively. For example, transformation module 214 could set the hip node to coordinates (0, 0, 0) and set the coordinates of every other node based on the spatial position of that node relative to the hip node. In some examples, the specific node could be the center of mass of the patient's body.

[0126] The anonymized representations disclosed in this disclosure (e.g., animation 108a) can be a three-dimensional (3D) representation of the patient's body. 3D representations offer more degrees of freedom in the camera's angle relative to the patient. Consequently, the patient has greater freedom to walk or move their muscles in different directions without having to be directly in front of the camera (a limitation of 2D representations).

[0127] As discussed above, the implementations provided in this disclosure offer patients the convenience of using their personal devices to assess their health. For example, in Figure 1 In this context, patient 102 can use the camera on their smartphone or mobile device (e.g., device 104) to capture video or a series of images of the patient while the patient is walking.

[0128] Software or applications can be downloaded to a personal device to analyze videos as described above. Alternatively or additionally, the device can send images or videos to an external device (e.g., to system 112) for analysis, to verify the output of the personal device's analysis, and / or to update the models or algorithms used for analysis.

[0129] In some implementations, the personal device may also provide instructions to the patient regarding physical activities. For example, an application may have been downloaded to the personal device that facilitates communication with an external server or system (e.g., 112) to provide the patient with instructions on what activities to perform for different impairments. These instructions are presented on the user interface of the personal device.

[0130] The application can also specify the type of activity based on the patient's health or medical history (e.g., based on the patient's previous diagnosis). For example, for a patient with an early stage of MLD (e.g., stage 1), the application can identify "walking" as the activity to be performed, while for a patient with a later stage of MLD (e.g., stage 4), the application can instruct the patient to perform activities based on "eating" or "talking".

[0131] The application can also provide details of the activity to be performed. For example, for walking activities, the application can specify the number of steps, walking direction, walking duration, etc. For speaking activities, the application can specify the specific words or phrases to be spoken, speaking speed, etc.

[0132] If a patient does not follow the instructions provided by the application, the application can also guide the patient to correct the activity already performed. For example, the personal device can compare the movement of one or more target body parts (e.g., the legs in a walking activity) with the instructions, and in response to determining that the movement does not conform to the instructions, present the patient with further instructions to guide the patient to try the activity again with suggested corrections. Additional instructions may include suggesting specific actions to correct the movement. For example, it may include increasing the number of steps if the patient is not walking far enough, or increasing the walking speed if the patient is walking slower than expected, or repeating a specific phrase that the patient missed or did not say.

[0133] In some implementations, the application (e.g., a smartphone application) has the primary responsibility for acquiring and anonymizing the video. In some implementations, the application may act as an implementer and / or real-time (e.g., at-home) guide for a research protocol designed for the patient. The research protocol includes a set of rules designed for the patient to perform, enabling the diagnosis of the patient's stage of impairment. The research protocol may be designed by a medical practitioner or system 200. For example, system 200 may identify / determine a specific research protocol for a patient from a predetermined set of research protocols stored at (or accessible by) system 200. The patient's research protocol is determined based on the patient's biomedical biomarkers and / or medical history.

[0134] In the example, the application initially performs real-time analysis of a patient's biomarkers, for instance, by analyzing the patient's motor function, and then guides the patient to perform tasks included in a specific research protocol determined for the patient. The application can determine the specific research protocol based on real-time analysis of video.

[0135] The application can provide patients with step-by-step (or task-by-task) guidance on performing a research protocol, for example, through the display of audio or text, images, or movie clips. An example of a specific protocol-specific step-by-step guide for the application may include the following steps (where the application is running on a “phone”): Please place the phone on the provided tripod; Please flip the phone so that the front camera / selfie camera and phone screen are facing you / the patient; Raise / lower the phone slightly; Rotate the phone left / right to center the subject on the screen.

[0136] In an example of a research protocol studying a patient's head movement, one could test the patient's ability to rotate their head or tilt it left and right, as well as up and down. In this example, the application guides the patient through the research protocol to turn their head left, right, etc., as they move forward. The application can also provide feedback to the patient for each step or task in the research protocol before moving to the next step / task. For example, the application could indicate that the patient has already turned their head to a good degree and it is now time to move to the next step to tilt their head to the right.

[0137] Another example of a research protocol is studying a patient's gait while walking. The application can provide details of each or more steps of the walk. For example, the application could instruct the patient to walk forward ten steps, then stop, then turn 180 degrees, and then return. As mentioned above, the application can provide guidance and / or give the patient an overall plan to perform after each step or task (e.g., after 10 steps). In an example task-by-task (or step-by-step) scenario, the application could output audio stating: "Begin walking." After the patient completes each of the 10 steps, the application can provide a count of the correctly taken steps and ask the patient to repeat any incorrectly taken steps. After ten steps, the application can ask the patient to turn. At the end, the application can indicate to the patient that the gait study is complete.

[0138] As mentioned earlier, there are other research protocols that can be used to study patient activity with respect to different parts of the body and depending on the target impairment. Examples of such research protocols include: sit-up, where the task involves raising one leg to the maximum knee angle and repeating this with the other leg; or sit-to-stand, for example, 10 times, where the task involves performing sit-to-stand (timed standing and walking), then walking 25 feet, walking back 25 feet, and sitting down.

[0139] In addition to or as a substitute for providing patients with real-time feedback on their performance during the study of the protocol, the application can provide patients with self-reflective summaries of their performance based on their past use of the application. For example, the application could indicate that a patient's walking speed is 10% faster than last week, or 10% faster than usual (usually the average walking speed over a specific period of time, such as a month).

[0140] As described above, the application can instruct patients on research protocols or specific tasks and guide them in performing those tasks. The personal device can present the guidance to the patient as audio, such as spoken words played through the device's speaker. Alternatively or additionally, the personal device can present the guidance as visual cues (at least in part) through its screen (e.g., a phone screen). For example, the guidance may include images or video clips showing the correct way to perform physical activities / research protocols or specific tasks within those protocols. Visual cues may include specific graphical user interface elements, such as arrows, highlighted areas of specific images, etc., to emphasize specific movements of body parts during a particular task.

[0141] The application can provide guidance on the task at hand, for example, in real time and while the patient is performing the task. For instance, if an error or deviation is detected in a patient's performance of a specific task within the research protocol, the application can alert the patient to the error and guide them to correct it through audio or visual presentations discussed above. Depending on the severity of the error and its impact on the overall study, the application can request the patient to redo only the specific task or the entire study (including multiple consecutive tasks).

[0142] Figure 4 An example executable process 400 according to an embodiment of this disclosure is depicted. Process 400 is performed by a computer system (e.g., Figure 2 The system described in the text (200) is executed.

[0143] In process 400, the computing system receives (402) a series of images (or videos) depicting a patient performing a specific physical activity. The system transforms (404) the images into anonymized representations. Each anonymized representation is an animation of the patient's body depicted in the images within the series of images. Each animation includes multiple nodes, each node representing a specific body part of the patient in the corresponding image represented by the animation. In the animation, each node can be connected to one or more adjacent nodes by lines. Each node can be identified by the spatial coordinates assigned to that node.

[0144] Depending on which specific body parts are of interest, the system identifies (406) the target nodes associated with those body parts in the animation. For example, if the movement of the legs is of interest, the system identifies the animation segment representing the legs in the animation by the target nodes associated with the legs.

[0145] The system determines (408) a corresponding sequence of coordinates representing the motion of a target node as depicted in a series of images. Each sequence of coordinates is associated with a node and includes the spatial coordinates of that node as it changes in the series of images.

[0146] The system analyzes (410) these coordinate sequences to determine the patient's impairment or impairment stage. Depending on the physical activity, one or more modules and sub-modules of the system may be involved in this analysis. For example, for walking activities, sub-modules analyzing arm nodes and / or leg nodes (e.g., 234, 236) may be involved, while for eating activities, one or more sub-modules of facial motion analyzer 238 may be involved. To determine a specific impairment or a specific impairment stage, the system can compare the movement of the corresponding body part with the typical motor abilities of a patient group with that specific impairment and output the impairment or its stage based on the closest match (i.e., the group whose patient's ability or motor limitation is closest to the patient's ability and motor limitation).

[0147] The system transmits (412) the identified obstacles or stages for presentation. For example, a processor that has made a determination can transmit the results to the system's display component. The processor and the display can both reside on the same device (e.g., Figure 1 104 in the middle) or on a separate device, or on a separate device (e.g., one on backend system 112 and the other on device 104).

[0148] Figure 5 Examples of computing devices 500 and mobile computing devices that can be used to implement the techniques described herein are shown. For example, Figure 2 The system 200 depicted herein may take the form of a computing device 500, a mobile computing device 550, or a combination thereof. The computing device 500 is intended to represent various forms of digital computers (such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers). The mobile computing device is intended to represent various forms of mobile devices (such as personal digital assistants, mobile phones, smartphones, and other similar computing devices). The components shown herein, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the embodiments of the invention described and / or claimed herein.

[0149] Computing device 500 includes a processor 502, a memory 504, a storage device 506, a high-speed interface 508 connected to the memory 504 and multiple high-speed expansion ports 510, and a low-speed interface 512 connected to a low-speed expansion port 514 and the storage device 506. Each of the processor 502, memory 505, storage device 506, high-speed interface 508, high-speed expansion port 510, and low-speed interface 512 is interconnected using multiple buses and may be mounted on a shared motherboard or otherwise, as appropriate. The processor 502 can process instructions executed within the computing device 500 (including instructions stored in the memory 504 or on the storage device 506) to display graphical information for a GUI on an external input / output device, such as a display 516 coupled to the high-speed interface 508. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and multiple memory types, may be used as appropriate. Furthermore, multiple computing devices may be connected, each providing a necessary operational component (e.g., as a server library, blade server group, or multiple processor system).

[0150] Memory 504 stores information within computing device 500. In some embodiments, memory 504 is one or more volatile memory cells. In some embodiments, memory 504 is one or more non-volatile memory cells. Memory 504 may also be another form of computer-readable medium (such as a magnetic disk or optical disk).

[0151] Storage device 506 provides large-capacity storage for computing device 500. In some embodiments, storage device 506 may be or contain computer-readable media, such as floppy disk devices, hard disk devices, optical disk devices, magnetic tape devices, flash memory or other similar solid-state storage devices or device arrays (including storage area networks or other configured devices). The computer program product may be tangibly embodied in an information carrier. The computer program product may also include instructions that, when executed, perform one or more methods, such as those described above. The computer program product may also be tangibly implemented in a computer-readable or machine-readable medium (such as memory 505, storage device 506, or memory on processor 502).

[0152] High-speed interface 508 manages bandwidth-intensive operations of computing device 500, while low-speed interface 512 manages lower bandwidth-intensive operations. This functional allocation is merely exemplary. In some embodiments, high-speed interface 508 is coupled to memory 505, display 516 (e.g., via a graphics processor or accelerator), and high-speed expansion port 510, which can accept various expansion cards (not shown). In embodiments, low-speed interface 512 is coupled to storage device 506 and low-speed expansion port 515. Low-speed expansion port 515, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), can be coupled to one or more input / output devices (such as keyboards, clicking devices, scanners, or networking devices such as switches or routers) via a network adapter.

[0153] The computing device 500 can be implemented in several different forms, as shown in the figure. For example, it can be implemented as a standard server 520, or as a group of such servers multiple times. Furthermore, it can be implemented in a personal computer such as a laptop computer 522. It can also be implemented as part of a rack-mounted server system 525. Alternatively, components from the computing device 500 can be combined with other components from mobile devices (not shown) such as mobile computing devices 550. Each such device can contain one or more of the computing device 500 and the mobile computing device 550, and the entire system can consist of multiple computing devices communicating with each other.

[0154] Mobile computing device 550 includes processor 552, memory 565, input / output devices (such as display 555), communication interface 566, and transceiver 568, as well as other components. Mobile computing device 550 may also provide storage devices (such as microdrives or other devices) to provide additional storage. Each of processor 552, memory 565, display 555, communication interface 566, and transceiver 568 is interconnected using multiple buses, and some components may be mounted on a shared motherboard or otherwise as appropriate.

[0155] Processor 552 can execute instructions within mobile computing device 550 (including instructions stored in memory 565). Processor 552 can be implemented as a chipset including single and multiple analog and digital processors. Processor 552 can provide, for example, coordination of other components of mobile computing device 550, such as control of patient interface, applications running on mobile computing device 550, and wireless communication of mobile computing device 550.

[0156] Processor 552 can communicate with the patient via control interface 558 and display interface 556 coupled to display 555. Display 554 can be, for example, a TFT (Thin Film Transistor Liquid Crystal Display) or OLED (Organic Light Emitting Diode) display, or other suitable display technology. Display interface 556 may include suitable circuitry for driving display 554 to present graphic and other information to the patient. Control interface 558 can receive commands from the patient and translate them for submission to processor 552. Furthermore, external interface 562 can provide communication with processor 552, enabling mobile computing device 550 to communicate with other devices in the near area. External interface 562 may provide, in some embodiments, wired communication, or in other embodiments, wireless communication, and multiple interfaces may be used.

[0157] Memory 564 stores information within the mobile computing device 550. Memory 564 may be implemented as one or more computer-readable media, one or more volatile memory cells, or one or more non-volatile memory cells. Extended memory 574 may also be provided and connected to the mobile computing device 550 via an extended interface 572, which may include, for example, a SIMM (Single In-line Memory Module) card interface. Extended memory 574 may provide additional storage space for the mobile computing device 550, or it may store applications or other information for the mobile computing device 550. In particular, extended memory 574 may include instructions for performing or supplementing the processes described above, and may also include security information. Thus, for example, extended memory 574 may be provided as a security module for the mobile computing device 550 and may be programmed with instructions that allow secure use of the mobile computing device 550. Furthermore, secure applications and additional information (such as placing identification information on the SIMM card in an unbreakable manner) may be provided via a SIMM card.

[0158] The memory may include, for example, flash memory and / or NVRAM (non-volatile random access memory), as discussed below. In some embodiments, the computer program product is tangibly embodied as an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The computer program product may be a computer-readable medium or a machine-readable medium (such as memory 565, extended memory 575, or memory on processor 552). In some embodiments, the computer program product may be received, for example, via transceiver 568 or external interface 562 to transmit signals.

[0159] Mobile computing device 550 can conduct wireless communication via communication interface 566, which may include digital signal processing circuitry when necessary. Communication interface 566 can provide communication in various modes or protocols (such as GSM voice calls (Global System for Mobile Communications), SMS (Short Message Service), EMS (Enhanced Messaging Service) or MMS messaging (Multimedia Messaging Service), CDMA (Code Division Multiple Access), TDMA (Time Division Multiple Access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), etc.). Such communication can be performed, for example, using a radio frequency transceiver 568. Furthermore, short-range communication can be performed using transceivers such as Bluetooth, WiFi, or others (not shown). Additionally, GPS (Global Positioning System) receiver module 570 can provide additional navigation-related and positioning-related wireless data to mobile computing device 550, which can be used, as appropriate, by applications running on mobile computing device 550.

[0160] The mobile computing device 550 can also use an audio codec 560 for audio communication, which can receive spoken information from the patient and convert it into usable digital information. The audio codec 560 can also generate audible sounds for the patient, such as through a speaker (e.g., in the receiver of the mobile computing device 550). Such sounds can include sounds from voice telephone calls, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications operating on the mobile computing device 550.

[0161] The mobile computing device 550 can be implemented in several different forms, as shown in the figure. For example, it can be implemented as a mobile phone 580. It can also be implemented as a smartphone 582, a personal digital assistant, or part of another similar mobile device.

[0162] The various implementations of the systems and techniques described herein can be achieved using digital electronic circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These different implementations may include implementations within one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be coupled for specific or general purposes to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transfer data and instructions to the storage system, at least one input device, and at least one output device.

[0163] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural languages ​​and / or object-oriented programming languages, and / or in assembly / machine language. As used herein, the terms machine-readable medium and computer-readable medium refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term machine-readable signal refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0164] To provide interaction with patients, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the patient, as well as a keyboard and pointing device (e.g., a mouse or trackball), through which the patient can provide input to the computer. Other types of devices can also be used to provide interaction with patients; for example, feedback provided to the patient can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the patient can be received in any form, including sound, speech, or tactile input.

[0165] The systems and techniques described herein can be implemented in computing systems that include back-end components (e.g., as data servers), middleware components (e.g., application servers), front-end components (e.g., client computers with graphical patient interfaces or web browsers through which patients can interact with implementations of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected via any form or medium of digital data communication (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0166] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established through computer programs running on the respective computers and having a client-server relationship with each other.

[0167] Several embodiments of this disclosure have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the claims.

Claims

1. A method performed by a computing system, the method comprising: Receive multiple consecutive images of a patient performing physical activities; The image is transformed into a corresponding anonymous representation of the patient's body, each anonymous representation including multiple nodes, each node representing a specific body part of the patient's body, and each node can be identified by corresponding spatial coordinates assigned to the node; Identify a set of target nodes from the plurality of nodes that represent one or more target body parts of the patient; Determine a set of coordinate sequences representing the motion of the one or more target body parts during the physical activity, each coordinate sequence in the set including the spatial coordinates of a corresponding target node associated with the target body part as depicted in the successive images; Analyze the set of coordinate sequences to determine the stage of the medical disorder suffered by the patient, which is a neurological or musculoskeletal disorder; as well as The identified stages are sent to the user interface for presentation.

2. The method as described in claim 1, wherein, The physical activity is at least one of the patient’s head movement, speaking, chewing or swallowing, and the one or more target body parts include at least one of the patient’s head, mouth, lips or neck.

3. The method as described in any one of claims 1 or 2, wherein, The physical activities include walking. Wherein, the one or more target body parts include at least a plurality of portions of the patient's legs, and the set of target nodes includes a first leg node representing a portion of the patient's first leg and a second leg node representing a portion of the patient's second leg, and The analysis of the coordinate sequence set includes: The first coordinate sequence of the first leg node is compared with the second coordinate sequence of the second leg node to obtain a signal representing the gait cycle during the walking process. Each gait cycle starts from the corresponding starting point when the first leg passes the second leg along the walking direction. The patient's gait parameters are determined based on the gait cycle, and these parameters include one or more of the following: gait duration, gait count, gait frequency, or gait tone. The stage of the medical disorder is determined based on the footstep parameters.

4. The method of claim 3, wherein, Each gait cycle ends at the corresponding point when the second leg passes the first leg along the walking direction.

5. The method as described in any one of claims 3 or 4, wherein, The gait cycle begins at the point where the first leg passes the second leg along the walking direction and ends when the first leg passes the second leg again.

6. The method of any one of claims 3 to 5, further comprising: Determine the frontal plane of the patient's body. Specifically, when the first coordinate of the first leg point changes from a negative value to a zero or a positive value in the direction perpendicular to the frontal plane, the first leg passes through the second leg.

7. The method of claim 6, further comprising determining that the body activity is walking by determining that the first coordinates alternately and periodically change between negative and positive values.

8. The method according to any one of claims 3 to 7, wherein, The first leg node represents at least the portion of the first leg between the hip and the ankle.

9. The method of claim 8, wherein, The first leg node represents the knee, ankle, tibia, or thigh of the first leg.

10. The method according to any one of claims 3 to 9, wherein, The footstep parameters include foot frequency or gait, and The stage of the medical obstacle is determined based on a predetermined correlation between different stages of the medical obstacle and different foot frequencies or paces.

11. The method according to any one of claims 1 to 10, wherein, The physical activities include walking. Wherein, the one or more target body parts include at least a portion of the patient's arm, and the set of target nodes includes an elbow node representing the elbow of the arm, an upper arm node representing the body portion between the elbow and the shoulder of the arm, and a lower arm node representing the fingers of the arm or the segment between the fingers and the elbow. The analysis of the coordinate sequence set includes determining an arm angle sequence, where each arm angle sequence is formed relative to the elbow node at the corresponding image of the walk, between the upper arm node and the lower arm node. Specifically, the stage of the medical obstacle is determined based on the analysis of the arm angle sequence.

12. The method of claim 11, wherein, The analysis of the arm angle sequence includes determining the differences in arm angles based on the changes in angles within the arm angle sequence, and The stage of the medical disorder is determined by comparing the difference with the difference in arm angle with a predetermined correlation between the two stages of the medical disorder.

13. The method of any one of claims 11 or 12, wherein, The analysis of the arm angle sequence includes: Signals representing gait cycles are determined by identifying the repetition of arm angle changes over consecutive time periods during the walking process. The patient's gait parameters are determined based on the gait cycle, and the gait parameters include at least one of gait duration, gait count, or gait frequency. The stage of the medical obstacle is determined based on the footstep parameters.

14. The method according to any one of claims 11 to 13, wherein, Analysis of the arm angle sequence includes determining the minimum arm angle in the arm angle sequence. The stage for determining the medical obstacle is based on the minimum arm angle.

15. The method of any one of claims 1 to 14, further comprising: Determine the hip node representing the patient's hip, and Set the spatial coordinates of the hip node to (0, 0, 0). Among them, the spatial coordinates of other nodes are determined relative to the hip node.

16. The method according to any one of claims 1 to 15, wherein, The anonymous representation is a 3D representation of the patient's body.

17. The method according to any one of claims 1 to 16, wherein, The computing system is a mobile device or smartphone.

18. The method of claim 17, wherein, The mobile device or the smartphone includes a camera for capturing the plurality of consecutive images.

19. The method of any one of claims 1 to 18, further comprising: The user interface is made to display instructions for performing the physical activities.

20. The method of claim 19, further comprising: Compare the movement of the one or more target body parts with the command; as well as In response to determining that the movement does not conform to the instruction, an additional instruction is sent to the user interface to guide the patient to repeat the physical activity, wherein the additional instruction includes suggesting specific movements to correct the movement.

21. The method according to any one of claims 1 to 20, wherein, The stage of the medical obstacle is determined based on its progression or regression over time, and the progression or regression is determined in the following ways: From the patient's medical history, obtain a set of old coordinate sequences determining previous physical activities performed by the patient at a past time prior to the physical activity, the past time indicating the start of the time period, and The set of coordinate sequences is compared with the old set of coordinate sequences to determine whether the patient’s motor function has improved or deteriorated, wherein the improvement is associated with the disappearance of the impairment and the deterioration is associated with the progression of the impairment.

22. The method of any one of claims 1 to 21, further comprising: The therapeutic efficacy of the treatment experienced by the patient is determined based on the identified stage of the obstacle and a previously identified stage, and based on the time elapsed since the previously identified stage.

23. The method of any one of claims 1 to 22, further comprising: The user interface provides real-time feedback on the patient's performance on the physical activity as the patient performs the activity.

24. A system comprising: One or more computers; as well as One or more computer-readable storage devices store operable instructions that, when executed by the one or more computers, cause the one or more computers to perform operations, the operations including: Receive multiple consecutive images of a patient performing physical activities; The image is transformed into a corresponding anonymous representation of the patient's body, each anonymous representation including multiple nodes, each node representing a specific body part of the patient's body, and each node can be identified by corresponding spatial coordinates assigned to the node; Identify a set of target nodes from the plurality of nodes that represent one or more target body parts of the patient; Determine a set of coordinate sequences representing the motion of the one or more target body parts during the physical activity, each coordinate sequence in the set including the spatial coordinates of a corresponding target node associated with the target body part as depicted in the successive images; Analyzing the set of coordinate sequences to determine the stage of the medical disorder suffered by the patient, said medical disorder being a neurological or musculoskeletal disorder; and The identified stages are sent to the user interface for presentation.

25. The system of claim 24, wherein, The physical activity is at least one of the patient’s head movement, speaking, chewing or swallowing, and the one or more target body parts include at least one of the patient’s head, mouth, lips or neck.

26. The system as claimed in any one of claims 24 or 25, wherein, The physical activities include walking. Wherein, the one or more target body parts include at least a plurality of portions of the patient's legs, and the set of target nodes includes a first leg node representing a portion of the patient's first leg and a second leg node representing a portion of the patient's second leg, and The analysis of the coordinate sequence set includes: The first coordinate sequence of the first leg node is compared with the second coordinate sequence of the second leg node to obtain a signal representing the gait cycle during the walking process. Each gait cycle starts from the corresponding starting point when the first leg passes the second leg along the walking direction. The patient's gait parameters are determined based on the gait cycle, and the gait parameters include one or more of the following: gait duration, gait count, or gait frequency. The stage of the medical disorder is determined based on the footstep parameters.

27. The system of claim 26, wherein, Each gait cycle ends at the corresponding point when the second leg passes the first leg along the walking direction.

28. The system as claimed in any one of claims 26 or 27, wherein, In this text, the gait cycle begins at the point where the first leg passes the second leg along the walking direction and ends when the first leg passes the second leg again.

29. The system as claimed in any one of claims 26 to 28, wherein, The operation further includes determining the frontal plane of the patient's body. Specifically, when the first coordinate of the first leg point changes from a negative value to a zero or a positive value in the direction perpendicular to the frontal plane, the first leg passes through the second leg.

30. The system of claim 29, wherein, The operation further includes determining that the physical activity is walking by determining that the first coordinate changes alternately and periodically between negative and positive values.

31. The system as claimed in any one of claims 26 to 30, wherein, The first leg node represents at least the portion of the first leg between the hip and the ankle.

32. The system of claim 31, wherein, The first leg node represents the knee, ankle, tibia, or thigh of the first leg.

33. The system as claimed in any one of claims 26 to 32, wherein, The footstep parameters include foot frequency or gait, and The stage of the medical obstacle is determined based on a predetermined correlation between different stages of the medical obstacle and different foot frequencies or paces.

34. The system as claimed in any one of claims 24 to 33, wherein, The physical activities include walking. Wherein, the one or more target body parts include at least a portion of the patient's arm, and the set of target nodes includes an elbow node representing the elbow of the arm, an upper arm node representing the body portion between the elbow and the shoulder of the arm, and a lower arm node representing the fingers of the arm or the segment between the fingers and the elbow. The analysis of the coordinate sequence set includes determining an arm angle sequence, where each arm angle sequence is formed relative to the elbow node at the corresponding image of the walk, between the upper arm node and the lower arm node. Specifically, the stage of the medical obstacle is determined based on the analysis of the arm angle sequence.

35. The system of claim 34, wherein, The analysis of the arm angle sequence includes determining the differences in arm angles based on the changes in angles within the arm angle sequence, and The stage of the medical disorder is determined by comparing the difference with the difference in arm angle with a predetermined correlation between the two stages of the medical disorder.

36. The system as claimed in any one of claims 34 or 35, wherein, The analysis of the arm angle sequence includes: Signals representing gait cycles are determined by identifying the repetition of arm angle changes over consecutive time periods during the walking process. The patient's gait parameters are determined based on the gait cycle, and the gait parameters include at least one of gait duration, gait count, or gait frequency. The stage of the medical obstacle is determined based on the footstep parameters.

37. The system as claimed in any one of claims 34 to 36, wherein, Analysis of the arm angle sequence includes determining the minimum arm angle in the arm angle sequence. The stage for determining the medical obstacle is based on the minimum arm angle.

38. The system as claimed in any one of claims 24 to 37, wherein, The operation further includes: Determine the hip node representing the patient's hip, and Set the spatial coordinates of the hip node to (0, 0, 0). Among them, the spatial coordinates of other nodes are determined relative to the hip node.

39. The system as claimed in any one of claims 24 to 38, wherein, The anonymous representation is a 3D representation of the patient's body.

40. The system as claimed in any one of claims 24 to 39, wherein, The one or more computers include mobile devices or smartphones.

41. The system of claim 40, wherein, The mobile device or the smartphone includes a camera for capturing the plurality of consecutive images.

42. The system as claimed in any one of claims 24 to 41, wherein, The operation further includes causing the user interface to display instructions for performing the physical activity.

43. The system of claim 42, wherein, The operation further includes: Compare the movement of the one or more target body parts with the command; and In response to determining that the movement does not conform to the instruction, an additional instruction is sent to the user interface to guide the patient to repeat the physical activity, wherein the additional instruction includes suggesting specific movements to correct the movement.

44. The system as claimed in any one of claims 24 to 43, wherein, The stage of the medical obstacle is determined based on its progression or regression over time, and the progression or regression is determined in the following ways: From the patient's medical history, obtain a set of old coordinate sequences determining previous physical activities performed by the patient at a past time prior to the physical activity, the past time indicating the start of the time period, and The set of coordinate sequences is compared with the old set of coordinate sequences to determine whether the patient’s motor function has improved or deteriorated, wherein the improvement is associated with the disappearance of the impairment and the deterioration is associated with the progression of the impairment.

45. The system as claimed in any one of claims 24 to 44, wherein, The operation further includes determining the therapeutic efficacy of the treatment experienced by the patient based on the determined stage of the obstacle and a previously determined stage, and based on the time elapsed since the previously determined stage.

46. ​​The system as claimed in any one of claims 24 to 45, wherein, The operation further includes enabling the user interface to present real-time feedback on the patient's performance on the physical activity as the patient performs the physical activity.

47. A non-transitory computer-readable medium storing one or more instructions, said one or more instructions causing the computing system to perform operations when executed by the computing system, said operations including: Receive multiple consecutive images of a patient performing physical activities; The image is transformed into a corresponding anonymous representation of the patient's body, each anonymous representation including multiple nodes, each node representing a specific body part of the patient's body, and each node can be identified by corresponding spatial coordinates assigned to the node; Identify a set of target nodes from the plurality of nodes that represent one or more target body parts of the patient; Determine a set of coordinate sequences representing the motion of the one or more target body parts during the physical activity, each coordinate sequence in the set including the spatial coordinates of a corresponding target node associated with the target body part as depicted in the successive images; Analyze the set of coordinate sequences to determine the stage of the medical disorder suffered by the patient, which is a neurological or musculoskeletal disorder; as well as The identified stages are sent to the user interface for presentation.

48. The non-transitory computer-readable medium of claim 47, wherein, The physical activity is at least one of the patient’s head movement, speaking, chewing or swallowing, and the one or more target body parts include at least one of the patient’s head, mouth, lips or neck.

49. The non-transitory computer-readable medium according to any one of claims 47 or 48, wherein, The physical activities include walking. Wherein, the one or more target body parts include at least a plurality of portions of the patient's legs, and the set of target nodes includes a first leg node representing a portion of the patient's first leg and a second leg node representing a portion of the patient's second leg, and The analysis of the coordinate sequence set includes: The first coordinate sequence of the first leg node is compared with the second coordinate sequence of the second leg node to obtain a signal representing the gait cycle during the walking process. Each gait cycle starts from the corresponding starting point when the first leg passes the second leg along the walking direction. The patient's gait parameters are determined based on the gait cycle, and the gait parameters include one or more of the following: gait duration, gait count, or gait frequency. The stage of the medical disorder is determined based on the footstep parameters.

50. The non-transitory computer-readable medium of claim 49, wherein, Each gait cycle ends at the corresponding point when the second leg passes the first leg along the walking direction.

51. The non-transitory computer-readable medium according to any one of claims 49 or 50, wherein, In this text, the gait cycle begins at the point where the first leg passes the second leg along the walking direction and ends when the first leg passes the second leg again.

52. The non-transitory computer-readable medium according to any one of claims 49 to 51, wherein, The operation further includes determining the frontal plane of the patient's body. Specifically, when the first coordinate of the first leg point changes from a negative value to a zero or a positive value in the direction perpendicular to the frontal plane, the first leg passes through the second leg.

53. The non-transitory computer-readable medium of claim 52, wherein, The operation further includes determining that the physical activity is walking by determining that the first coordinate changes alternately and periodically between negative and positive values.

54. The non-transitory computer-readable medium according to any one of claims 49 to 53, wherein, The first leg node represents at least the portion of the first leg between the hip and the ankle.

55. The non-transitory computer-readable medium of claim 54, wherein, The first leg node represents the knee, ankle, tibia, or thigh of the first leg.

56. The non-transitory computer-readable medium according to any one of claims 49 to 55, wherein, The footstep parameters include foot frequency or gait, and The stage of the medical obstacle is determined based on a predetermined correlation between different stages of the medical obstacle and different foot frequencies or paces.

57. The non-transitory computer-readable medium according to any one of claims 47 to 56, wherein, The physical activities include walking. Wherein, the one or more target body parts include at least a portion of the patient's arm, and the set of target nodes includes an elbow node representing the elbow of the arm, an upper arm node representing the body portion between the elbow and the shoulder of the arm, and a lower arm node representing the fingers of the arm or the segment between the fingers and the elbow. The analysis of the coordinate sequence set includes determining an arm angle sequence, where each arm angle sequence is formed relative to the elbow node at the corresponding image of the walk, between the upper arm node and the lower arm node. Specifically, the stage of the medical obstacle is determined based on the analysis of the arm angle sequence.

58. The non-transitory computer-readable medium of claim 57, wherein, The analysis of the arm angle sequence includes determining the differences in arm angles based on the changes in angles within the arm angle sequence, and The stage of the medical disorder is determined by comparing the difference with the difference in arm angle with a predetermined correlation between the two stages of the medical disorder.

59. The non-transitory computer-readable medium according to any one of claims 57 or 58, wherein, The analysis of the arm angle sequence includes: Signals representing gait cycles are determined by identifying the repetition of arm angle changes over consecutive time periods during the walking process. The patient's gait parameters are determined based on the gait cycle, and the gait parameters include at least one of gait duration, gait count, or gait frequency. The stage of the medical obstacle is determined based on the footstep parameters.

60. The non-transitory computer-readable medium according to any one of claims 57 to 59, wherein, Analysis of the arm angle sequence includes determining the minimum arm angle in the arm angle sequence. The stage for determining the medical obstacle is based on the minimum arm angle.

61. The non-transitory computer-readable medium according to any one of claims 47 to 60, wherein, The operation further includes: Determine the hip node representing the patient's hip, and Set the spatial coordinates of the hip node to (0, 0, 0). Among them, the spatial coordinates of other nodes are determined relative to the hip node.

62. The non-transitory computer-readable medium as described in claims 47 to 61, wherein, The anonymous representation is a 3D representation of the patient's body.

63. The non-transitory computer-readable medium as described in claims 47 to 62, wherein, The one or more computers include mobile devices or smartphones.

64. The non-transitory computer-readable medium of claim 63, wherein, The mobile device or the smartphone includes a camera for capturing the plurality of consecutive images.

65. The non-transitory computer-readable medium as described in claims 47 to 64, wherein, The operation further includes causing the user interface to display instructions for performing the physical activity.

66. The non-transitory computer-readable medium of claim 65, wherein, The operation further includes: Compare the movement of the one or more target body parts with the command; and In response to determining that the movement does not conform to the instruction, an additional instruction is sent to the user interface to guide the patient to repeat the physical activity, wherein the additional instruction includes suggesting specific movements to correct the movement.

67. The non-transitory computer-readable medium as described in claims 47 to 66, wherein, The stage of the medical obstacle is determined based on its progression or regression over time, and the progression or regression is determined in the following ways: From the patient's medical history, obtain a set of old coordinate sequences determining previous physical activities performed by the patient at a past time prior to the physical activity, the past time indicating the start of the time period, and The set of coordinate sequences is compared with the old set of coordinate sequences to determine whether the patient’s motor function has improved or deteriorated, wherein the improvement is associated with the disappearance of the impairment and the deterioration is associated with the progression of the impairment.

68. The non-transitory computer-readable medium as described in claims 47 to 67, wherein, The operation further includes determining the therapeutic efficacy of the treatment experienced by the patient based on the determined stage of the obstacle and a previously determined stage, and based on the time elapsed since the previously determined stage.

69. The non-transitory computer-readable medium as described in claims 47 to 68, wherein, The operation further includes enabling the user interface to present real-time feedback on the patient's performance on the physical activity as the patient performs the physical activity.

Images