Classification of Parkinson's disease tremors using machine learning

Abstract

A method, system, and apparatus comprising a computer program encoded in a computer storage medium for classifying tremors in a subject having Parkinson's disease. In one embodiment, a method is provided comprising: receiving accelerometer data characterizing the movement of a subject having Parkinson's disease; processing the accelerometer data using a tremor classification neural network to generate a score distribution across a set of tremor classes according to the values ​​of set parameters of the tremor classification neural network, wherein the set of tremor classes comprises a plurality of tremor classes, and each tremor class in the set of tremor classes corresponds to the respective severity of the tremor; and classifying the tremors of a subject having Parkinson's disease based on the score distribution across the set of tremor classes.

Description

Technical Field

[0001] This specification relates to the tremor classification of Parkinson's disease using machine learning.

Background Art

[0002] A machine learning model receives an input and generates an output, for example, a predicted output, based on the received input. Some machine learning models are parametric models that generate an output based on the received input and the values of the model's parameters.

[0003] Some machine learning models are deep models that use multiple layers of the model to generate an output for the received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to the received input to generate an output.

[0004] Parkinson's disease is a chronic progressive neurological disorder caused by the gradual loss of nerve cells in the brain. The loss of nerve cells causes a deficiency of dopamine, a neurotransmitter involved in the control of movement. When nerve cells degenerate or die, the body has difficulty controlling movement, and the symptoms of Parkinson's disease begin to appear. Common symptoms of Parkinson's disease include tremors, slowness and stiffness of movement, difficulty with balance and coordination, and an inability to make facial expressions. In advanced cases, patients may also suffer from cognitive decline, including depression and dementia. The diagnosis of Parkinson's disease is made by neurological examination, medical history, and imaging tests.

[0005] Dyskinesia is a movement disorder characterized by involuntary, often irregular movements such as writhing, jerking, twisting, or squirming. This is a common side effect of the long-term use of drugs that affect dopamine levels in the brain, such as those used to treat Parkinson's disease.

Summary of the Invention

Means for Solving the Problems

[0006] This specification describes a classification system implemented as a computer program on one or more computers in one or more locations that can process accelerometer data characterizing the movement of a subject with Parkinson's disease to classify the subject's tremor, classify the subject's dyskinesia, or both.

[0007] Throughout this specification, "subject" refers to a human subject, that is, a person.

[0008] Throughout this specification, the term "subnetwork" in a neural network refers to a part of a neural network.

[0009] Throughout this specification, “embedding” of an entity (e.g., an accelerometer signal) refers to a representation of the entity in latent space generated by a neural network (or a subnetwork of a neural network). Embedding can refer to an ordered collection of numbers, such as a vector of numbers, a matrix, or other tensor.

[0010] Throughout this specification, "block" (for example, "convolutional block" or "recurrent block") refers to a group of neural network layers in a neural network.

[0011] Throughout this specification, the terms "accelerometer data" and "accelerometer signal" are used interchangeably.

[0012] In one example, a method is provided that is performed by one or more computers, the method comprising: receiving accelerometer data characterizing the movement of a subject having Parkinson's disease; processing the accelerometer data using a tremor classification neural network to generate a score distribution across a set of tremor classes according to the values ​​of a set parameter of the tremor classification neural network, wherein the set of tremor classes comprises multiple tremor classes, and each tremor class in the set of tremor classes corresponds to the respective severity of the tremor; and classifying the tremor of a subject having Parkinson's disease based on the score distribution across the set of tremor classes.

[0013] In some embodiments, accelerometer data is generated by the wearable device in question.

[0014] In some embodiments, accelerometer data is represented as one or more one-dimensional (1D) time signals.

[0015] In some embodiments, accelerometer data is represented as a two-dimensional (2D) spectrogram signal.

[0016] In some embodiments, the set of tremor classes includes at least three tremor classes.

[0017] In some embodiments, the set of tremor classes corresponds to the Clinical Tremor Scale.

[0018] In some embodiments, each tremor class in the set of tremor classes corresponds to the magnitude, frequency, and duration of the respective tremor.

[0019] In some embodiments, the method further includes receiving video data characterizing the motion of an object, processing accelerometer data using a tremor classification neural network, or processing accelerometer data and video data together using a tremor classification neural network.

[0020] In some embodiments, the method further includes receiving surface electromyography (EMG) data characterizing the potentials generated during the muscle contraction of the subject, processing accelerometer data using a tremor classification neural network, or processing accelerometer data and surface EMG data together using a tremor classification neural network.

[0021] In some embodiments, processing accelerometer data using a tremor classification neural network to generate a score distribution across a set of tremor classes includes: processing the accelerometer data using a convolutional block containing one or more convolutional neural network layers to generate a convolutional block output; processing the convolutional block output using a recurrent block containing one or more recurrent neural network layers to generate a recurrent block output; and processing the recurrent block output using a high-density block containing one or more high-density neural network layers to generate a score distribution across a set of tremor classes.

[0022] In some embodiments, the convolutional block output has a lower temporal resolution than the accelerometer data.

[0023] In some embodiments, the recurrent neural network layer is a long-term short-term memory (LSTM) neural network layer.

[0024] In some embodiments, classifying the tremors of subjects with Parkinson's disease based on a score distribution across a set of tremor classes includes classifying the subject's tremors into the tremor class associated with the highest score within the set of tremor classes.

[0025] In some embodiments, the method further includes generating a series of tremor classifications by repeatedly performing tremor classifications over a series of time intervals, wherein each tremor classification in the series of tremor classifications corresponds to a respective time interval, and the method classifies and generates the tremors of interest during the time intervals.

[0026] In some embodiments, the method further includes processing a series of tremor classifications to classify the progression of the subject's Parkinson's disease from a set of progression states to a progression state.

[0027] In some embodiments, the method further includes administering to the subject a drug for treating Parkinson's disease or symptoms of Parkinson's disease, at least in part based on the progression state of Parkinson's disease in the subject.

[0028] In some embodiments, the method further includes generating a notification indicating a classification of tremors of a subject having Parkinson's disease.

[0029] In some embodiments, processing the accelerometer data using a tremor classification neural network to generate a score distribution over a set of tremor classes includes processing the accelerometer data using an encoder subnetwork of the tremor classification neural network to generate an embedding of the accelerometer data in a latent space, and processing the embedding of the accelerometer data in the latent space using a tremor subnetwork of the tremor classification neural network to generate a score distribution over a set of tremor classes.

[0030] In some embodiments, the method further includes processing the embedding of the accelerometer data in the latent space using a dyskinesia subnetwork of the tremor classification neural network to generate a score distribution over a set of dyskinesia classes, the set of dyskinesia classes including a plurality of dyskinesia classes, each dyskinesia class in the set of dyskinesia classes corresponding to a respective severity of dyskinesia.

[0031] In some embodiments, the method further includes classifying the dyskinesia of a subject having Parkinson's disease based on the score distribution over the set of dyskinesia classes.

[0032] In some embodiments, the tremor classification neural network is trained by an operation that includes pre-training an encoder subnetwork of the tremor classification neural network to perform an auxiliary task, wherein the auxiliary task is not a tremor classification task, and then training the tremor classification neural network to perform a tremor classification task after the encoder subnetwork has been pre-trained.

[0033] In some embodiments, pre-training an encoder subnetwork of a tremor classification neural network to perform an auxiliary task includes: acquiring a base accelerometer signal; generating positive pair embeddings in latent space, which includes: processing a first transformed version of the base accelerometer signal using the encoder subnetwork to generate embeddings corresponding to the latent space; processing a second transformed version of the base accelerometer signal using the encoder subnetwork to generate embeddings corresponding to the latent space; and training the encoder subnetwork to optimize the auxiliary loss that depends on (i) the embeddings of the first transformed version of the base accelerometer signal and (ii) the embeddings of the second transformed version of the base accelerometer signal.

[0034] In some embodiments, the method further includes generating a first transformed version of a base accelerometer signal, which includes randomly sampling a first transform from a space of transforms, and applying a first transform to the base accelerometer signal to generate a first transformed version of the base accelerometer signal.

[0035] In some embodiments, the method further includes generating a second transformed version of a base accelerometer signal, which includes randomly sampling a second transform from a space of transforms, and applying a second transform to the base accelerometer signal to generate a second transformed version of the base accelerometer signal.

[0036] In some embodiments, the auxiliary loss measures the error between (i) the embedding of a first transformed version of the base accelerometer signal and (ii) the embedding of a second transformed version of the base accelerometer signal.

[0037] In some embodiments, pre-training an encoder subnetwork of a tremor classification neural network to perform an auxiliary task includes acquiring an accelerometer signal, generating a masked accelerometer signal by masking a portion of the accelerometer signal, processing the masked accelerometer signal using the encoder subnetwork to generate an embedding of the masked accelerometer signal, processing the embedding of the masked accelerometer signal using the decoder neural network to generate a predictive reconstruction of the accelerometer signal, and training the encoder subnetwork and decoder neural network together to optimize an auxiliary loss that measures the error of the predictive reconstruction of the accelerometer signal.

[0038] In some embodiments, pre-training an encoder subnetwork of a tremor classification neural network to perform an auxiliary task includes: acquiring an accelerometer signal; generating a noisy accelerometer signal by adding a noisy accelerometer signal; processing the noisy accelerometer signal using the encoder subnetwork to generate an embedding of the noisy accelerometer signal; processing the embedding of the denoised accelerometer signal using the decoder neural network to generate a denoised accelerometer signal; and training the encoder subnetwork and decoder neural network together to optimize an auxiliary loss that measures the error of the denoised accelerometer signal.

[0039] In some embodiments, pre-training an encoder subnetwork of a tremor classification neural network to perform an auxiliary task includes (i) obtaining an accelerometer signal and (ii) a target label for the accelerometer signal, processing the accelerometer signal using the encoder subnetwork to generate an embedding for the accelerometer signal, processing the embedding for the accelerometer signal using the predictive neural network to generate a predicted label for the accelerometer signal, and training the encoder subnetwork and the predictive neural network together to optimize an auxiliary loss that measures the error between (i) the target label and (ii) the predicted label.

[0040] In some embodiments, the target label of the accelerometer signal defines the number of steps taken by the subject during the duration covered by the accelerometer signal.

[0041] In some embodiments, the target label of the accelerometer signal defines the action performed by the subject during the duration covered by the accelerometer signal.

[0042] In some embodiments, training a tremor classification neural network to perform a tremor classification task includes (i) obtaining a training accelerometer signal and (ii) a target tremor classification of the training accelerometer signal; processing the training accelerometer signal using an embedding subnet to generate an embedding of the training accelerometer signal; processing the embedding of the training accelerometer signal using a tremor subnet to generate a score distribution across a set of tremor classes; and training a tremor subnet of the tremor classification neural network to optimize a tremor objective function that measures the error between (i) the score distribution across a set of tremor classes and (ii) the target tremor classification.

[0043] In some embodiments, the parameter values ​​of the encoder subnetwork are frozen during the training of the tremor classification neural network in order to perform the tremor classification task.

[0044] In another embodiment, a system is provided comprising one or more computers and one or more storage devices communicably coupled to one or more computers, the one or more storage devices storing instructions that, when executed by one or more computers, cause one or more computers to perform operations in the manner described herein.

[0045] In another embodiment, when executed by one or more computers, one or more non-temporary computer storage media are provided that store instructions causing one or more computers to perform the operation of the method described herein.

[0046] Certain embodiments of the subject matter described herein can be implemented to achieve one or more of the following advantages:

[0047] The classification systems described herein can use a classification neural network to process accelerometer data characterizing the motion of an object to classify the object's tremor, classify the object's dyskinesia, or both. The classification neural network can generate tremor classifications or dyskinesia classifications by directly processing raw accelerometer data without requiring manual or heuristic feature engineering. More specifically, the classification neural network can be trained by machine learning training techniques to identify and extract relevant feature representations from raw accelerometer data in order to accurately classify tremor and dyskinesia.

[0048] The classification system enables granular and real-time monitoring of the status and progression of Parkinson's disease in subjects without requiring them to undergo regular medical examinations or utilize medical facilities. More specifically, the classification system can automatically process accelerometer data generated by the subject's personal devices, such as wearable devices or smartphones, to produce accurate classifications of tremors and dyskinesia, without requiring intermediate analysis by a physician. Furthermore, the classification system enables objective and reproducible assessment of tremors and dyskinesia in subjects, thus favoring self-report assessments (which can be inaccurate and vary from subject to subject) and assessments performed by physicians (which can only be performed at discrete moments, in contrast to classification systems that provide continuous monitoring).

[0049] A classification system can be trained by a two-step process that involves first pre-training the encoder subnetwork of the classification neural network for one or more auxiliary tasks, and then fine-tuning the classification neural network to perform tremor classification or dyskinesia classification. Pre-training the encoder subnetwork encourages it to generate rich and useful features that characterize the structure of accelerometer data, thereby reducing the amount of training data and the number of training iterations required to fine-tune the classification neural network. Therefore, pre-training the encoder subnetwork reduces the consumption of computational resources such as memory and computing power during the fine-tuning of the classification neural network.

[0050] Details of one or more embodiments of the subject matter of this specification will be described in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0051] [Figure 1]An exemplary classification system, including a classification neural network, is shown. [Figure 2] An exemplary architecture for a classification neural network is shown. [Figure 3] This shows an exemplary architecture of an encoder subnetwork included in a classification neural network. [Figure 4] This is an illustrative flowchart of a process for classifying tremors in subjects with Parkinson's disease, or for classifying dyskinesia in subjects with Parkinson's disease. [Figure 5] An exemplary training system is shown. [Figure 6] This is an illustrative process flowchart for pre-training an encoder subnetwork of a classification neural network to perform a controlled embedding task. [Figure 7] This is an illustrative process flowchart for pre-training an encoder subnetwork of a classification neural network to perform a masked reconstruction task. [Figure 8] This is an illustrative process flowchart for pre-training an encoder subnetwork of a classification neural network to perform a noisy reconstruction task. [Figure 9] This is an illustrative process flowchart for pre-training an encoder subnetwork of a classification neural network to perform supervised aiding tasks. [Figure 10] This is an illustrative flowchart of the process of training a classification neural network to perform a tremor classification task. [Figure 11] This is an exemplary flowchart illustrating the process of training a classification neural network to perform a dyskinesia classification task. [Figure 12] This section presents an example of experimental results for a classification system used to categorize tremors in patients with Parkinson's disease. [Figure 13](i) a wearable device worn by, for example, a person with Parkinson's disease; (ii) accelerometer data generated by the wearable device's accelerometer, represented as one or more one-dimensional (1D) time signals; and (iii) accelerometer data generated by the accelerometer in the wearable device, represented as a two-dimensional (2D) time-frequency spectrogram. [Modes for carrying out the invention]

[0052] Similar reference numbers and symbols in various drawings indicate the same elements.

[0053] Figure 1 shows an exemplary classification system 100. Classification system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations where the systems, components, and technologies described below are implemented.

[0054] The classification system 100 is configured to receive accelerometer data 104 that characterize the movements of a subject 102 having Parkinson's disease. The classification system 100 processes the accelerometer data 104 to generate one or both of the following: (i) a tremor classification 116 that characterizes the severity of the subject 102's tremor, or (ii) a dyskinesia classification 118 that characterizes the severity (and / or presence) of the subject 102's dyskinesia.

[0055] The accelerometer data 104 can characterize the movement of the object 102 over time intervals, for example, 1 second, 2 seconds, 10 seconds, or 1 minute. The accelerometer data 104 can be generated by an accelerometer device placed on or near the object 102. For example, the accelerometer data 104 can be generated by an accelerometer in a wearable device worn by the object 102, for example, a wearable device worn on the object's wrist. As another example, the accelerometer data 104 can be generated by an accelerometer in a portable device (e.g., a smartphone) held by the object.

[0056] The classification system 100 can represent the accelerometer data 104 in any of the following formats. For example, the classification system 100 can represent the accelerometer data 104 by one or more one-dimensional (1D) time signals, for example, by each 1D time signal characterizing each acceleration in the x, y, and z directions (where the x, y, and z directions are defined with reference to an appropriate reference coordinate system). As another example, the classification system 100 can represent the accelerometer data 104 by a two-dimensional (2D) spectrogram in the time-frequency domain, i.e., by a spectrogram containing one dimension representing time and two dimensions representing frequency.

[0057] Optionally, the classification system 100 may receive additional inputs, i.e., inputs in addition to the accelerometer data 104. These additional inputs may include any appropriate data characterizing the subject 102. The classification system 100 may process these additional inputs as part of the generation of a tremor classification 116, a dyskinesia classification 118, or both. These additional inputs may be derived from any appropriate modality and may provide the classification system 100 with additional information sources to improve the accuracy of the tremor classification 116 and the dyskinesia classification 118. Several examples of additional inputs for the classification system 100 are described below.

[0058] In some embodiments, the classification system 100 further receives video data characterizing the movement of the object 102. More specifically, the video data may include video showing a part of the object (e.g., the object's hand, or face, or torso) or the entire object (e.g., the entire body) over a set time interval. The time interval can be, for example, 1 second, 2 seconds, 10 seconds, or 1 minute, and can characterize the object over the same time interval as the accelerometer data 104. The video data may be captured by a video recording device such as a webcam or a smartphone video recording device. The video data may be represented as a series of video frames captured at any suitable sampling frequency, for example, 24 frames / second.

[0059] In some embodiments, the classification system 100 further receives surface electromyography (EMG) data characterizing the potentials generated during muscle contraction of the subject 102. Surface EMG data can be measured over any suitable time interval, e.g., 1 second, 2 seconds, 10 seconds, or 1 minute, and can characterize the subject over the same time interval as the accelerometer data. Surface EMG data measures the electrical activity of the subject's muscles. Capturing surface EMG data may involve placing electrodes on the skin surface over the muscle of interest. These electrodes sense the electrical activity generated when the muscle contracts. The classification system 100 can represent the surface EMG data, for example, by an array of 1D time signals, or in any other suitable format.

[0060] The classification system 100 can process accelerometer data 104 and any additional inputs characterizing the subject 102 using the classification neural network 200, tremor classification engine 112, and dyskinesia classification engine 118, respectively, as described below.

[0061] The classification neural network 200 is configured to receive a network input 106 which includes accelerometer data 104 and optionally any additional inputs that characterize the subject 102, such as video data or surface EMG data. The classification neural network 200 processes the network input 106 according to the values ​​of a set of classification neural network parameters to generate one or both of the following: (i) a score distribution 108 over a set of tremor classes, and (ii) a score distribution 110 over a set of dyskinesia classes.

[0062] A score distribution across a set of tremor classes 108 may include a score for each tremor class within the set of tremor classes. The scores for each tremor class can determine the likelihood that the subject has a tremor included in that class. Each tremor class may correspond to a different severity of tremor. The severity of tremor can be characterized, for example, by magnitude, frequency, and duration (consistency) of the tremor. A set of tremor classes may include any appropriate number of tremor classes, e.g., three, four, or five tremor classes. A set of tremor classes may include a "no tremor" class, i.e., a class indicating that the subject does not exhibit tremor. A set of tremor classes may correspond to a clinical tremor scale, e.g., the Essential Tremor Rating Scale (TETRAS).

[0063] The score distribution 110 across the set of dyskinesia classes may include the respective scores for each dyskinesia class within the set. The scores for each dyskinesia class can determine the likelihood that the subject has a dyskinesia included in that class. Each dyskinesia class may correspond to a different severity of dyskinesia. The severity of dyskinesia can be characterized, for example, by the magnitude, frequency, and duration (consistency) of involuntary movements by the subject, such as writhing, convulsing, twisting, or struggling. The set of dyskinesia classes may include any appropriate number of dyskinesia classes, e.g., three, four, or five dyskinesia classes. The set of dyskinesia classes may include a "no dyskinesia" class, i.e., a class indicating that the subject does not exhibit symptoms of dyskinesia.

[0064] In some embodiments, the classification neural network 200 generates a score distribution only across a set of tremor classes, but not across a set of dyskinesia classes. In some embodiments, the classification neural network 200 generates a score distribution only across a set of dyskinesia classes, but not across a set of tremor classes. In some embodiments, the classification neural network generates both a score distribution across a set of tremor classes and a score distribution across a set of dyskinesia classes.

[0065] The classification neural network 200 may have any suitable neural network architecture that enables it to perform its described function, for example, generating a score distribution over a set of tremor classes, or generating a score distribution over a set of dyskinesia classes, or both. In particular, the classification neural network may include any suitable number of (e.g., 5, 10, 50, etc.) any suitable neural network layers (e.g., convolutional layers, attention layers, fully connected layers, recurrent layers, etc.) and connect them in any suitable configuration (e.g., as a linear series of layers). An exemplary architecture of the classification neural network will be described in more detail with reference to Figure 2.

[0066] The classification system 100 can use the training system to train the classification neural network 200 to perform a tremor classification task, a dyskinesia classification task, or both. Optionally, the training system can pre-train portions of the classification neural network 200 to perform one or more unsupervised or supervised auxiliary tasks before training the classification neural network 200 to perform a tremor classification task, a dyskinesia classification task, or both. An example of a training system for training a classification neural network is described in more detail with reference to Figure 5.

[0067] The tremor classification engine 112 is configured to process a score distribution 108 across a set of tremor classes in order to generate a tremor classification 116. For example, the tremor classification engine 112 can generate a tremor classification that classifies a subject as belonging to the tremor class associated with the highest score under the score distribution 108 across the set of tremor classes.

[0068] Optionally, the tremor classification engine 112 can generate a confidence score that characterizes the confidence of the classification system 100 in the tremor classification 116 generated for the subject 102. The tremor classification engine 112 can generate the confidence score of the tremor classification 116 in any suitable way. For example, the tremor classification engine 112 can generate the confidence score by calculating the entropy of the score distribution 108 across the set of tremor classes. In this example, a higher entropy of the score distribution 108 across the set of tremor classes may reflect a higher uncertainty in the classification system 100 in the tremor classification.

[0069] The dyskinesia classification engine 118 is configured to process a score distribution 110 across a set of dyskinesia classes to generate a dyskinesia classification 118. For example, the dyskinesia classification engine 118 can generate a dyskinesia classification that classifies a subject as belonging to the dyskinesia class associated with the highest score under the score distribution 110 across the set of dyskinesia classes.

[0070] Optionally, the dyskinesia classification engine 118 can generate a confidence score that characterizes the confidence of the classification system 100 in the dyskinesia classification 118 generated for the subject 102. The dyskinesia classification engine 118 can generate the confidence score of the dyskinesia classification 118 in any suitable way. For example, the dyskinesia classification engine 118 can generate the confidence score by calculating the entropy of the score distribution over the set of dyskinesia classes 110. In this example, a higher entropy of the score distribution over the set of dyskinesia classes 110 may reflect a higher uncertainty in the classification system 100 in the dyskinesia classification.

[0071] The classification system 100 can use the tremor classification 116 generated for the subject 102 by any of the following methods. Some exemplary uses of the tremor classification 116 are described below.

[0072] In some embodiments, the classification system 100 can generate a notification indicating the tremor classification 116, which can, for example, be sent to subject 102 or the healthcare provider of subject 102. For example, the classification system 100 can send the notification over a data communication network, for example, as an email or text message.

[0073] In some embodiments, the classification system 100 can automatically store the tremor classification in the subject 102's electronic medical record. For example, the classification system 100 can interface with a database storing the subject 102's electronic medical record, for example, by an application programming interface (API), and store the tremor classification 116 in an appropriate field in the subject 102's electronic medical record.

[0074] In some embodiments, the classification system 100 receives accelerometer data 104 (and optionally other inputs such as video data or surface EMG data) for each time interval in a series of time intervals. The classification system 100 can process the accelerometer data 104 for each time interval to generate a corresponding tremor classification 116. Thus, the classification system 100 can generate a series of tremor classifications 116, each corresponding to a respective time interval and characterizing the severity of the tremor in the subject 102 over that time interval. The series of tremor classifications can provide real-time monitoring of the state and progression of the tremor in the subject 102 in detail without requiring the subject 102 to undergo regular medical examinations or utilize medical facilities.

[0075] The classification system 100 can process a series of tremor classifications to classify the progression of a given Parkinson's disease from a set of progression states into individual progression states. The set of progression states can include any appropriate number of progression states, for example, three, four, or five progression states, each progression state corresponding to a specific progression of the given Parkinson's disease. For example, the set of progression states may include an "early" progression state, an "intermediate" progression state, and an "late" progression state.

[0076] The classification system 100 can process a set of tremor classifications to classify the progression of the subject's Parkinson's disease in one of several ways. For example, the classification system 100 can process a set of tremor classifications to determine, for each tremor class, the number of times the subject's tremor was classified as belonging to that tremor class over a previous time window (e.g., a one-week time window). Thus, the classification system 100 can generate a frequency distribution across the set of tremor classes, which associates each tremor class with the number of times the subject's tremor was classified as belonging to that tremor class over a previous time window. The classification system 100 can then classify the progression of the subject's Parkinson's disease based on the frequency distribution across the set of tremor classes. For example, the classification system 100 can classify the progression of the subject's Parkinson's disease as "late" if the subject's tremor is classified into a tremor class associated with the highest severity of the tremor at least a threshold number of times.

[0077] The classification system 100 can generate treatment recommendations for subject 102 based on the classification of the progression of Parkinson's disease in subject 102. For example, each progression state in a set of progression states can be associated with a corresponding recommended treatment, such as a drug dosage. After classifying the progression of Parkinson's disease in subject 102 into progression states (based on a series of tremor classifications), the classification system 100 can generate recommendations for providing subject 102 with treatments related to the progression states. The classification system 100 can provide treatment recommendations to a user, such as a physician, via a user interface, such as a graphical user interface. Treatment can be applied to subject 102 at least in part based on the treatment recommendations generated by the classification system 100. (The treatment can be, for example, self-administered by the subject or administered to the subject by a physician or another third party).

[0078] The classification system 100 can enable an objective and reproducible evaluation of the effectiveness of treatments for Parkinson's disease, particularly those aimed at reducing the severity of a subject's tremors. More specifically, the classification system 100 can automatically generate a series of tremor classifications for a subject at any appropriate level of temporal granularity, for example, by classifying the severity of the subject's tremors every 10 seconds, every minute, or every hour. The series of tremor classifications generated for a subject receiving treatment (e.g., medication) provides an objective and reproducible metric for evaluating the effectiveness of the treatment, in contrast to self-reports by the subject, which can be inaccurate and inconsistent depending on the subject.

[0079] The classification system 100 can use the dyskinesia classification 118 generated for the subject 102 by any of the following methods. Some exemplary applications of the dyskinesia classification 118 are described below.

[0080] In some embodiments, the classification system 100 can generate a notification indicating a dyskinesia classification 118, which can, for example, notify subject 102 or the healthcare provider of subject 102. For example, the classification system 100 can send the notification over a data communication network, for example, as an email or text message.

[0081] In some embodiments, the classification system 100 can automatically store the dyskinesia classification in the subject 102's electronic medical record. For example, the classification system 100 can interface with a database storing the subject 102's electronic medical record, for example, by an application programming interface (API), and store the dyskinesia classification 118 in an appropriate field of the subject 102's electronic medical record.

[0082] In some embodiments, the classification system 100 receives accelerometer data 104 (and optionally other inputs such as video data or surface EMG data) for each time interval in a series of time intervals. The classification system 100 can process the accelerometer data 104 for each time interval to generate a corresponding dyskinesia classification 118. Thus, the classification system 100 can generate a series of dyskinesia classifications 118, each corresponding to a respective time interval and characterizing the severity of dyskinesia in the subject 102 over the time interval. The series of dyskinesia classifications can provide real-time monitoring of the state and progression of dyskinesia in the subject 102 in detail without requiring the subject 102 to regularly visit a doctor or use a hospital.

[0083] The classification system 100 can process a series of dyskinesia classifications to classify the progression of a target dyskinesia into a single progression state from a set of progression states. The set of progression states can include any appropriate number of progression states, for example, three, four, or five, and each progression state can correspond to each progression of the target dyskinesia. For example, the set of progression states may include an "early" progression state, a "mid" progression state, and a "late" progression state.

[0084] The classification system 100 can process a series of dyskinesia classifications to classify the progression of the dyskinesia of subject 102 in any of the following ways. For example, the classification system 100 can process a series of dyskinesia classifications to determine, for each dyskinesia class, the number of times over a previous time window (e.g., a one-week time window) that the subject's dyskinesia was classified as belonging to a dyskinesia class. Thus, the classification system 100 can generate a frequency distribution over the set of dyskinesia classes, which associates each dyskinesia class with the number of times the subject's dyskinesia was classified as belonging to a dyskinesia class over a previous time window. The classification system 100 can then classify the progression of the subject's dyskinesia based on the frequency distribution over the set of dyskinesia classes. For example, the classification system 100 can classify the progression of the subject's dyskinesia as "late" if the subject's dyskinesia is classified into a dyskinesia class that has been associated with the highest severity of dyskinesia at least a threshold number of times.

[0085] The classification system 100 can generate treatment recommendations for subject 102 based on the classification of the progression of dyskinesia in subject 102. For example, each progression state in a set of progression states can be associated with a corresponding recommended treatment, such as a drug dosage. After classifying the progression of dyskinesia in subject 102 into progression states (based on a series of dyskinesia classifications), the classification system 100 can generate recommendations for providing subject 102 with treatments related to the progression states. The classification system 100 can provide treatment recommendations to a user, such as a physician, via a user interface, such as a graphical user interface. Treatment can be applied to subject 102 at least in part based on the treatment recommendations generated by the classification system 100. (The treatment can be administered, for example, by the subject themselves, or by a physician or another third party.)

[0086] The classification system 100 can enable an objective and reproducible evaluation of the effectiveness of treatment for dyskinesia. More specifically, the classification system 100 can automatically generate a series of dyskinesia classifications for a subject at any appropriate level of temporal granularity by classifying the severity of the subject's dyskinesia, for example, every 10 seconds, every minute, or every hour. The series of dyskinesia classifications generated for a subject receiving treatment (e.g., medication) provides an objective and reproducible metric for evaluating the effectiveness of the treatment, in contrast to relying on self-reports by the subject, which can be inaccurate and inconsistent depending on the subject.

[0087] Figure 2 shows an exemplary architecture of a classification neural network 200 included in the classification system 100 described, for example, with reference to Figure 1. The classification neural network 200 is configured to receive a network input 106 which includes accelerometer data 104 that characterizes the movement of a subject 102 having Parkinson's disease. The network input 106 may include additional data that characterizes the subject, such as video data or surface EMG data, as described above with reference to Figure 1. The classification neural network 200 processes the network input 106 to generate one or both of the following: (i) a tremor classification 116 that characterizes the severity of the subject 102's tremor, or (ii) a dyskinesia classification 118 that characterizes the severity (and / or presence) of the subject 102's dyskinesia.

[0088] The classification neural network 200 includes an encoder subnetwork 300 and one or both of the following: (i) a tremor subnetwork 204 and (ii) a dyskinesia subnetwork 206, which are described in detail below.

[0089] The encoder subnetwork 300 is configured to process the network input 106 in order to generate an embedding 202 of the network input 106 in latent space. The encoder subnetwork 300 may have any suitable neural network architecture that enables the encoder subnetwork 300 to generate the embedding 202 of the network input 106. In particular, the encoder subnetwork may include any suitable type of neural network layer (e.g., convolutional layers, recurrent layers, attention layers, fully connected layers, etc.) in any number (e.g., 5, 10, or 50 layers) and in any suitable configuration (e.g., as a linear series of layers). The architecture of a specific example of the encoder subnetwork 300 will be described in more detail with reference to Figure 3.

[0090] The network input 106 may include data from multiple modalities, such as accelerometer data, video data, and surface EMG data. The architecture of the encoder subnetwork 300 can be configured in any of several possible ways to enable the encoder subnetwork 300 to process multimodal data. For example, the encoder subnetwork 300 may include a set of encoder neural network layers, each corresponding to a modality. For each modality, the encoder subnetwork 300 can process the data derived from that modality using the corresponding set of encoder neural network layers to generate a modality-specific embedding of the data. The encoder subnetwork 300 can combine the modality-specific embeddings of each modality included in the network input 106 to generate an embedding 202 of the network input 106. The encoder subnetwork 300 can combine modality-specific embeddings, for example, by concatenating them or by pooling them (e.g., averaging, summing, or max pooling).

[0091] The tremor subnetwork 204 is configured to process the embedding 202 of the network input 106 to generate a score distribution 108 across a set of tremor classes. The tremor subnetwork 204 may have any suitable neural network architecture that enables the tremor subnetwork 204 to generate a score distribution across a set of tremor classes. In particular, the tremor subnetwork may include any suitable type of neural network layer (e.g., convolutional layers, recurrent layers, attention layers, fully connected layers, etc.) in any number (e.g., 5, 10, or 50 layers) and in any suitable configuration (e.g., as a linear series of layers). In a particular example, the tremor subnetwork 204 may include a series of dense (fully connected) neural network layers.

[0092] The dyskinesia subnetwork 206 is configured to process the embedding 202 of the network input 106 to generate a score distribution 110 across a set of dyskinesia classes. The dyskinesia subnetwork 206 can have any suitable neural network architecture that enables the dyskinesia subnetwork 206 to generate a score distribution across a set of dyskinesia classes. In particular, the dyskinesia subnetwork may include any suitable type of neural network layer (e.g., convolutional layers, recurrent layers, attention layers, fully connected layers, etc.) in any number (e.g., 5, 10, or 50 layers) and in any suitable configuration (e.g., as a linear series of layers). In a particular example, the dyskinesia subnetwork 206 may include a series of dense (fully connected) neural network layers.

[0093] In some embodiments, the classification neural network 200 includes an encoder subnetwork 300 and a tremor subnetwork 204, but does not include a dyskinesia subnetwork 206. In some embodiments, the classification neural network 200 includes an encoder subnetwork 300 and a dyskinesia subnetwork 206, but does not include a tremor subnetwork 204. In some embodiments, the classification neural network 200 includes both a tremor subnetwork 204 and a dyskinesia subnetwork 206.

[0094] Figure 3 shows an exemplary architecture of an encoder subnetwork 300 included in a classification neural network 200, as described with reference to Figure 1, for example, of a classification system 100, as described with reference to Figure 1, for example. In this example, the encoder subnetwork 300 includes (i) a convolutional block 302 containing one or more convolutional neural network layers, and (ii) a recurrent block 306 containing one or more recurrent neural network layers.

[0095] The convolutional block 302 can be configured to process accelerometer data 104 characterizing the motion of an object with Parkinson's disease by one or more convolutional neural network layers in order to produce a convolutional block output having a lower temporal resolution than the accelerometer data. In particular, the accelerometer data may include a series of accelerometer data elements, for example, each accelerometer data element being represented as the acceleration of a 3D vector of x, y, and z vectors. The convolutional block output may include a series of feature vectors, the series of feature vectors having a shorter length than the series of accelerometer data elements, and therefore having a lower temporal resolution than the series of accelerometer data elements.

[0096] The recurrent block 306 can be configured to process the convolutional block output 304 by one or more recurrent neural network layers to generate an embedding 202 of the accelerometer data 104. The recurrent neural network layers can be, for example, long short-term memory (LSTM) recurrent neural network layers, gated recurrent unit (GRU) recurrent neural network layers, or any other suitable type of recurrent neural network layer. The recurrent neural network layers in the recurrent block 306 can be arranged sequentially. Each recurrent neural network layer can be configured to sequentially process a set of feature vectors to generate an updated set of feature vectors by a set of operations of the recurrent neural network layer. The first recurrent neural network layer of the recurrent block 306 can receive a set of feature vectors contained in the output of the convolutional block. Each subsequent recurrent neural network layer of the recurrent block 306 can receive a set of feature vectors generated by the preceding recurrent neural network layer of the recurrent block. The final recurrent neural network layer of the recurrent block 306 can generate a set of feature vectors that collectively define the embeddings 202 of the accelerometer data 104.

[0097] Figure 4 is a flowchart of an exemplary process 400 for classifying tremors in subjects with Parkinson's disease, or for classifying dyskinesia in subjects with Parkinson's disease. For convenience, process 400 is described as being performed by one or more computer systems located in one or more locations. For example, a classification system, such as the classification system 100 in Figure 1 as appropriately programmed according to this specification, can perform process 400.

[0098] The system receives accelerometer data characterizing the movement of an object with Parkinson's disease (402). The accelerometer data can be generated by the object's wearable device. The accelerometer data can be represented as one or more one-dimensional (1D) time signals or as two-dimensional (2D) spectrogram signals. Optionally, the system may receive additional data, such as video data characterizing the object's movement or surface EMG data characterizing the potentials generated during the object's muscle contractions.

[0099] The system uses a classification neural network to process network inputs, including accelerometer data (and optionally additional data such as video data or surface EMG data), according to the values ​​of a set of classification neural network parameters, to generate a score distribution across a set of tremor classes, or a score distribution across a set of dyskinesia classes, or both (404). A set of tremor classes includes multiple tremor classes, where each tremor class corresponds to a severity of each tremor. A set of dyskinesia classes includes multiple dyskinesia classes, where each dyskinesia class corresponds to a severity of each dyskinesia.

[0100] The system classifies the subject's tremor, dyskinesia, or both (406). The system can classify the subject's tremor based on the score distribution across a set of tremor classes, for example, by classifying the subject's tremor as belonging to the tremor class with the highest score. The system can classify the subject's dyskinesia based on the score distribution across a set of dyskinesia classes, for example, by classifying the subject's dyskinesia as belonging to the dyskinesia class with the highest score.

[0101] Figure 5 shows an exemplary training system 500. The training system 500 is an example of a system implemented as a computer program on one or more computers in one or more locations where the systems, components, and techniques described below are implemented.

[0102] The training system 500 is configured to train a diagnostic machine learning model 200 included in the classification system 100, as described with reference to Figure 1. The classification neural network 200 is configured to process network inputs, including accelerometer data characterizing the movements of subjects with Parkinson's disease, to generate one or both of the following: (i) a score distribution across a set of tremor classes, and (ii) a score distribution across a set of dyskinesia classes, as described with reference to Figure 1.

[0103] The classification neural network 200 includes an encoder subnetwork and (i) a tremor subnetwork, and (ii) a dyskinesia subnetwork, which may include one or both of the following, as described with reference to Figure 2: The encoder subnetwork can process the network input, which includes accelerometer data, to generate an embedding of the network input. The tremor subnetwork can process the embedding of the network input to generate a score distribution across a set of tremor classes. The dyskinesia subnetwork can process the embedding of the network input to generate a score distribution across a set of dyskinesia classes.

[0104] The training system 500 can train a classification neural network in a series of two stages. In the first stage, the training system 500 can pre-train the encoder subnetwork 300 of the classification neural network 200 to perform one or more auxiliary tasks. In the second stage, the training system 500 can train the classification neural network 200 to perform a tremor classification task, or a dyskinesia classification task, or both. Pre-training of the encoder subnetwork 300 can result in an effective initialization of the parameter values ​​of the encoder neural network, thus facilitating the fine-tuning of the classification neural network for performing tremor classification or dyskinesia classification. In particular, pre-training can prompt the encoder subnetwork 300 to generate embeddings that encode rich features characterizing the structure of accelerometer data (and optionally, other types of data, e.g., video data or surface EMG data). Pre-training the encoder subnetwork 300 allows the classification neural network 200 to achieve higher predictive accuracy for tremor classification and dyskinesia classification tasks, while requiring less training data than would otherwise be necessary.

[0105] In the first stage of training, the training system 500 can initialize the set of parameters of the encoder subnetwork 300 to, for example, random values. The training system 500 can then train the encoder subnetwork 300 to perform auxiliary tasks. Several examples of auxiliary tasks are described below.

[0106] In some embodiments, the training system 500 can pre-train the encoder subnetwork 300 to perform a symmetric embedding task 502. More specifically, the training system 500 can train the encoder subnetwork to generate similar embeddings of “positive” pairs of accelerometer signals and different embeddings of “negative” pairs of accelerometer signals. A “pair” of accelerometer signals includes a first accelerometer signal and a second accelerometer signal. A “positive” pair of accelerometer signals can refer to a pair of accelerometer signals where both the first and second accelerometer signals are transformed versions of the same underlying accelerometer signal. A “negative” pair of accelerometer signals can refer to a pair of accelerometer signals where the first accelerometer signal is derived from a different accelerometer signal than the second accelerometer signal. An exemplary process for training the encoder subnetwork 300 to perform a symmetric embedding task 502 is described in more detail with reference to Figure 6.

[0107] In some embodiments, the training system 500 can pre-train the encoder subnetwork 300 to perform a masked reconstruction task 504. More specifically, the training system can train the encoder subnetwork to process a masked accelerometer signal to generate an embedding that enables accurate reconstruction of the complete (unmasked) accelerometer signal. An exemplary process for training the encoder subnetwork 300 to perform the masked reconstruction task 504 is described in more detail with reference to Figure 7.

[0108] In some embodiments, the training system 500 can pre-train the encoder subnetwork 300 to perform a noisy reconstruction task 506. More specifically, the training system can train the encoder subnetwork to process a noisy accelerometer signal to generate an embedding that enables accurate reconstruction of the original (de-noised) accelerometer signal. An exemplary process for training the encoder subnetwork 300 to perform the noisy reconstruction task 506 is described in more detail with reference to Figure 8.

[0109] In some embodiments, the training system 500 can pre-train the encoder subnetwork 300 to perform one or more supervised aid tasks 508. More specifically, the training system can train the encoder subnetwork to process accelerometer signals to generate embeddings that enable accurate prediction of one or more features of the accelerometer signals, such as the number of steps taken by the subject during the duration covered by the accelerometer signals, or the actions performed by the subject during the duration covered by the accelerometer signals. An exemplary process for training the encoder subnetwork to perform a supervised aid task 508 is described in more detail with reference to Figure 9.

[0110] Optionally, the training system 500 can pre-train the encoder subnetwork to perform multiple auxiliary tasks, that is, not just a single auxiliary task. For example, the training system 500 can pre-train the encoder subnetwork to perform two auxiliary tasks, three auxiliary tasks, or four auxiliary tasks.

[0111] In the second stage of training, the training system 500 can initialize the parameter sets of the tremor subnetwork, the dyskinesia subnetwork, or both, to, for example, random values. (The parameter set of the encoder subnetwork 300 may have values ​​determined during the pre-training of the encoder subnetwork in the first stage of training.) The training system 500 can then train the classification neural network 200 to perform the tremor classification task 510, or the dyskinesia classification task 512, or both. An exemplary process for training the classification neural network to perform the tremor classification task 510 is described in more detail with reference to Figure 10. An exemplary process for training the classification neural network to perform the dyskinesia classification task is described in more detail with reference to Figure 11.

[0112] In some embodiments, the training system 500 trains a classification neural network 200 to perform both tremor classification and dyskinesia classification tasks. In these embodiments, the training system 500 can leverage the synergies between the tremor classification task and the dyskinesia classification task to achieve higher predictive accuracy in both tasks. More specifically, as part of training the classification neural network to perform the tremor classification task, the training system can backpropagate the gradient of the tremor classification objective function through the tremor subnetwork to the encoder subnetwork of the classification neural network. Similarly, as part of training the classification neural network to perform the dyskinesia classification task, the training system can backpropagate the gradient of the dyskinesia classification objective function through the dyskinesia subnetwork to the encoder subnetwork of the classification neural network. Thus, the parameter values ​​of the encoder subnetwork can be jointly trained using training signals from both the tremor classification task and the dyskinesia classification task, thereby enabling the encoder subnetwork to learn to leverage the synergies and commonalities between the tasks.

[0113] For convenience, the first stage of training—in particular, the pre-training of the encoder neural network—is described as taking place before the second stage of training—in particular, the fine-tuning of the classification neural network for tremor classification and / or dyskinesia classification. However, the two stages of training can overlap, for example, so that both the pre-training of the encoder neural network and the fine-tuning of the classification neural network are performed in one or more training iterations.

[0114] After training the classification neural network 200, the training system can be equipped with the classification neural network 200 for use by the classification system 100, as described with reference to Figure 1.

[0115] Figure 6 is a flowchart of an exemplary process 600 for pre-training an encoder subnetwork of a classification neural network to perform a controlled embedding task. For convenience, process 600 is described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 500 of Figure 5, appropriately programmed according to this specification, can perform process 600. The training system can iteratively perform the steps of process 600 as part of the pre-training of the encoder subnetwork.

[0116] The system acquires a set of "base" accelerometer signals (602).

[0117] The system generates one or more pairs of positive accelerometer signals (604). To generate pairs of positive accelerometer signals, the system selects base accelerometer signals from a set of base accelerometer signals (e.g., by random sampling). The system generates a first transformed version of the base accelerometer signal by, for example, randomly sampling a first transformation from the space of transformations and applying the first transformation to the base accelerometer signal. The system generates a second transformed version of the base accelerometer signal by, for example, randomly sampling a second transformation from the space of transformations and applying the second transformation to the base accelerometer signal. The first and second transformed versions of the base accelerometer signal together define a pair of positive accelerometer signals.

[0118] The transformation space may include, for example, flip transforms (including flipping the accelerometer signal around the time axis), inversion transforms (including reversing the time axis direction of the accelerometer signal), zoom transforms (including cropping and resizing the accelerometer signal), swap transforms (including swapping the order of the accelerometer signals), noise transforms (including adding random noise to the accelerometer signal), resize transforms (including changing the amplitude of the accelerometer signal), and identity transforms (which do not affect the accelerometer signal). In some cases, a positive pair of accelerometer signals may include accelerometer signals identical to the base accelerometer signal, for example, in situations where the system selects an identity transform to apply to the base accelerometer signal.

[0119] The system generates one or more positive pair embeddings (606). In particular, for each positive pair of accelerometer signals, the system processes the first accelerometer signal using an encoder subnetwork to generate a first accelerometer signal embedding, and the system processes the second accelerometer signal to generate a second accelerometer signal embedding. The first accelerometer signal embedding and the second accelerometer signal embedding together define a positive pair embedding.

[0120] The system generates one or more pairs of negative accelerometer signals (608). To generate pairs of negative accelerometer signals, the system selects a pair of base accelerometer signals that include a first base accelerometer signal and a second different base accelerometer signal (e.g., by random sampling). The system generates a transformed version of the first base accelerometer signal, for example, by randomly sampling a transform from a space of transforms and applying the transform to the first base accelerometer signal. The system generates a transformed version of the second base accelerometer signal, for example, by randomly sampling a transform from a space of transforms and applying the transform to the second base accelerometer signal. The transformed versions of the first and second base accelerometer signals together define a pair of negative accelerometer signals. The system may select the identity transform of the first or second base accelerometer signal such that the pair of negative accelerometer signals include the base accelerometer signals (rather than the modified accelerometer signals).

[0121] The system generates one or more negative pair embeddings (610). Specifically, for each negative pair of accelerometer signals, the system processes the first accelerometer signal using an encoder subnetwork to generate a first accelerometer signal embedding, and the system processes the second accelerometer signal to generate a second accelerometer signal embedding. The first accelerometer signal embedding and the second accelerometer signal embedding together define a negative pair embedding.

[0122] The system trains an encoder subnetwork to optimize an auxiliary similarity loss that depends on positive pair embeddings and negative pair embeddings (612). More specifically, the auxiliary loss can measure the error (e.g., Euclidean distance) between a first embedding and a second embedding for each pair embedding. In particular, for each positive pair embedding, the auxiliary loss can promote a higher similarity between the first and second embeddings. For each negative pair embedding, the auxiliary loss can promote a lower similarity between the first and second embeddings.

[0123] Figure 7 is a flowchart of an exemplary process 700 for pre-training an encoder subnetwork of a classification neural network to perform a masked reconstruction task. For convenience, process 700 is described as being performed by one or more computer systems located in one or more locations. For example, a training system, e.g., the training system 500 of Figure 5, appropriately programmed according to this specification, can perform process 700. The training system can iteratively perform the steps of process 700 as part of the pre-training of the encoder subnetwork.

[0124] The system acquires the accelerometer signal (702).

[0125] The system generates a masked accelerometer signal by masking a portion of the accelerometer signal (704). The system can mask a portion of the accelerometer signal by replacing that portion with default data, for example, a predetermined value (e.g., a value of 0) or random noise (e.g., sampled from a normal distribution). The system can randomly select the portion of the accelerometer signal to be masked.

[0126] The system uses an encoder subnetwork to process the masked accelerometer signals and generate an embedding of the masked accelerometer signals (706).

[0127] The system uses a decoder neural network to process the embedding of the masked accelerometer signal to generate a predictive reconstruction of the accelerometer signal (708). The decoder neural network may have any suitable neural network architecture that enables the decoder neural network to generate a predictive reconstruction of the accelerometer signal. In particular, the decoder neural network may include any suitable type of neural network layer (e.g., convolutional layers, recurrent layers, attention layers, fully connected layers, etc.) in any number (e.g., 5, 10, or 50 layers) and in any suitable configuration (e.g., as a linear series of layers).

[0128] The system trains both an encoder subnetwork and a decoder neural network to optimize an auxiliary reconstruction loss that measures the error in predictive reconstruction of the accelerometer signal (710). The error can be measured, for example, as an L1 error, or as an L2 error, or using any other appropriate error metric.

[0129] Figure 8 is a flowchart of an exemplary process 800 for pre-training an encoder subnetwork of a classification neural network to perform a noisy reconstruction task. For convenience, process 800 is described as being performed by one or more computer systems located in one or more locations. For example, a training system, e.g., training system 500 of Figure 5 as appropriately programmed according to this specification, can perform process 800. The training system can iteratively perform the steps of process 800 as part of the pre-training of the encoder subnetwork.

[0130] The system acquires the accelerometer signal (802).

[0131] The system generates a noisy accelerometer signal by adding noise to the accelerometer signal (804). The system can sample noise from a predetermined probability distribution, such as a normal distribution.

[0132] The system uses an encoder subnetwork to process the noisy accelerometer signal and generate an embedding of the noisy accelerometer signal (806).

[0133] The system uses a decoder neural network to process the embedding of the noisy accelerometer signal to generate a predicted denoised accelerometer signal, i.e., a predicted reconstruction of the original (denoised) accelerometer signal (808). The decoder neural network may have any suitable neural network architecture that enables the decoder neural network to generate the predicted denoised accelerometer signal. In particular, the decoder neural network may include any suitable type of neural network layer (e.g., convolutional layers, recurrent layers, attention layers, fully connected layers, etc.) in any number (e.g., 5, 10, or 50 layers) and in any suitable configuration (e.g., as a linear series of layers).

[0134] The system trains an encoder subnetwork and a decoder neural network together to optimize an auxiliary denoising loss that measures the error in the denoised accelerometer signal (810). More specifically, the auxiliary denoising loss can measure the error between (i) the original accelerometer signal and (ii) the predicted denoised accelerometer signal. The error can be measured, for example, as an L1 error, or as an L2 error, or using any other appropriate error metric.

[0135] Figure 9 is a flowchart of an exemplary process 900 for pre-training an encoder subnetwork of a classification neural network to perform a supervised aid task. For convenience, process 900 is described as being performed by one or more computer systems located in one or more locations. For example, a training system, e.g., training system 500 of Figure 5 as appropriately programmed according to this specification, can perform process 900. The training system can iteratively perform the steps of process 900 as part of the pre-training of the encoder subnetwork.

[0136] The system obtains (i) an accelerometer signal and (ii) a target label for the accelerometer signal (902). In some embodiments, the target label for the accelerometer signal defines the number of steps taken by the subject during the duration covered by the accelerometer signal. In some embodiments, the target label for the accelerometer signal defines the action performed by the subject during the duration covered by the accelerometer signal.

[0137] The system uses an encoder subnetwork to process the accelerometer signal and generate an embedding of the accelerometer signal (904).

[0138] The system uses a predictive neural network to process the embedding of the accelerometer signal and generate a predictive output that characterizes the predicted labels of the accelerometer signal (906). In some embodiments, the predictive output directly defines the predicted labels. In some embodiments, the predictive output defines a score distribution across a set of possible labels.

[0139] The system trains both an encoder subnetwork and a predictive neural network and optimizes an auxiliary supervised loss that measures the error between (i) a target label and (ii) a predicted output characterizing the predicted label (908). The auxiliary supervised loss can measure the error, for example, as a cross-entropy error or as a squared error.

[0140] Figure 10 is a flowchart of an exemplary process 1000 for training a classification neural network to perform a tremor classification task. For convenience, process 1000 is described as being performed by one or more computer systems located in one or more locations. For example, a training system, such as the training system 500 in Figure 5, appropriately programmed according to this specification, can perform process 1000. The training system can iteratively perform the steps of process 1000 as part of training the classification neural network.

[0141] The system acquires (i) training accelerometer signals and (ii) target tremor classification of the training accelerometer signals (1002).

[0142] The system processes the training accelerometer signals using an embedded subnetwork to generate an embedding of the training accelerometer signals (1004).

[0143] The system uses a tremor subnetwork to process the embedding of training accelerometer signals and generate a score distribution across a set of tremor classes (1006).

[0144] The system trains a tremor subnetwork of a classification neural network to optimize a tremor objective function that measures the error between (i) a score distribution across a set of tremor classes, and (ii) a target tremor classification (1008). The tremor objective function can measure the error, for example, as cross-entropy error.

[0145] The system determines the gradient of the tremor objective function, for example, using backpropagation, and then uses the gradient to adjust the values ​​of the set of parameters of the tremor subnetwork by appropriate gradient descent optimization techniques, such as RMSprop or Adam update rules. That is, the system backpropagates the gradient of the tremor objective function through the tremor subnetwork. In some embodiments, the system trains both the tremor subnetwork and the encoder subnetwork by backpropagating the gradient through the tremor subnetwork to the encoder subnetwork. In other embodiments, the system freezes the parameter values ​​of the encoder subnetwork after pretraining the encoder subnetwork to perform one or more auxiliary tasks, and does not adjust the parameter values ​​of the encoder subnetwork to optimize the tremor objective function.

[0146] Figure 11 is a flowchart of an exemplary process 1100 for training a classification neural network to perform a dyskinesia classification task. For convenience, process 1100 is described as being performed by one or more computer systems located in one or more locations. For example, a training system, such as the training system 500 in Figure 5, appropriately programmed according to this specification, can perform process 1100. The training system can iteratively perform the steps of process 1100 as part of training the classification neural network.

[0147] The system acquires (i) the training accelerometer signal and (ii) the target dyskinesia classification of the training accelerometer signal (1102).

[0148] The system processes the training accelerometer signals using an embedded subnetwork to generate an embedding of the training accelerometer signals (1104).

[0149] The system uses a dyskinesia subnetwork to process the embedding of training accelerometer signals and generate a score distribution across a set of dyskinesia classes (1106).

[0150] The system trains a dyskinesia subnetwork of a classification neural network to optimize a tremor objective function that measures the error between (i) a score distribution across a set of dyskinesia classes, and (ii) a target dyskinesia classification (1108). The dyskinesia objective function can measure the error, for example, as cross-entropy error.

[0151] The system determines the gradient of the dyskinesia objective function, for example, using backpropagation, and then uses the gradient to adjust the values ​​of the set of parameters in the dyskinesia subnetwork by an appropriate gradient descent optimization technique, such as RMSprop or Adam update rules. That is, the system backpropagates the gradient of the dyskinesia objective function through the dyskinesia subnetwork. In some embodiments, the system trains the dyskinesia subnetwork and the encoder subnetwork together by backpropagating the gradient into the encoder subnetwork through the dyskinesia subnetwork. In other embodiments, the system freezes the parameter values ​​of the encoder subnetwork after pretraining the encoder subnetwork to perform one or more auxiliary tasks, and does not adjust the parameter values ​​of the encoder subnetwork to optimize the dyskinesia objective function.

[0152] Figure 12 shows an example of experimental results for a classification system used to classify tremors in subjects with Parkinson's disease.

[0153] Figure 13 shows (i) a wearable device 1302 worn by, for example, a person with Parkinson's disease, (ii) accelerometer data generated by the wearable device's accelerometer, represented as one or more 1D time signals 1304, and (iii) accelerometer data generated by the accelerometer in the wearable device, represented as a 2D time-frequency spectrogram 1306.

[0154] In this specification, the term “configured” is used in relation to systems and computer program components. For a system of one or more computers, being configured to perform a particular operation or action means that software, firmware, hardware, or a combination thereof is installed on the system to cause the system to perform that operation or action while it is in operation. For one or more computer programs to be configured to perform a particular operation or action means that the one or more computer programs contain instructions that, if executed by a data processing device, cause that device to perform that operation or action.

[0155] Embodiments of subject matter and functional operation described herein may be implemented in digital electronic circuits, tangibly implemented computer software or firmware, computer hardware including structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of subject matter described herein may be implemented as one or more modules of computer programs, i.e., computer program instructions coded in a tangible non-temporary storage medium for execution by or control of the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be coded in artificially generated propagating signals, such as mechanically generated electrical, optical, or electromagnetic signals generated to code information for transmission to a suitable receiving device to be executed by a data processing device.

[0156] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, equipment, and machines for data processing, including, for example, a programmable processor, a computer, or multiple processors or computers. A device may also be, or further include, dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Optionally, in addition to hardware, a device may include code that forms the execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or one or more combinations thereof.

[0157] Computer programs, also called or written as programs, software, software applications, apps, modules, software modules, scripts, or code, may be written in any form of programming language, including compilers or interpreters, or declarative or procedural languages, and may be deployed as standalone programs or modules, in any form, including components, subroutines, or other units, suitable for use in a computer environment. Programs may or may not correspond to files in a file system. Programs may be stored in part of a file that holds other programs or data, in one or more scripts stored, for example, in a markup language document, in a single file dedicated to the program of interest, or in multiple collaborative files, for example, in a file that stores one or more modules, subprograms, or parts of code. Computer programs may be deployed to run on one computer, or on multiple computers located in one place or distributed across multiple locations and interconnected by a data communication network.

[0158] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components and installed on one or more computers located in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines can be installed and operated on the same computer.

[0159] The processes and logic flows described herein may be executed by one or more programmable computers that run one or more computer programs to perform functions by acting on input data and generating outputs. The processes and logic flows may also be executed by dedicated logic circuits, such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmed computers.

[0160] A computer suitable for running computer programs may be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory or random-access memory, or both. Essential elements of a computer are a central processing unit that executes or carries out instructions, and one or more memory devices that store instructions and data. The central processing unit and memory may be reinforced or incorporated by dedicated logic circuits. Generally, a computer will also include one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, or be operable to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer may be incorporated into another device, to name just a few, such as a mobile phone, personal digital assistant (PDA), mobile voice or video player, game console, Global Positioning System (GPS) receiver, or portable storage device, such as a Universal Serial Bus (USB) flash drive.

[0161] Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM and EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and all forms of non-volatile memory, media, and memory devices, including CD-ROM and DVD-ROM disks.

[0162] To enable interaction with the user, the subject matter described herein may be implemented in a computer having a display device that displays information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and an indicator device that allows the user to provide input to the computer, such as a keyboard and mouse or trackball. Other types of devices can also be used to enable interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, voice, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents with the user's device, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and then receiving response messages from the user.

[0163] The data processing unit implementing the machine learning model may also include, for example, a dedicated hardware accelerator unit to handle the general and computationally intensive parts of machine learning training or production, i.e., inference, workload.

[0164] Machine learning models may be implemented and deployed using machine learning frameworks, such as the TensorFlow framework or the Jax framework.

[0165] Multiple embodiments of the subject matter described herein may be implemented in a computing system that includes, for example, a backend element as a data server, or a middleware element, such as an application server, or a client computer equipped with a frontend element, such as a graphical user interface, web browser, or app that allows a user to interact with an implementation of the subject matter described herein, or any combination of one or more such backend, middleware, or frontend elements. The elements of the system may be interconnected by digital data communications of any form or medium, such as communication networks. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the internet.

[0166] A computing system may include a client and a server. The client and server are generally located remotely from each other and typically interact via a communication network. The client-server relationship arises from computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server transmits data, such as an HTML page, to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving input from the user. Data generated on the user device, such as the results of the interaction with the user, can be received from that device on the server side.

[0167] This specification contains many specific implementation details, but these should not be interpreted as limiting the scope of any invention or claim, but rather as descriptions of features specific to a particular embodiment of a particular invention. Features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately or in any suitable partial combination in multiple embodiments. Furthermore, while multiple features may be described above as operating in a particular way in combination, and may be described in the claims from the outset as such, one or more features from a claimed combination may, in some cases, be excluded from that combination, and the claimed combination may refer to a partial combination or a variation of a partial combination.

[0168] Similarly, while operations are shown in the drawings and enumerated in the claims in a specific order, it should not be understood that such operations must be performed in a specific or sequential order illustrated, or that all illustrated operations must be performed, in order to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged in multiple software products.

[0169] We have described specific embodiments of the subject matter. Other embodiments are also included in the scope of the following claims. For example, the desired results can be achieved by performing the operations mentioned in the claims in a different order. As an example, the processes shown in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method that is performed by one or more computers, Receiving accelerometer data characterizing the movement of a subject with Parkinson's disease, Processing the accelerometer data using a tremor classification neural network to generate a score distribution across a set of tremor classes according to the set parameter values ​​of the tremor classification neural network, The aforementioned set of tremor classes includes multiple tremor classes, Each tremor class in the aforementioned set of tremor classes is to be treated in accordance with the respective severity of the tremor, and A method comprising classifying the tremors of the subject having Parkinson's disease based on the score distribution across the set of tremor classes.

2. The method according to claim 1, wherein the accelerometer data is generated by the wearable device in question.

3. The method according to claim 1 or 2, wherein the accelerometer data is represented as one or more one-dimensional (1D) time signals.

4. The method according to claim 1 or 2, wherein the accelerometer data is represented as a two-dimensional (2D) spectrogram signal.

5. The method according to any one of claims 1 to 4, wherein the set of tremor classes comprises at least three tremor classes.

6. The method according to any one of claims 1 to 5, wherein the set of tremor classes corresponds to a clinical tremor scale.

7. The method according to any one of claims 1 to 6, wherein each tremor class in the set of tremor classes corresponds to the magnitude, frequency, and duration of each tremor.

8. The further includes receiving video data characterizing the movement of the aforementioned object, Processing the accelerometer data using the tremor classification neural network is The method according to any one of claims 1 to 7, comprising processing the accelerometer data and the video data together using the tremor classification neural network.

9. The method further includes receiving surface electromyography (EMG) data that characterizes the potential generated during muscle contraction of the subject, Processing the accelerometer data using the tremor classification neural network is The method according to any one of claims 1 to 8, comprising processing the accelerometer data and the surface EMG data together using the tremor classification neural network.

10. Using the tremor classification neural network, the accelerometer data is processed to generate the score distribution across the set of tremor classes. To generate a convolutional block output, the accelerometer data is processed using a convolutional block that includes one or more convolutional neural network layers. The process of the convolutional block output using a recurrent block containing one or more recurrent neural network layers to generate a recurrent block output, and The method according to any one of claims 1 to 9, comprising processing the recurrent block output using a high-density block including one or more high-density neural network layers to generate the score distribution across the set of tremor classes.

11. The method according to claim 10, wherein the convolutional block output has a lower temporal resolution than the accelerometer data.

12. The method according to claim 10 or 11, wherein the recurrent neural network layer is a long-term short-term memory (LSTM) neural network layer.

13. Classifying the tremors of the subject having Parkinson's disease based on the score distribution across the set of tremor classes, The method according to any one of claims 1 to 12, comprising classifying the tremor in question into the tremor class associated with the highest score from among the set of tremor classes.

14. The method according to any one of claims 1 to 13, further comprising repeatedly performing the tremor classification over a series of time intervals to generate a series of tremor classifications, wherein each tremor classification in the series of tremor classifications corresponds to a respective time interval, and the tremor of the subject is classified and generated during the time interval.

15. The method according to claim 14, further comprising processing the series of tremor classifications to classify the progression of the subject's Parkinson's disease into a single progression from a set of progression states.

16. The method according to claim 15, further comprising administering a drug for treating Parkinson's disease or symptoms of Parkinson's disease to the subject, based at least in part on the progression of Parkinson's disease in the subject.

17. The method according to any one of claims 1 to 16, further comprising generating a notification indicating the classification of the tremor in the subject having Parkinson's disease.

18. Using the tremor classification neural network, the accelerometer data is processed to generate the score distribution across the set of tremor classes. The encoder subnetwork of the tremor classification neural network is used to process the accelerometer data to generate an embedding of the accelerometer data in latent space, and The method according to any one of claims 1 to 17, comprising processing the embedding of the accelerometer data in the latent space using a tremor subnetwork of the tremor classification neural network to generate the score distribution across the set of tremor classes.

19. The method further includes processing the embedding of the accelerometer data in the latent space using the dyskinesia subnetwork of the tremor classification neural network to generate a score distribution across a set of dyskinesia classes, The set of dyskinesia classes includes multiple dyskinesia classes, The method according to claim 18, wherein each dyskinesia class in the set of dyskinesia classes corresponds to the respective severity of dyskinesia.

20. The method according to claim 19, further comprising classifying the dyskinesia of the subject having Parkinson's disease based on the score distribution across the set of dyskinesia classes.

21. The aforementioned tremor classification neural network is Pre-training the encoder subnetwork of the tremor classification neural network to perform an auxiliary task, wherein the auxiliary task is not a tremor classification task, and The method according to any one of claims 18 to 20, wherein the tremor classification neural network is trained by an operation that includes pre-training the encoder subnetwork and then training the tremor classification neural network to perform the tremor classification task.

22. Pre-training the encoder subnetwork of the tremor classification neural network in order to perform the aforementioned auxiliary task is To acquire the base accelerometer signal, This involves generating positive pair embeddings in the latent space, Using the encoder subnetwork, process a first transformed version of the base accelerometer signal to generate an embedding corresponding to the latent space. The process includes processing a second transformed version of the base accelerometer signal using the encoder subnetwork to generate an embedding corresponding to the latent space. The method according to claim 21, comprising training the encoder subnetwork to optimize auxiliary losses that depend on (i) the embedding of the first converted version of the base accelerometer signal and (ii) the embedding of the second converted version of the base accelerometer signal.

23. To generate the first converted version of the base accelerometer signal, Randomly sampling a first transformation from the transformation space, and The method according to claim 22, further comprising applying the first transformation to the base accelerometer signal in order to generate the first transformed version of the base accelerometer signal.

24. To generate the second converted version of the base accelerometer signal, Randomly sampling a second transformation from the transformation space, and The method according to claim 22 or 23, further comprising generating, which includes applying the second transformation to the base accelerometer signal in order to generate the second transformed version of the base accelerometer signal.

25. The method according to any one of claims 22 to 24, wherein the auxiliary loss measures the error between (i) the embedding of the first converted version of the base accelerometer signal and (ii) the embedding of the second converted version of the base accelerometer signal.

26. Pre-training the encoder subnetwork of the tremor classification neural network in order to perform the aforementioned auxiliary task is To acquire accelerometer signals, By masking a portion of the aforementioned accelerometer signal, a masked accelerometer signal is generated. Processing the masked accelerometer signal using the encoder subnetwork to generate the embedding of the masked accelerometer signal, Using a decoder neural network to process the embedding of the masked accelerometer signal to generate a predictive reconstruction of the accelerometer signal, and The method according to any one of claims 21 to 25, comprising training the encoder subnetwork and the decoder neural network together to optimize an auxiliary loss for measuring the error of the predictive reconstruction of the accelerometer signal.

27. Pre-training the encoder subnetwork of the tremor classification neural network in order to perform the aforementioned auxiliary task is To acquire accelerometer signals, By adding the aforementioned accelerometer signal with a lot of noise, a noisy accelerometer signal is generated. The encoder subnetwork is used to process the noisy accelerometer signal to generate an embedding of the noisy accelerometer signal. Processing the embedding of the noisy accelerometer signal in order to generate a denoised accelerometer signal using a decoder neural network, The method according to any one of claims 21 to 26, comprising training the encoder subnetwork and the decoder neural network together to optimize the auxiliary loss for measuring the error of the noise-reduced accelerometer signal.

28. Pre-training the encoder subnetwork of the tremor classification neural network in order to perform the aforementioned auxiliary task is (i) to obtain the accelerometer signal, and (ii) to obtain the target label of the accelerometer signal. The encoder subnetwork is used to process the accelerometer signal and generate an embedding of the accelerometer signal. Using a predictive neural network to process the embedding of the accelerometer signal to generate a predictive label for the accelerometer signal, and The method according to any one of claims 21 to 27, comprising training the encoder subnetwork and the prediction neural network together to optimize an auxiliary loss for measuring the error between (i) the target label and (ii) the prediction label.

29. The method according to claim 28, wherein the target label of the accelerometer signal determines the number of steps taken by the target during the duration covered by the accelerometer signal.

30. The method according to claim 28, wherein the target label of the accelerometer signal defines an action performed by the object during the duration covered by the accelerometer signal.

31. Training the tremor classification neural network to perform the aforementioned tremor classification task is (i) to obtain the training accelerometer signal, and (ii) to obtain the target tremor classification of the training accelerometer signal. Processing the training accelerometer signal using the embedded subnetwork to generate an embedding of the training accelerometer signal, Using the tremor subnetwork, process the embedding of the training accelerometer signals to generate a score distribution across the set of tremor classes, and The method according to any one of claims 1 to 30, comprising training the tremor subnetwork of the tremor classification neural network to optimize a tremor objective function that measures the score distribution across a set of tremor classes and the error between the target tremor classifications.

32. The method according to claim 31, wherein the parameter values ​​of the encoder subnetwork are frozen during the training of the tremor classification neural network to perform the tremor classification task.

33. It is a system, One or more computers, A system comprising: one or more storage devices communicably coupled to one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operation of each of the methods described in any one of claims 1 to 32.

34. One or more non-temporary computer storage media, which, when executed by one or more computers, store instructions causing one or more computers to perform the operation of each of the methods described in any one of claims 1 to 32.

Images