Systems and methods for detecting cognitive decline using speech analysis

Abstract

The present invention discloses a system and method for detecting cognitive decline of a subject using a classification system, wherein the classification system is used to detect cognitive decline of a subject based on a speech sample. The classification system is trained using speech data corresponding to speech audio recordings of normal patients and patients with cognitive decline to generate an integrated classifier including an integration module and multiple component classifiers. Each component classifier in the multiple component classifiers is a machine learning classifier configured to generate a component output that identifies sample data as corresponding to a normal patient or a patient with cognitive decline. The machine learning classifier is generated based on an available feature subset. The integration module receives component outputs from all component classifiers, and generates an integrated output that identifies sample data as corresponding to a normal patient or a patient with cognitive decline based on the component outputs.

Description

[0001] Priority declaration

[0002] This application claims priority to U.S. Provisional Application Serial No. 62 / 834,170, filed on April 15, 2019, entitled “System and Method for Predicting Cognitive Decline,” the entire contents of which are hereby incorporated by reference herein. Background Art

[0003] Mild cognitive impairment (MCI) causes a slight but noticeable and measurable decline in cognitive abilities, including memory and thinking skills. The changes caused by MCI are not yet severe enough to affect daily life, and people with MCI may not meet the diagnostic guidelines for dementia. However, those with MCI are at an increased risk of eventually developing Alzheimer's disease (AD) or another type of dementia. Early therapeutic intervention may provide a better prospect for success.

[0004] Episodic memory is the memory of events or "episodes". It includes an anterograde (newly encountered information) component or a retrograde (past events) component. Verbal episodic memory decline first appears in patients with preclinical / prodromal AD and predicts disease progression. Assessment of verbal episodic memory decline in MCI represents early cognitive changes and can be used as a screening tool for timely detection and initiation of treatment for early / preclinical AD. Summary of the invention

[0005] An exemplary embodiment of the present invention relates to a method for detecting cognitive decline in a subject. The method includes: obtaining subject baseline voice data, the subject baseline voice data corresponding to a plurality of voice audio recordings of the subject in response to a first instruction set provided to the subject; and obtaining subject test voice data, the subject test voice data corresponding to another voice audio recording of the subject in response to a second instruction set provided to the subject. The method also includes the following steps: extracting a plurality of features from the subject baseline voice data and the subject test voice data; and generating subject test data by normalizing the subject test voice data using the subject baseline voice data. The method further includes analyzing the subject test data using a trained ensemble classifier. The trained ensemble classifier includes an ensemble module and a plurality of component classifiers. Each of the plurality of component classifiers is configured to generate a component output that identifies the subject test data as corresponding to a normal patient or a patient with cognitive decline. Each component classifier is configured to analyze a subset of features selected from a plurality of features. The integration module is configured to receive the component outputs from the component classifiers and generate an integrated output that identifies the subject test data as corresponding to a normal patient or a cognitive decline patient based on the component outputs. The plurality of component classifiers are trained using training baseline speech data corresponding to previous speech audio recordings of the normal patient and cognitive decline patient groups in response to a first set of instructions, and the training test speech data corresponds to previous speech audio recordings of the normal patient and cognitive decline patient groups in response to a second set of instructions.

[0006] An apparatus for detecting cognitive decline in a subject is provided. The apparatus includes: an audio output arrangement configured to generate an audio output; an audio input arrangement configured to receive an audio signal and generate data corresponding to a recording of the audio signal; and a display. The apparatus also includes a processor and a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including an instruction set executable by the processor. The instruction set is operable to: instruct the audio output arrangement structure to audibly provide a first instruction set to the subject multiple times; receive a subject baseline voice data corresponding to a plurality of voice audio recordings of the subject in response to the first instruction set from the audio input arrangement structure; instruct the audio output arrangement structure to audibly provide a second instruction set to the subject; receive a subject test voice data corresponding to another voice audio recording of the subject in response to the second instruction set from the audio input arrangement structure; extract a plurality of features from the subject baseline voice data and the subject test voice data; generate subject test data by normalizing the subject test voice data using the subject baseline voice data; analyze the subject test data using the trained ensemble classifier to generate an output indicating whether the subject may suffer from cognitive decline; and instruct the display to provide a visual representation of the output to the user. The device also includes a memory configured to store the trained ensemble classifier. The trained ensemble classifier includes an ensemble module and a plurality of component classifiers. Each of the plurality of component classifiers is configured to generate a component output that identifies the subject test data as corresponding to a normal patient or a patient with cognitive decline. Each component classifier is configured to analyze a feature subset selected from a plurality of features. The integration module is configured to receive component outputs from the component classifiers and generate an integration output that identifies the subject test data as corresponding to a normal patient or a cognitive decline patient based on the component outputs. The plurality of component classifiers are trained using training baseline speech data corresponding to previous speech audio recordings of the normal patient and cognitive decline patient groups in response to a first instruction set, and the training test speech data corresponds to previous speech audio recordings of the normal patient and cognitive decline patient groups in response to a second instruction set.

[0007] In another exemplary embodiment, a computer-implemented method for training a classification system is provided. The classification system is configured to detect cognitive decline of a subject based on a speech sample of the subject. The method includes acquiring training baseline voice data and training test voice data from a normal patient and a cognitive decline patient group. The training baseline voice data corresponds to a voice audio recording of a normal patient and a cognitive decline patient group in response to a first instruction set, and the training test voice data corresponds to a voice audio recording of a normal patient and a cognitive decline patient group in response to a second instruction set. The method also includes extracting multiple features from (i) the training baseline voice data and (ii) the training test voice data. The method also includes generating an integrated classifier including an integration module and multiple component classifiers. Each component classifier in the multiple component classifiers is configured to generate a component output that identifies the sample data as corresponding to a normal patient or a cognitive decline patient. Each component classifier is configured to analyze a feature subset selected from a plurality of features. The integration module is configured to receive a component output from a component classifier, and based on the component output, generates an integrated output that identifies the sample data as corresponding to a normal patient or a cognitive decline patient. The method also includes: generating a training data set by normalizing the training trial voice data using the training baseline voice data; and training the ensemble classifier using the training data set.

[0008] A system for training a classification system is also provided. The classification system is configured to detect cognitive decline of a subject based on a speech sample of the subject. The system includes a database configured to store training baseline speech data and training test speech data from a normal patient and a cognitive decline patient group. The training baseline speech data corresponds to speech audio recordings of the normal patient and the cognitive decline patient group in response to a first instruction set, and the training test speech data corresponds to speech audio recordings of the normal patient and the cognitive decline patient group in response to a second instruction set. The system also includes a computing device that can be operatively connected to communicate with the database. The computing device includes a processor and a non-transitory computer-readable storage medium, which includes an instruction set that can be executed by the processor. The instruction set is operable to: retrieve training baseline speech data and training test speech data from a database; extract multiple features from (i) training baseline speech data and (ii) training test speech data; generate an integrated classifier including an integration module and multiple component classifiers. Each of the multiple component classifiers is configured to generate a component output that identifies the sample data as corresponding to a normal patient or a cognitive decline patient. Each component classifier is configured to analyze a feature subset selected from a plurality of features. The integration module is configured to receive the component outputs from the component classifiers and generate an integration output that identifies the sample data as corresponding to a normal patient or a patient with cognitive decline based on the component outputs. The instruction set is further operable to: generate a training data set by normalizing the training trial speech data using the training baseline speech data; and train the integration classifier using the training data set. The system also includes a memory configured to store the trained integration classifier.

[0009] These and other aspects of the present invention will become apparent to those skilled in the art after reading the following detailed description of the invention, including the accompanying drawings and appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A system for training a classification system for detecting cognitive decline based on a speech sample of a subject according to an exemplary embodiment of the present application is shown.

[0011] Figure 2 A method for training a classification system for detecting cognitive decline based on a speech sample of a subject according to an exemplary embodiment of the present application is shown.

[0012] Figure 3 An integrated classifier including an integration module and a plurality of component classifiers according to an exemplary embodiment of the present application is shown.

[0013] Figure 4 Shown is the Figure 3 An exemplary method of an integrated classifier in which each component classifier independently selects a subset of speech features.

[0014] Figure 5 An apparatus for detecting cognitive decline based on a speech sample of a subject according to an exemplary embodiment of the present application is shown.

[0015] Figure 6 A method for detecting cognitive decline based on a speech sample of a subject according to an exemplary embodiment of the present application is shown.

[0016] Figure 7 An exemplary system for detecting cognitive decline based on a subject's speech sample according to Embodiment 1 of the present application is shown, the exemplary system including an integrated classifier.

[0017] Figure 8 Data corresponding to the average number of words recalled at different steps in an exemplary embodiment of the Rey Auditory Verbal Learning Test (RAVLT) according to Example 1 of the present application are shown. DETAILED DESCRIPTION

[0018] The present application relates to an apparatus and method for detecting and / or predicting cognitive decline (particularly MCI) by analyzing data corresponding to speech samples of a subject or patient obtained from a neuropsychological test such as a word list recall (WLR) test using a computer-implemented method. The neuropsychological test is capable of screening the cognitive abilities of the subject, including, for example, memory. In addition, the present application includes systems and methods for training a classification system configured to detect cognitive decline (e.g., MCI) and / or predict the onset of cognitive decline or dementia (e.g., AD) based on the subject's WLR speech samples.

[0019] Figure 1 An exemplary embodiment of a system 100 for training a classification system for detecting and / or predicting cognitive decline, particularly MCI, based on speech samples of a subject is shown. The system 100 includes a database 110 for storing various types of data, including data corresponding to audio recordings of previously administered WLR tests. Specifically, the database 110 includes training baseline speech data 112, and one or more sets of training trial speech data 114, each acquired under different experimental conditions, as will be further described below. The database 110 may be stored on one or more non-transitory computer-readable storage media.

[0020] The database 110 may be capable of being operatively connected to the computing device 120 for providing a portion or all of the data stored in the database 110 to the computing device 120, or for allowing the computing device 120 to retrieve a portion or all of the data stored in the database 110. Figure 1 As shown, the database 110 is connected to the computing device 120 via a communication network 140 (e.g., the Internet, a wide area network, a local area network, a cellular network, etc.). However, it is also envisioned that the database 110 can be directly connected to the computing device 120 via a wired connection. The computing device 120 in this embodiment includes a processor 122, a computer-accessible medium 124, and an input / output device 126, which is used to receive and / or transmit data and / or instructions to and / or from the computing device 120. The processor 122 may include, for example, one or more microprocessors, and uses instructions stored on a computer-accessible medium 124 (e.g., a memory storage device). The computer-accessible medium 124 may, for example, be a non-transitory computer-accessible medium containing executable instructions. The system 100 may further include a memory storage device 130, which is provided separately from the computer-accessible medium 124 for storing the integrated classifier 300 generated and trained by the system 100. The memory storage device 130 may be part of the computing device 120, or may be external to the computing device 120 and capable of being operatively connected to the computing device. The memory storage device 130 may also be connected to a separate computing device (not shown) for detecting and / or predicting cognitive decline in a subject. In another embodiment, the integrated classifier 300 may be stored on another memory storage device (not shown) that is connected to a separate computing device (not shown) for detecting and / or predicting cognitive decline in a subject.

[0021] Figure 2 An exemplary embodiment of a method 200 for training a classification system is shown, and the classification system is used to detect and / or predict cognitive decline based on a speech sample of a subject. Specifically, method 200 generates and trains an integrated classifier 300, which is used to analyze a speech sample (e.g., a WLR speech sample) to determine whether the sample is more relevant to a sample of a normal patient or a sample of a patient with cognitive decline (e.g., an MCI patient). The WLR speech sample can be collected from a database 110, which can store data corresponding to previously recorded audio files from various WLR tests, such as experiments using RAVLT to assess language episodic memory. RAVLT is a tool used by examiners based on a vocabulary list, which can be used to measure language episodic memory. It can be used to detect and / or generate scores related to language memory, including learning rates, short-term and delayed language memory, recall performance after interference stimulation, cognitive memory, and learning patterns (serial position effect) related to cognitive abilities.

[0022] In step 202, the computing device 110 receives multiple sets of training baseline speech data 112 from the database 110, the multiple sets corresponding to multiple sets of previous speech audio recordings of normal patients and cognitive decline patient groups. Each set of training baseline speech data 112 corresponds to speech of normal patients and cognitive decline patient groups in response to the same set of instructions for listening to a vocabulary list and recalling and speaking the same vocabulary list. For example, the training baseline speech data 112 corresponds to speech of normal patients and cognitive decline patient groups in response to a first set of instructions for listening to a first vocabulary list and immediately recalling and speaking the first vocabulary list. In a specific embodiment, the training baseline speech data 112 may include data corresponding to speech audio recordings of normal patients and cognitive decline patient groups from a learning trial of a WLR test such as, for example, a RAVLT test. Step 202 may utilize any suitable number of sets of training baseline speech data 112 to establish an average baseline characteristic, and the training trial speech data 114 may be compared with the average baseline characteristic to identify and / or enhance the feature discriminant signal contained in the training trial speech data 114. Specifically, the training baseline speech data 112 may include learning phase data from any suitable WLR test. The training baseline speech data 112 (specifically, the learning phase data) is used as a baseline feature, and the training trial speech data 114 can be compared with the baseline feature to generate a quantitative representation of cognitive load for normal patients and cognitive decline patient groups. In some embodiments, at least 3 sets, at least 5 sets, or at least 10 sets of the training baseline speech data 112 can be used. In an exemplary embodiment, 5 sets of the training baseline speech data 112 are used.

[0023] The computing device 110 also receives a set of training trial speech data 114 from the database 110. The method 200 uses the set of training trial speech data 114 to generate a training data set 330 for training the integrated classifier 300 and / or to generate the integrated classifier 300, as will be further described below. Compared to the training baseline speech data 112, the training trial speech data 114 corresponds to a previous speech audio recording of a normal patient and a cognitive decline patient group in response to at least one different instruction set, and the at least one different instruction set is used to recall and speak. For example, the different instruction sets may instruct the patient to listen to a different vocabulary list and immediately recall and speak the different vocabulary list. For another example, the different instruction sets may instruct the patient to listen to the same vocabulary list as the vocabulary list used for training the baseline speech data 112, but recall and speak the vocabulary list after a distraction task and / or after a delay period of time (e.g., at least or about 10 minutes, at least or about 20 minutes, or at least or about 30 minutes). In one specific embodiment, the training trial speech data 112 may include data corresponding to speech audio recordings from groups of normal patients and cognitive decline patients from WLR tests such as, for example, the distraction test, the post-distraction test, and / or the delayed test of the RAVLT test.

[0024] In one specific embodiment, the method 200 for training a classification system for detecting and / or predicting cognitive decline in a subject based on speech samples of the subject may utilize clinical trial data obtained from normal patients and MCI patients as part of the RAVLT test. Figure 8 Data corresponding to the average number of words recalled at different steps of an exemplary embodiment of the RAVLT test are provided in FIG. Figure 8 As shown, the RAVLT test may include a plurality of different trials (eg, trials IV 811 -815 ) that are part of a learning phase 802 of the RAVLT test. In step 202 , the computing device 110 may retrieve speech data that is part of the learning phase 802 of the RAVLT test as training baseline speech data 112 . Figure 8 The RAVLT test is also shown to include different types of recall trials: distraction trial B 822, recall after distraction 824, and 20-minute delayed recall 826. Computing device 110 may retrieve speech data collected during any of these recall trials as training trial speech data 114 of step 202.

[0025] In step 204, the computing device 110 analyzes and extracts a plurality of speech features from each of the speech data sets ((i) multiple sets of training baseline speech data 112, and (ii) sets of training test speech data 114) received from the database 110. Specifically, the computing device 110 extracts the same type of speech feature set from each of the above-mentioned data sets. The computing device 110 may extract any suitable type of acoustic attribute as a speech feature from the speech data set for analyzing an audio recording of spoken speech, including average data values ​​and standard deviation data values ​​corresponding to such acoustic attributes. For example, the speech feature may include one or more of the exemplary acoustic attributes of the speech audio recording listed and defined in Table 1 below. In some embodiments, the speech feature may include all or a subset of the average data values ​​and / or standard deviation data values ​​corresponding to these exemplary acoustic attributes on frames of the speech audio recording.

[0026] Table 1.

[0027]

[0028] The computing device 120 can extract any suitable number of speech features from each speech data set. The speech features of increasing the number can improve the prediction performance and / or analysis performance of the system and method of the present application, but may become cumbersome in calculation. Therefore, an appropriate number of speech features can be selected to achieve a balance between prediction performance and / or analysis performance and computational efficiency. In some embodiments, the computing device 120 can extract at least 24, at least 30, at least 50 or at least 100 different speech features from each speech data set. In one embodiment, the computing device 120 can extract 5 to 150 speech features, 10 to 100 speech features, 12 to 50 speech features from each speech data set.

[0029] In step 206, the computing device 120 generates an integrated classifier 300 based on the set of training test speech data 114 received from the database 110 and the relevant speech features of the training test speech data 114 acquired in the previous step (step 204). Figure 3As shown, the integrated classifier 300 includes multiple component classifiers 310 and an integration module 320. Each component classifier 310 is a machine learning classifier configured to generate a component output that identifies sample data as sample data corresponding to a normal patient or a patient with cognitive decline. When the component classifier 310 is trained using data acquired from a normal patient and an MCI patient, the component classifiers 310 are each a machine learning classifier configured to generate a component output that identifies sample data as sample data corresponding to a normal patient or an MCI patient. More specifically, each component classifier in the component classifier 310 is a support vector machine (SVM). It is also contemplated that each component classifier in the component classifier 310 may alternatively utilize other suitable supervised learning modules, such as, for example, a logistic regression module, an extreme gradient boosting module, a random forest module, or a naive Bayes module. Each component classifier in the component classifier 310 is generated by the computing device 120 to analyze a downsampled subset of the speech features of the training test speech data 114. For each component classifier 310, the SVM is used Figure 4 The method 400 shown and described further below independently selects a subset of speech features. The integrated classifier 300 may include any suitable number (N) of component classifiers 310. For example, the integrated classifier 300 may include at least 10, at least 20, at least 30, or at least 50 component classifiers 310. In a specific embodiment, the integrated classifier 300 includes 30 component classifiers 310.

[0030] The integration module 320 is configured to receive the component outputs from all the component classifiers 310 and generate an integration output 340 that identifies the sample data as corresponding to a normal patient or an MCI patient based on the component outputs. The integration module 320 may utilize any suitable method to determine the integration output 340 based on the component output provided by each of the component classifiers 310. For example, the integration module 320 may utilize a bagging method or an aggregation method that receives and considers the component output from each of the component classifiers 310 with equal weight. Alternatively, the integration module 320 may utilize other methods in which the component outputs are weighted differently, such as an adaptive boosting method or a gradient boosting method.

[0031] Figure 4An exemplary embodiment of a method 400 for independently selecting a subset of speech features for each component classifier 310 is shown. Each time step 206 selects a desired downsampled subset of speech features for use in generating each component classifier 310, the computing device 120 repeats the method 400. In other words, the computing device 120 uses the method 400 to analyze the training trial speech data 114 to select a desired downsampled subset of speech features for generating a first component classifier 311, repeats the analysis to select a different downsampled subset of speech features to generate the next component classifier 312, and further repeats the analysis to generate subsequent component classifiers 313 to 317 until the computing device 120 has selected a downsampled subset of speech features and generated the Nth component classifier 317. The method 400 selects a different downsampled subset of speech features for each of the component classifiers 310, which allows the computing device 120 to generate an ensemble classifier 300 having component classifiers 310, each of which models only a subset of all available speech features, but samples together on a greater number of speech features. This structure of the ensemble classifier 300 provides a module with improved predictive performance and / or analysis performance that combines a greater number of speech features while limiting each of the component classifiers 310 to analyze only a selected subset of the speech features of each component classifier to provide improved computational efficiency.

[0032] In step 402, the computing device 120 analyzes the training test speech data 114 to obtain a subsampled sample of the training test speech data 114. As described below, the computing device 120 uses the subsampled sample to identify the desired parameters and features of the component classifier 310 generated for the reduced number of speech features. The subsampled sample includes a first number of samples from normal patients of the training test speech data 114 and a second number of samples from cognitive decline patients of the training test speech data 114. The first number of samples can be randomly selected from entries from normal patients in the training test speech data 114. Similarly, the second number of samples can be randomly selected from entries from cognitive decline patients in the training test speech data 114. In order to provide a basic balance between two different patient categories (i.e., normal patients and cognitive decline patients) in step 402, the second number of samples is at least 80% of the first number of samples, at least 90% of the first number of samples, or at least 95% of the first number of samples. Preferably, the subsampled samples are balanced so that the ratio of the first number of samples to the second number of samples is 1:1. This preferred subsampling allows the remainder of the method 400 to continue processing data that is equally balanced between the two categories of normal patients and patients with cognitive decline. Typically, the training trial speech data 114 may include more normal patients than patients with cognitive decline, and therefore provide data that is unbalanced between the two categories. Balanced subsampling helps to resolve any class imbalance that may occur in the training trial speech data 114, which would otherwise result in a classifier that is biased towards the larger category (normal patients), which may result in false negatives in the classifier and omission of patients that should have been identified as corresponding to patients with cognitive decline. Therefore, the integrated classifier 300 combines several individual component classifiers 310 for subsets of speech features selected using substantially equal or equal proportions of normal patients and patients with cognitive decline, and thus provides a classifier that is balanced in samples of different speech features and is therefore capable of learning from the entire training data set.

[0033] In step 404, the computing device 120 analyzes all speech features of the subsampled samples of the training test speech data 114 and sorts the speech features based on a predetermined criterion. The speech features may be sorted based on any suitable statistical criterion for identifying those features that contribute most significantly to the feature discriminant signal, as observed in the subsampled samples of the training test speech data 114. Specifically, the speech features may be sorted based on the feature importance of each of the speech features to the subsampled samples of the training test speech data 114. Specifically, when the component classifier 310 is an SVM, the computing device 120 may sort the speech features based on the importance of the speech features in specifying the difference (e.g., decision boundary) between normal patients and cognitive decline patients, as observed in the subsampled samples obtained from the previous step (step 402). More specifically, each speech feature may be sorted according to its corresponding coefficient to the SVM hyperplane (e.g., decision boundary) based on the subsampled samples of the training test speech data 114. Speech features with lower coefficients for the SVM hyperplane are considered to be relatively less important for specifying the decision boundary of the SVM, and therefore have lower feature importance and lower ranks.

[0034] In step 406, the computing device 120 selects a subset of speech features from the ranking of speech features generated by the previous step (step 404) based on a predetermined ranking threshold (e.g., only selecting for the first x speech features). The computing device 120 uses the selected subset of x speech features to generate a component classifier 310 for analyzing the downsampled x features. By selecting the first x ranked speech features, the computing device 120 generates a component classifier 310 that models those features that contribute most significantly to the feature discriminant signal contained in the training test speech data 114, while limiting the computational cost of the component classifier 310 to the number of x speech features of the downsampled number. It is contemplated that any suitable number (x) of highly ranked features may be selected. For example, step 406 may select at least 10, at least 20, at least 30 of the highly ranked features. In a specific embodiment, for each component classifier 310, the top 20 ranked features are selected.

[0035] Returning to method 200, in step 208, the computing device 120 generates a training data set 330 for training the integrated classifier 300 generated from the previous step (step 206). Specifically, the computing device 120 generates a training data set 330, which includes training test speech data 114 normalized with training baseline speech data 112 (specifically, multiple sets of training baseline speech data 112). In one embodiment, the training baseline data 112 includes learning phase data from a WLR test (e.g., a RAVLT test), and the training data set 330 is generated by normalizing the training test speech data 114 with the learning phase data from the WLR test. The training data set obtained from this embodiment provides a quantitative value of cognitive load corresponding to a normal patient and a cognitive decline patient group. Specifically, the computing device 120 generates a training data set 330, which includes each feature of the training test speech data 114 normalized with the average value of the corresponding feature on multiple sets of training baseline speech data 112. More specifically, each feature of the training data set 330 can be obtained by subtracting the average value of the features on multiple sets of the training baseline speech data 112 from the features of the training test speech data 114. Normalizing the speech features of the training test speech data 114 using multiple sets of the training baseline speech data 112 can enhance the feature discrimination signal contained in the training test speech data 114 and improve the prediction performance and / or analysis performance of the ensemble classifier 300 trained using such normalized data.

[0036] In step 210, the computing device 120 uses the training data set 330 generated by step 208 to train the integrated classifier 300 to generate a trained integrated classifier 516. Each component classifier in the component classifier 310 is trained with a downsampled portion of the training data set 330, the downsampled portion corresponding to the subset of features selected by the component classifier for modeling. Therefore, a unique downsampled portion of the training data set 330 is provided to train each component classifier in the component classifier 310. In other words, the downsampled portion of the training data set 330 used to train the component classifier 311 is different from the downsampled portion of the training data set 330 used to train other component classifiers 312 to 317. The trained integrated classifier 516 includes the component classifiers 310, each of which has selected a subset of speech features and a corresponding weighting coefficient value for each feature generated by training the component classifier 310 with the training data set 330. The trained ensemble classifier 516 may be stored in any suitable memory and may be loaded into a user device for analyzing new patient speech data to detect and / or predict cognitive decline, particularly MCI, in the subject.

[0037] Figure 5An exemplary device 500 for detecting and / or predicting cognitive decline of a subject based on a speech sample of the subject is shown. The device 500 utilizes a trained ensemble classifier 516, such as generated and trained by the above-described method 200. The device 500 includes an audio input arrangement structure 506 for receiving an audio signal and generating data corresponding to a recording of the audio signal. For example, the audio input arrangement structure 506 may include a microphone for capturing the vocalized speech of the subject. The device 500 may also include an audio output arrangement structure 508 for generating an audio output. For example, the audio output arrangement structure 508 may include a speaker for audibly providing instructions for WLR testing to the subject. The device 500 also includes a display 512 for generating visual output to a user. The device 500 may also include an input / output device 510 for receiving and / or transmitting data or instructions to or from the device 500. The device 500 also includes a processor 502 and a computer-accessible medium 504. The processor 502 can be operatively connected to the audio output arrangement 508 and the display 512 to control the audio output and visual output of the device 500. The processor 502 can also be operatively connected to the audio input arrangement 506 to receive and analyze data corresponding to the audio recording captured by the audio input arrangement 506. The processor 502 can include, for example, one or more microprocessors and use instructions stored on a computer-accessible medium 504 (e.g., a memory storage device). The computer-accessible medium 504 can be, for example, a non-transitory computer-accessible medium containing executable instructions. The device 500 can further include a memory storage device 514, which is provided separately from the computer-accessible medium 504, for storing the trained ensemble classifier 516. The memory storage device 514 can be part of the device 500, or can be external to the device 500 and can be operatively connected to the device.

[0038] Figure 6An exemplary method 600 for detecting and / or predicting cognitive decline of a subject based on a speech sample of the subject according to an exemplary embodiment of the present application is shown. Specifically, the method 600 obtains a WLR speech sample from the subject to determine whether the sample is more relevant to a sample of a normal patient or a sample of a patient with cognitive decline. In step 602, a first instruction set is provided to the subject multiple times by a clinician operating the device 500 or by a processor 502, the processor instructs the audio output arrangement structure 506 to audibly provide the first instruction set to the patient, and the processor 502 receives the subject's baseline speech data corresponding to a plurality of speech audio recordings of the subject in response to the first instruction set from the audio input arrangement structure 506. A second instruction set is provided to the subject by a clinician or by the processor 502, the processor instructs the audio output arrangement structure 506 to audibly provide the second instruction set, and the processor 502 receives the subject's test speech data corresponding to the subject's speech audio recording in response to the second instruction set from the audio input arrangement structure 506. The first set of instructions may correspond to those instructions used to generate the training baseline speech data 112 in step 202 described above, and the second set of instructions may correspond to those instructions used to generate the training trial speech data 114 in step 202 .

[0039] In step 604, processor 502 analyzes and extracts multiple features from subject baseline voice data and subject trial voice data in a similar manner as described above with respect to step 204. In step 606, processor 502 generates subject test data in a similar manner as described above with respect to the training data set in step 208, and the subject test data includes each feature of the subject trial voice data normalized with the corresponding feature of the subject baseline voice data. Similar to step 208, in one embodiment, the subject baseline voice data includes learning phase data from a WLR test (e.g., RAVLT test) applied to the subject, and generates subject test data by normalizing the subject trial voice data with the learning phase data from the WLR test. The resulting subject test data obtained from this embodiment provides a quantitative value corresponding to the subject cognitive load. In step 608, processor 502 uses trained integrated classifier 516 to analyze the subject test data. Each component classifier in the multiple component classifiers 310 of the trained integrated classifier 516 analyzes the subject test data to generate a component output that identifies the subject test data as corresponding to a normal patient or a patient with cognitive decline. The integrated module 320 of the trained integrated classifier 516 receives the component output from the component classifier 310, and analyzes the component output to generate an integrated output that identifies the subject test data as corresponding to a normal patient or a patient with cognitive decline. The processor 502 can also generate an output based on the integrated output generated by the trained integrated classifier 516 by analyzing the subject test data, indicating whether the subject has an increased risk of neurodegeneration and / or may suffer from cognitive decline. When the integrated output identifies the subject test data as corresponding to a patient with cognitive decline, the output can indicate that the subject has an increased risk of neurodegeneration and / or may suffer from cognitive decline. In contrast, when the integrated output identifies the subject test data as corresponding to a normal patient, the output can indicate that the subject does not have an increased risk of neurodegeneration and / or is unlikely to suffer from cognitive decline. In step 610 , processor 502 instructs display 512 to provide a visual display of the display output.

[0040] The integrated output generated by the trained integrated classifier 516 by analyzing the subject test data and / or the output generated by the device 500 can be provided to the clinician to screen and identify those patients with increased risk of neurodegeneration and / or cognitive decline, including but not limited to verbal episodic memory decline, reduced learning rate, short-term verbal memory decline, delayed verbal memory decline, recall performance decline after interference stimulus, recognition memory decline, etc. The subject whose subject test data is identified by the integrated output as corresponding to a patient with cognitive decline may have an increased risk of suffering from a neurodegenerative disease (such as AD or another type of dementia). Therefore, the integrated output generated by analyzing the subject test data and / or the output generated by the device 500 can help clinicians guide patients at risk to further cognitive testing in order to confirm the integrated output and / or the output from the method 600. Therefore, the device and method of the present application can identify those patients at risk of neurodegeneration earlier than conventional diagnosis of neurodegenerative diseases, dementia or AD, and provide better care for these patients at risk, and / or start treatment at an early stage of cognitive decline. In addition, when the integrated output indicates that the subject test data corresponds to a patient with cognitive decline and / or the output generated by the device 500 indicates that the subject has an increased risk of neurodegeneration and / or may suffer from cognitive decline, the processor 502 generates instructions for administering the treatment to the subject. In some embodiments, the instructions may instruct to automatically administer the treatment without any intervening user intervention.

[0041] Suitable treatment may include treatment for improving cognitive ability and / or treatment for delaying cognitive ability degradation. In one example, treatment may include non-pharmacological treatment, such as, for example, digital therapy (e.g., brain training module), to improve the cognitive ability of the subject. In some embodiments, when the integrated output indicates that the subject's test data corresponds to a patient with cognitive decline and / or the output generated by the device 500 indicates that the subject has an increased risk of neurodegeneration and / or may suffer from cognitive decline, digital therapy may be automatically initiated. The brain training module includes an instruction set that can be executed by a processor for applying brain training exercises to improve the cognitive ability of the subject. The brain training module may be connected to a user interface to display brain training exercise instructions to the subject and receive input from the user in response to such instructions.

[0042] In another example, treatment may include administering one or more pharmaceutically active agents for preventing and / or delaying the progression of neurodegeneration. Specifically, treatment may include administering one or more pharmaceutically active agents for preventing and / or delaying the progression of dementia (or more specifically, for preventing and / or delaying the progression of AD). Suitable pharmaceutically active agents may include one or more pharmaceutically active agents for preventing or reducing the aggregation of beta-amyloid or tau proteins in the brain, improving synaptic or cellular elasticity of the brain, regulating the expression of ApoE4 genes, regulating neuroinflammation-related pathways, etc. In one example, pharmaceutically active agents may include tranquilizers, acetylcholinesterase inhibitors, etc.

[0043] It should be understood by those skilled in the art that the exemplary embodiments described herein can be implemented in any number of ways, including in the form of separate software modules, in the form of a combination of hardware and software, etc. For example, the exemplary method can be an implementation in one or more programs stored in a non-transitory storage medium and containing lines of code, which can be executed by one or more processor cores or separate processors when compiled. A system according to one embodiment includes multiple processor cores and a set of instructions executed on multiple processor cores to perform the above-mentioned exemplary methods. The processor core or separate processor can be incorporated into any suitable electronic device or can communicate with any suitable electronic device (for example, on a board processing arrangement structure within the device or a processing arrangement structure outside the device), such as a mobile computing device, a smart phone, a computing tablet, a computing device, etc., which can communicate with at least a portion of the device.

[0044] Example

[0045] Embodiment 1

[0046] In Example 1, an exemplary word list recall test is manually administered by an examiner (similar to the examiner of the RAVLT test) and is also administered via a computer device to a total of 106 patients, 84 of whom are normal patients and 22 patients suffer from MCI. Each of the patients is provided with a first instruction set for listening to a first word list and immediately recalling and speaking the first word list. The first word list includes the following words: drum, helmet, music, coffee, school, parents, machine, garden, radio, farmer, nose, sailor, color, house, river. The speech of each patient is recorded to generate a baseline speech data set corresponding to the speech recording. For the entire patient group, the presentation and immediate recall of the first word list are repeated a total of 5 times to generate 5 different baseline speech data sets.

[0047] The patients are then subjected to a distraction task. Specifically, each patient is then provided with a second set of instructions for listening to a second vocabulary list and recalling and speaking the second vocabulary list. The second vocabulary list includes the following words: table, ranger, bird, shoe, stove, mountain, glasses, towel, cloud, boat, lamb, clock, pencil, church, fish. The voice of each patient is recorded to generate a test voice data set corresponding to the voice recording of the distraction test. After the distraction task, the patient is then asked to recall and speak the first vocabulary list. The voice of each patient is recorded to generate a test voice data set corresponding to the voice recording of the post-distraction test. After a 20-minute delay, the patient is then asked to recall and speak the first vocabulary list again. The voice of each patient is recorded to generate a test voice data set corresponding to the voice recording of the delayed recall test.

[0048] An exemplary speech feature set (e.g., mean and standard deviation values ​​of exemplary acoustic properties (as listed in Table 1 above) over speech frames of speech audio recordings) is extracted from each of five different baseline speech datasets, experimental speech datasets for distraction trials, experimental speech datasets for post-distraction trials, and experimental speech datasets for delayed recall trials. Figure 7 An exemplary integrated classifier 700 of Example 1 is shown in FIG. Figure 7 As shown, the ensemble classifier 700 of Example 1 includes 30 component classifiers 701 to 730, each of which is a support vector machine (SVM) generated based on a downsampled subset of the first 20 features, which are determined based on a subsampled sample of training data balanced between training data related to control patients and training data related to MCI patients. Specifically, the subsampled sample includes 20 entries corresponding to control patients and 20 entries corresponding to MCI patients.

[0049] like Figure 7 As shown, the ensemble classifier 700 is trained and validated by 10-fold cross validation using a dataset 760, which includes each feature of the test speech data normalized by the average value of the corresponding feature on the five baseline speech datasets. Figure 7 As shown, data 760 is randomly distributed and divided into 10 folds of equal size. In these 10-fold partitions, data from 9 folds are used as training data 770 for the integrated classifier 700, and the remaining folds are used as validation data 780. Training and validation are repeated using folds different from the validation data 780, while the remaining folds are used as training data 770 until each fold has been used as validation data 780 once. Although Example 1 uses a 10-fold cross-validation method, it is contemplated that a k-fold cross-validation method can be used to validate the integrated classifier 700, where k is any suitable positive integer.

[0050] The average values ​​of certain performance measures generated based on the validation data on each of the 10 folds of each of the distraction test, the post-distraction test, and the delayed recall test are provided below in Table 2. As shown in the performance measures of Table 2, the post-distraction test had the highest discrimination signal between MCI patients and normal patients, followed by the distraction test, and then the delayed recall test.

[0051] Table 2.

[0052]

[0053] The scope of the present invention described herein and protected by the claims is not limited by the specific embodiments disclosed herein, because these embodiments are intended to be examples of several aspects of the present invention. Any equivalent embodiments are intended to be within the scope of the present invention. In fact, in addition to those shown and described herein, various modifications of the present invention will become apparent to those skilled in the art based on the foregoing description. Such modifications are also intended to fall within the scope of the appended claims. All publications cited herein are incorporated by reference in their entirety.

Claims

1. A method for detecting cognitive decline in a subject, comprising: acquiring subject baseline speech data corresponding to a plurality of audio recordings of the subject's speech in response to a first set of instructions provided to the subject, wherein the first set of instructions instructs the subject to listen to a first list of words and to immediately recall and speak the first list of words; acquiring subject trial speech data corresponding to an additional audio recording of the subject's speech in response to a second set of instructions provided to the subject, wherein the second set of instructions instructs the subject to: (i) listen to a second list of words different from the first list of words and immediately recall and speak the second list of words, or (ii) listen to the first list of words and recall and speak the first list of words after a distraction task and / or after a delay; extracting a plurality of features from the subject baseline speech data and the subject test speech data; generating subject test data by normalizing the subject test speech data using the subject baseline speech data, wherein normalizing the subject test speech data comprises subtracting an average value of each of the features over the plurality of speech audio recordings forming the subject baseline speech data from a corresponding feature of the subject test speech data; as well as The subject test data is analyzed using a trained ensemble classifier, the trained ensemble classifier comprising: Integration module and multiple component classifiers, where Each of the plurality of component classifiers is configured to generate a component output identifying the subject test data as corresponding to a normal patient or a patient with cognitive decline, each component classifier being configured to analyze a subset of features selected from the plurality of features, and The integration module is configured to receive the component outputs from the component classifiers and generate an integration output based on the component outputs that identifies the subject test data as corresponding to the normal patient or the cognitive decline patient, The multiple component classifiers are trained using training baseline speech data corresponding to previous speech audio recordings of normal patients and cognitive decline patient groups in response to the first instruction set, and the training test speech data corresponds to previous speech audio recordings of the normal patients and cognitive decline patient groups in response to the second instruction set.

2. The method according to claim 1, wherein: The feature subset of the trained ensemble classifier is independently selected during training for each of the plurality of component classifiers by comprising the steps of: sorting the plurality of features of the subsampled samples of the training test speech data based on a predetermined criterion, the subsampled samples consisting of a first number of samples of the training test speech data corresponding to normal patients and a second number of samples of the training test speech data corresponding to patients with cognitive decline, and The subset of features is selected from the plurality of features based on a predetermined ranking threshold.

3. The method according to claim 2, wherein: The second number of samples is at least 80% of the first number of samples.

4. The method according to claim 3, wherein: The ratio of the first number of samples to the second number of samples is 1:

1.

5. The method according to claim 2, wherein: The predetermined criterion is feature importance.

6. The method according to claim 1, wherein: The plurality of component classifiers are trained using a training data set generated by normalizing training trial speech data using training baseline speech data.

7. The method according to claim 1, wherein: The method detects mild cognitive impairment (MCI) in the subject, and the normal patient and cognitive decline patient groups consist of normal patients and MCI patients.

8. The method according to claim 1, wherein: Each of the plurality of component classifiers includes a plurality of weighted feature coefficients trained using the training baseline speech data and the training trial speech data.

9. The method according to claim 1, wherein: Each component classifier of the plurality of component classifiers is a machine learning classifier.

10. The method according to claim 9, wherein: Each component classifier of the plurality of component classifiers is a support vector machine (SVM).

11. The method according to claim 1, further comprising: When the integrated output identifies the subject test data as corresponding to the cognitive decline patient, an output indicating that the subject is likely to suffer from cognitive decline is displayed, and when the integrated output identifies the subject test data as corresponding to the normal patient, an output indicating that the subject is unlikely to suffer from cognitive decline is displayed.

12. The method according to claim 1, further comprising: When the integrated output identifies the subject test data as corresponding to the cognitive decline patient, an output is generated based on the integrated output indicating that the subject has verbal episodic memory decline.

13. The method according to claim 12, wherein: The first set of instructions corresponds to a word list recall test and the second set of instructions corresponds to a delayed word list recall test.

14. The method according to claim 1, further comprising: When the integrated output identifies the subject test data as corresponding to the cognitive decline patient, a treatment for improving cognitive ability or for slowing the deterioration of cognitive ability is administered.

15. The method according to claim 1, further comprising: When the integrated output identifies the subject test data as corresponding to the cognitive decline patient, a pharmaceutically active agent for preventing or delaying progression of dementia is administered.

16. The method according to claim 1, further comprising: When the integrated output identifies the subject test data as corresponding to the cognitive decline patient, a pharmaceutically active agent for preventing or reducing aggregation of beta-amyloid or tau protein in the brain of the subject is administered.

17. A device for detecting cognitive decline in a subject, comprising: an audio output arrangement configured to generate an audio output; an audio input arrangement configured to receive an audio signal and generate data corresponding to a recording of the audio signal; monitor; A processor and a non-transitory computer-readable storage medium comprising a set of instructions executable by the processor, the set of instructions being operable to: instructing the audio output arrangement to audibly provide a first set of instructions to a subject a plurality of times, wherein the first set of instructions instructs the subject to listen to a first list of words and to immediately recall and speak the first list of words, receiving from the audio input arrangement subject baseline speech data corresponding to a plurality of audio recordings of speech of the subject in response to the first set of instructions, instructing the audio output arrangement to audibly provide a second set of instructions to the subject, wherein the second set of instructions instructs the subject to: (i) listen to a second list of words different from the first list of words and immediately recall and speak the second list of words, or (ii) listen to the first list of words and recall and speak the first list of words after a distraction task and / or after a delay, receiving from the audio input arrangement subject trial speech data corresponding to a further speech audio recording of the subject in response to a second set of instructions, extracting a plurality of features from the subject baseline speech data and the subject test speech data, generating subject test data by normalizing the subject test speech data using the subject baseline speech data, wherein normalizing the subject test speech data comprises subtracting an average value of each of the features over the plurality of speech audio recordings forming the subject baseline speech data from a corresponding feature of the subject test speech data, analyzing the subject test data using the trained ensemble classifier to generate an output indicating whether the subject is likely to suffer from cognitive decline, and instructing the display to provide a visual representation of the output to a user; and a memory configured to store the trained ensemble classifier, the trained ensemble classifier comprising: Integration module and multiple component classifiers, where Each of the plurality of component classifiers is configured to generate a component output identifying the subject test data as corresponding to a normal patient or a patient with cognitive decline, each component classifier being configured to analyze a subset of features selected from the plurality of features, and The integration module is configured to receive the component outputs from the component classifiers and generate an integrated output based on the component outputs that identifies the subject test data as corresponding to the normal patient or the cognitive decline patient, wherein the multiple component classifiers are trained using training baseline speech data corresponding to previous speech audio recordings of the normal patient and cognitive decline patient groups in response to the first instruction set, and the training test speech data corresponds to previous speech audio recordings of the normal patient and cognitive decline patient groups in response to the second instruction set.

18. The device according to claim 17, wherein: The feature subset for each of the plurality of component classifiers is independently selected from the plurality of features during training for each of the plurality of component classifiers by comprising the steps of: sorting the plurality of features of the subsampled samples of the training test speech data based on a predetermined criterion, the subsampled samples consisting of a first number of samples of the training test speech data corresponding to normal patients and a second number of samples of the training test speech data corresponding to patients with cognitive decline, and The subset of features is selected from the plurality of features based on a predetermined ranking threshold.

19. A computer-implemented method for training a classification system, the classification system being configured to detect cognitive decline in a subject based on a speech sample of the subject, the method comprising: Training baseline speech data and training test speech data are obtained from a group of normal patients and a group of patients with cognitive decline, wherein the training baseline speech data corresponds to speech audio recordings of the normal patients and the group of patients with cognitive decline in response to a first set of instructions, and the training test speech data corresponds to speech audio recordings of the normal patients and the group of patients with cognitive decline in response to a second set of instructions, wherein The first set of instructions instructs the subject to listen to a first list of words and immediately recall and speak the first list of words; and The second set of instructions instructs the subject to: (i) listen to a second word list different from the first word list and immediately recall and speak the second word list, or (ii) listen to the first word list and recall and speak the first word list after a distraction task and / or after a delay; extracting a plurality of features from (i) the training baseline speech data and (ii) the training test speech data; Generate an integrated classifier including an integrated module and multiple component classifiers, where Each of the plurality of component classifiers is configured to generate a component output identifying the sample data as corresponding to a normal patient or a patient with cognitive decline, each component classifier is configured to analyze a subset of features selected from the plurality of features, and The integration module is configured to receive the component outputs from the component classifiers and generate an integration output that identifies the sample data as corresponding to the normal patient or the cognitive decline patient based on the component outputs; generating a training data set by normalizing the training test speech data using the training baseline speech data, wherein normalizing the subject test speech data comprises subtracting an average value of each of the features over the plurality of speech audio recordings forming the subject baseline speech data from a corresponding feature of the subject test speech data; as well as The ensemble classifier is trained using the training dataset.

20. The computer-implemented method of claim 19, wherein: The feature subset is independently selected from the plurality of features for each of the plurality of component classifiers by the steps comprising: sorting the plurality of features of the subsampled samples of the training test speech data based on a predetermined criterion, the subsampled samples consisting of a first number of samples of the training test speech data corresponding to normal patients and a second number of samples of the training test speech data corresponding to patients with cognitive decline, and The subset of features is selected from the plurality of features based on a predetermined ranking threshold.

21. A system for training a classification system, the classification system being configured to detect cognitive decline in a subject based on a speech sample of the subject, the system comprising: A database configured to store training baseline speech data and training test speech data from a group of normal patients and a group of patients with cognitive decline, the training baseline speech data corresponding to speech audio recordings of the normal patients and the group of patients with cognitive decline in response to a first set of instructions, and the training test speech data corresponding to speech audio recordings of the normal patients and the group of patients with cognitive decline in response to a second set of instructions, wherein The first set of instructions instructs the subject to listen to a first list of words and immediately recall and speak the first list of words; and The second set of instructions instructs the subject to: (i) listen to a second word list different from the first word list and immediately recall and speak the second word list, or (ii) listen to the first word list and recall and speak the first word list after a distraction task and / or after a delay; a computing device operatively connected to communicate with the database, the computing device comprising a processor and a non-transitory computer-readable storage medium comprising a set of instructions executable by the processor, the set of instructions operable to: retrieving the training baseline speech data and the training test speech data from the database, extracting a plurality of features from (i) the training baseline speech data and (ii) the training test speech data, Generate an integrated classifier including an integrated module and multiple component classifiers, where Each of the plurality of component classifiers is configured to generate a component output identifying the sample data as corresponding to a normal patient or a patient with cognitive decline, each component classifier is configured to analyze a subset of features selected from the plurality of features, and The integration module is configured to receive the component outputs from the component classifiers and generate an integration output based on the component outputs that identifies the sample data as corresponding to the normal patient or the cognitive decline patient, generating a training data set by normalizing the training test speech data using the training baseline speech data, wherein normalizing the subject test speech data comprises subtracting an average value of each of the features over the plurality of speech audio recordings forming the subject baseline speech data from a corresponding feature of the subject test speech data, and Using the training data set to train the ensemble classifier; as well as A memory configured to store the trained ensemble classifier.

22. The system for training the classification system according to claim 21, wherein: The set of instructions is further operable to independently select the subset of features for each of the plurality of component classifiers from the plurality of features by the steps comprising: sorting the plurality of features of the subsampled samples of the training test speech data based on a predetermined criterion, the subsampled samples consisting of a first number of samples of the training test speech data corresponding to normal patients and a second number of samples of the training test speech data corresponding to patients with cognitive decline, and The subset of features is selected from the plurality of features based on a predetermined ranking threshold.

Images