Selecting Audio Features to Build a Model to Detect Medical Conditions

Abstract

To provide systems, computer-implemented methods and non-transitory computer-readable media for selecting features of speech to be used to build or train a mathematical model for detecting or diagnosing a medical condition.SOLUTION: A computing device improves the performance of a mathematical model by appropriately selecting features 1021, 1022 to be used with the mathematical model. The device computes a feature selection score 1031 for each acoustic feature 1021 and each language feature 1022, and then uses the scores 1031 to select the features 1021, 1022 with the highest scores 1031. The device then uses the selected features 1021, 1022 to train and deploy a mathematical model. In some implementations, the device may use computed prompt selection scores 1041 to select prompts, and use the deployed mathematical model with the selected prompts.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to the selection of audio features used to build mathematical models for detecting pathologies in order to improve the performance of the models. [Background technology]

[0002] Early diagnosis of a medical condition, such as Alzheimer's disease or concussion, can be undertaken with the aim of improving treatment and improving the quality of life for those who develop the condition. One method that can be used to detect a medical condition is to process a person's voice, as the sound of a person's voice or the words they use can provide useful information for making a medical diagnosis.

[0003] To detect pathologies from human voice, features can be extracted from the voice and processed by a mathematical model. The type and number of features extracted from the voice can affect the performance of the model, especially when the amount of training data for training the model is limited. Therefore, selecting appropriate features can improve the performance of the model. Summary of the Invention

[0004] Described herein are techniques for selecting audio features that are used to build or train mathematical models that detect or diagnose medical conditions. The techniques described herein can be used for any suitable medical condition, but for clarity, concussion and Alzheimer's disease are used as example medical conditions. However, the techniques described herein are not limited to any particular medical condition. [Brief explanation of the drawings]

[0005] The present invention and the following detailed description of certain embodiments thereof can be understood by reference to the following figures. [Figure 1]FIG. 1 is a schematic block diagram illustrating another embodiment of an audio-based medical evaluation system. [Figure 2] FIG. 1 is a schematic block diagram illustrating an embodiment of a system for processing audio data with a mathematical model to perform a medical diagnosis. [Figure 3] FIG. 1 is a schematic block diagram illustrating one embodiment of a training corpus of speech data. [Figure 4] FIG. 1 is a schematic block diagram illustrating one embodiment of a list of prompts for use in diagnosing a medical condition. [Figure 5] FIG. 1 is a schematic block diagram illustrating one embodiment of a system for selecting features to train a mathematical model for diagnosing a medical condition. [Figure 6A] FIG. 1 is a schematic block diagram illustrating one embodiment of a graphical representation of pairs of feature values ​​and diagnostic values. [Figure 6B] FIG. 10 is a schematic block diagram illustrating another embodiment of a graphical representation of pairs of feature and diagnostic values. [Figure 7] FIG. 1 is a schematic flow chart diagram illustrating one embodiment of a method for selecting features to train a mathematical model for diagnosing a medical condition. [Figure 8] FIG. 1 is a schematic flow chart diagram illustrating one embodiment of a method for selecting prompts to use with a mathematical model to diagnose a medical condition. [Figure 9] FIG. 1 is a schematic flow chart diagram illustrating one embodiment of a method for training a mathematical model to diagnose a medical condition appropriate to a set of selected prompts. [Figure 10] FIG. 1 is a schematic block diagram illustrating one embodiment of a computing device that can be used to train and deploy mathematical models to diagnose medical conditions. DETAILED DESCRIPTION OF THE INVENTION

[0006] Figure 1 illustrates an example system 100 for diagnosing a medical condition using a person's voice. Figure 1 includes a medical condition diagnosis service 140 that receives the person's voice data and processes the voice data to determine whether the person has a medical condition. For example, the medical condition diagnosis service 140 processes the voice data to calculate a "yes" or "no" decision regarding whether the person has a medical condition, or a probability or likelihood that the person has a medical condition, and / or A score can be calculated that indicates the severity of the condition.

[0007] As used herein, a diagnosis relates to any determination as to whether a person may have a medical condition, or any determination as to the possible severity of a medical condition. A diagnosis can include any form of assessment, conclusion, opinion, or judgment regarding a medical condition. In some cases, a diagnosis may be inaccurate, and a person diagnosed with a medical condition may not actually have the condition.

[0008] The medical condition diagnostic service 140 can receive the person's voice data using any suitable technique. For example, the person may speak into the mobile device 110, which can record the voice and transmit the recorded voice data to the medical condition diagnostic service 140 over the network 130. Any suitable technique and any suitable network can be used to transmit the voice data recorded by the mobile device 110 to the medical condition diagnostic service 140. For example, an application or "app" can be installed on the mobile device 110 and can communicate using a REST (representational state transfer) API (application programming interface). ) calls may be used to transmit voice data over the Internet or a mobile phone network. In another example, a healthcare provider may have a healthcare provider computer 120 that is used to record a person's voice and transmit the voice data to a medical condition diagnostic service 140.

[0009] In some implementations, medical condition diagnostic service 140 may be installed on mobile device 110 or healthcare provider computer 120, eliminating the need to transmit audio data over a network. The example of Figure 1B is not limiting, and any suitable technique may be used to transmit audio data for processing by the mathematical model.

[0010] The output of the medical condition diagnostic service 140 can then be used for any suitable purpose, for example, to present information to the person who provided the audio data or to a medical professional treating this person.

[0011] 2 is an example system 200 for processing audio data with a mathematical model to perform medical diagnosis. When processing the audio data, features can be calculated from the audio data, and these features can then be processed with the mathematical model. Any suitable type of feature can be used.

[0012] The features can include acoustic features, which are any features computed from the audio data without involving or relying on performing speech recognition on the audio data (e.g., the acoustic features do not use information about the speech data uttered in the audio data). For example, the acoustic features may include mel-frequency cepstral coefficients, perceptual linear prediction features, jitter, or shimmer.

[0013] The features can include linguistic features, where the linguistic features are computed using the results of speech recognition. For example, the linguistic features may include speaking rate (e.g., number of vowels or syllables per second), number of pause fillers (e.g., "ums" and "eres"), word difficulty (e.g., less commonly used words), or the portion of the phonetic sequence of the word following the filler.

[0014] 2, speech data is processed by an acoustic feature computation component 210 and a speech recognition component 220. The acoustic feature computation component 210 can compute acoustic features, such as any of the acoustic features described herein, from the speech data. The speech recognition component 220 can perform automatic speech recognition on the speech data using any suitable technique (e.g., Gaussian mixture models, acoustic modeling, language modeling, and neural networks).

[0015] Because the speech recognition component 220 may use acoustic features when performing speech recognition, some of the processing of these two components may overlap, and other configurations are possible. For example, the acoustic features component 210 could calculate the acoustic features required by the speech recognition component 220, thus eliminating the need for the speech recognition component 220 to calculate acoustic features altogether.

[0016] The linguistic feature computation component 230 can receive speech recognition results from the speech recognition component 220 and process the speech recognition results to determine linguistic features, such as any of the linguistic features described herein. The speech recognition features can be in any suitable format and can include any suitable information. For example, the speech recognition results can include a word lattice that includes multiple possible word sequences, information about fillers, and the timing of words, syllables, vowels, fillers, or any other units of speech.

[0017] The medical condition classifier 240 may process the acoustic and linguistic features through a mathematical model and output one or more diagnostic scores indicating whether the person has a medical condition, such as a score indicating the probability or likelihood that the person has the medical condition and / or a score indicating the severity of the medical condition. The medical condition classifier 240 may use any suitable technique, such as a classifier implemented with a support vector machine or a neural network such as a multilayer perceptron.

[0018] The performance of the pathology classifier 240 is based on the acoustic feature computation component. The accuracy of the features may depend on the features computed by the language feature computation component 210 and the language feature computation component 230. Furthermore, one set of features that provides correct processing for one condition may not provide correct processing for another condition. For example, speech difficulties may be an important feature for diagnosing Alzheimer's disease, but may not be useful for determining whether a person has a concussion. As another example, features related to the pronunciation of vowels, syllables, or words may be important for Parkinson's disease, but may be less important for other conditions. Thus, a technique is needed to determine a first set of features that provides correct processing for a first condition, and the process may need to be repeated to determine a second set of features that provides correct processing for a second condition.

[0019] In some implementations, medical condition classifier 240 may use other features, which may be referred to as non-vocal features, in addition to acoustic and linguistic features. For example, features may be derived from a person's demographic information (e.g., gender, age, location), information from medical history (e.g., weight, most recent blood pressure reading, or previous diagnoses), or any other suitable information. , or may be calculated from these.

[0020] The selection of features for diagnosing a medical condition may become even more important in situations where the amount of training data for training a mathematical model is relatively small. For example, training a mathematical model for diagnosing concussion may require training data that includes speech data from a large number of individuals immediately after experiencing a concussion. Such data may exist in small amounts, and obtaining additional examples of such data may require significant periods of time.

[0021] When training a mathematical model, the smaller the amount of training data, the greater the risk of overfitting. In this case, the mathematical model may adapt to specific training data, but the small amount of training data may cause the model to be unable to process new data correctly. For example, a model may be able to detect all concussions in the training data, but may have a high degree of difficulty when processing production data of people at risk of concussion. This can result in a high error rate.

[0022] One technique for preventing overfitting when training a mathematical model is to reduce the number of features used to train the mathematical model. The amount of training data required to train the model without overfitting increases as the number of features increases. Therefore, by using fewer features, it becomes possible to build a model using a smaller amount of training data.

[0023] When a model needs to be trained with a small number of features, it becomes increasingly important to select features that will enable the model to perform correctly. For example, when a large amount of training data is available, the model can be trained using hundreds of features, and the likelihood that the right features will be used is greater. Conversely, when only a small amount of training data is available, the model may be trained using as few as 10 features, and it becomes increasingly important to select the features that are most important for diagnosing the medical condition.

[0024] We now present examples of features that can be used to diagnose a medical condition. Acoustic features are calculated using short-time segment features. When processing audio data, the duration of this audio data may vary. For example, some audio may be one or two seconds long, while other audio may be several minutes or longer. For consistency when processing audio data, it is useful to process it in short-term segments (sometimes called frames). For example, each short-term segment may be 25 milliseconds long, and the segments may advance in 10-millisecond increments, with a 15-millisecond overlap between two consecutive segments.

[0025] The following are non-limiting examples of short-term segmental features: spectral features (such as mel-frequency cepstral coefficients or perceptual linear prediction), prosodic features (features like voicing tone, energy, probability), speech quality features (features like jitter, jitter of jitter, fluctuation, or harmonic to noise ratio), entropy (if entropy can be calculated posteriorly from acoustic models trained on natural speech data, e.g., how accurately an utterance is made, (pronounced) to capture).

[0026] Short-term segment features can be combined to compute acoustic features for speech. For example, a 2-second speech sample will generate 200 short-term segment features for pitch. These can be combined to compute one or more acoustic features for the tone.

[0027] Any suitable technique can be used to combine short-term segmentation features to generate a Acoustic features can be calculated. In some embodiments, acoustic features can be calculated using statistics of short-time segment features (e.g., arithmetic mean, standard deviation, skewness, kurtosis, 1st quartile, 2nd quartile, 3rd quartile, 2nd quartile minus 1st quartile, 3rd quartile minus 2nd quartile, 0.01 percentile, 0.99th percentile, 0.99th percentile minus 0.01 percentile), percentage of short-time segments whose values ​​are higher than a threshold (e.g., the threshold is 75% of the range plus the minimum), percentage of segments whose values ​​are higher than a threshold (e.g., the threshold is 90% of the range plus the minimum), slope of a linear approximation of the value, offset of a linear approximation of the value, linear error calculated as the difference between the linear approximation and the actual value, or quadratic error calculated as the difference between the linear approximation and the actual value). In some implementations, acoustic features can also be calculated as i-vectors or identity vectors of short-term segment features. The vector can be calculated using any suitable technique, such as performing a matrix-to-vector transformation using factorial analysis techniques and Gaussian mixture models.

[0028] The following are non-limiting examples of linguistic features: speaking rate, such as calculated by dividing the duration of all spoken words by the number of vowels, or any other suitable measure of speaking rate; the number of fillers, which may indicate hesitation in speech, calculated by (1) dividing the number of fillers by the duration of spoken words, or (2) dividing the number of fillers by the number of spoken words; and a measure of word difficulty or unusual word use. For example, word difficulty can be calculated using statistics of 1-gram probabilities of spoken words, such as by classifying words according to word frequency percentiles (e.g., 5%, 10%, 15%, 20%, 30%, or 40%). The speech parts of words following filler words, such as (1) the number of each part-of-speech class divided by the number of words spoken, or (2) the number of each part-of-speech class divided by the sum of the number of all parts-of-speech.

[0029] In some implementations, the linguistic features may also include determining whether a person answered a question correctly. For example, a person may be asked what year it is or who the president of the United States is. The person's speech can be processed to determine what the person said in response to the question and further to determine whether the person answered the question correctly.

[0030] To train a model to diagnose medical conditions, a corpus of training data can be collected that includes speech examples that indicate a person's diagnosis, such as whether they have no concussion, a mild, moderate, or severe concussion.

[0031] Figure 3 shows an example of a training corpus containing audio data for training a model to diagnose concussion. For example, in the table of Figure 3, rows may correspond to entries in a database. In this example, each entry includes an identifier for a person, a known diagnosis for that person (e.g., no concussion, mild, moderate, or severe concussion), an identifier for a prompt or question presented to the person (e.g., "How are you feeling today?"), and a filename for a file containing the audio data. The training data may be stored in any suitable format using any suitable storage technology.

[0032] The training corpus can store representations of human speech using any suitable format. For example, the speech data items in the training corpus may include digital samples of an audio signal received at a microphone, or may include processed versions of the audio signal, such as Mel-frequency cepstral coefficients.

[0033] A single training corpus may contain speech data for multiple medical conditions, or A separate training corpus may be used for each condition (e.g., a first training corpus for concussion and a second training corpus for Alzheimer's disease). A separate training corpus may be used to store speech data from people with unknown or undiagnosed medical conditions, since this training corpus can be used to train models for multiple medical conditions.

[0034] Figure 4 shows an example of stored prompts that can be used to diagnose a medical condition. Each prompt can be presented to a person, either by a person (e.g., a medical professional) or a computer, to obtain the person's speech in response to the prompt. Each prompt can have a prompt identifier so that it can be cross-referenced with prompt identifiers in a training corpus. The prompts in Figure 4 may be stored using any suitable storage technique, such as a database.

[0035] 5 illustrates an example system 500 that can be used to select features for training a mathematical model to diagnose a medical condition, and then use the selected features to train the mathematical model. System 500 can be used multiple times to select features for different medical conditions. For example, a first use of system 500 can select features for diagnosing a concussion, and a second use of system 500 can select features for diagnosing Alzheimer's disease.

[0036] 5 includes a training corpus 510 of speech data items for training a mathematical model for diagnosing a medical condition. The training corpus 510 may include any suitable information, such as speech data of a plurality of people with and without a medical condition, labels indicating whether a person has a medical condition, and any other information described herein.

[0037] The acoustic feature computation component 210, speech recognition component 220, and linguistic feature computation component 230 can be implemented as described above to compute acoustic and linguistic features for the speech data in the training corpus. The acoustic feature computation component 210 and linguistic feature computation component 230 can compute multiple features so that the best performing features can be determined. This is shown in FIG. 2, it is fair to say that this is in contrast to the case where these components are used in a production system and therefore only need to calculate previously selected features.

[0038] The feature selection score calculation component 520 may calculate a selection score for each feature (which may be an acoustic feature, a linguistic feature, or any other feature described herein). To calculate a selection score for a feature, a pair of numbers may be created for each speech data item in the training corpus. The first number in the pair is the value of the feature, and the second number in the pair is an indicator of a medical condition diagnosis. The value of the indicator of a medical condition diagnosis may have two values ​​(e.g., 0 if the person does not have the medical condition and 1 if the person has the medical condition) or may have more than one number (e.g., a real number between 0 and 1, or multiple integers indicating the likelihood or severity of the medical condition).

[0039] Thus, for each feature, a pair of numerical values ​​can be obtained for each speech data item in the training corpus. Figures 6A and 6B show two conceptual plots of the numerical value pairs for the first feature and the second feature. For Figure 6A, there does not appear to be a pattern or correlation between the values ​​of the first feature and the corresponding diagnostic values, while for Figure 6B, there appears to be a pattern or correlation between the values ​​of the second feature and the diagnostic values. Therefore, it can be concluded that the second feature is likely to be a useful feature for determining whether a person has a medical condition, while the first feature is not.

[0040] The feature selection score calculation component 520 can calculate a selection score for the feature using the feature value and diagnostic value pairs. The feature selection score calculation component 520 can calculate any suitable score that indicates a pattern or correlation between the feature values ​​and the diagnostic values. For example, the feature selection score calculation component 520 can calculate a Rand index, an adjusted Rand index, mutual information, adjusted mutual information, a Pearson correlation, an absolute Pearson correlation, a Spearman correlation, or an absolute Spearman correlation.

[0041] The selection score can indicate the usefulness of the feature in detecting a pathology, for example, a high selection score may indicate that a feature should be used when training a mathematical model, and a low selection score may indicate that the feature should not be used when training a mathematical model.

[0042] The feature stability determination component 530 can determine whether a feature (which may be an acoustic feature, a linguistic feature, or any other feature described herein) is stable or unstable. To perform the stability determination, the audio data items can be divided into groups, sometimes referred to as folds. For example, the audio data items may be divided into five folds. In one implementation, the audio data items may be divided into folds such that each fold has an approximately equal number of audio data items for different gender and age groups.

[0043] Statistics for each fold can be compared to statistics for other folds. For example, for the first fold, the median (or mean, or any other statistical value relating to the center or middle of a distribution) feature value (denoted M1) can be determined. Statistics can also be calculated for combinations of other folds. For example, for combinations of multiple other folds, the median (denoted M0) of the feature values ​​and a statistical measure of the variability of the feature values, such as the interquartile range, variance, or standard deviation (denoted V0) can be calculated. If the median of the first fold is too different from the median of the second fold, the feature can be determined to be unstable. For example,

[0044]

number

[0045] If , the feature can be determined to be unstable. where C is the scaling factor. This process can then be repeated for each of the other folds. For example, as described above, the median of the second fold can be compared to the median and variability of the other folds.

[0046] In one embodiment, after comparing each fold with the other folds, if the median of each fold is not too far from the medians of the other folds, the feature can be determined to be stable. Conversely, if the median of any fold is too far from the medians of the other folds, the feature can be determined to be unstable.

[0047] In some implementations, the feature stability determination component 530 may output a Boolean value for each feature to indicate whether the feature is stable or not. In some implementations, the stability determination component 530 may output a stability score for each feature. For example, the stability score may be calculated by dividing the largest distance (e.g., Mahalanobis distance) between the median of one fold and the median of another fold. It may also be calculated as the distance between the

[0048] The feature selection computation component 540 can receive the selection scores from the feature selection score computation component 520 and the stability determination from the feature stability determination component 530 and select a subset of features to be used to train the mathematical model. The feature selection component 540 can select the features that have the highest selection scores and are sufficiently stable.

[0049] In some implementations, the number of features to be selected (or the maximum number of features to be selected) may be preset. For example, the number N may be determined based on the amount of training data, and N features may be selected. The feature selection may be determined by removing unstable features (e.g., features determined to be unstable or features with stability scores below a threshold), and then selecting the N features with the highest selection scores.

[0050] In some implementations, the number of features selected may be based on the selection score and a stability determination, for example, feature selection may be determined by removing unstable features and then selecting all features with a selection score above a threshold.

[0051] In some embodiments, the selection score and stability score may be combined when selecting features. For example, a combined score may be calculated for each feature (such as by adding or multiplying the selection score and stability score for the feature), and this combined score may be used to select features.

[0052] The selected features can then be used by model training component 550 to train a mathematical model. For example, model training component 550 can iterate through the speech data items of the training corpus to obtain selected features for the speech data items, and then use the selected features to train the mathematical model. In some implementations, dimensionality reduction techniques, such as principal component analysis or linear discriminant analysis, may be applied to the selected features as part of model training. Any suitable mathematical model can be trained, such as any of the mathematical models described herein.

[0053] In some embodiments, other techniques such as wrapper methods may be used for feature selection or may be used in combination with the feature selection techniques described above. Wrapper methods can select a set of features, train a mathematical model using this selected set of features, and then use the trained model to evaluate the performance of the set of features. Possible Features If the number of features is relatively small and / or the training time is relatively short, all possible sets of features may be evaluated and the best performing set selected. When the number of features is relatively large and / or training time is a significant factor, optimization techniques may be used to iteratively find a set of features that performs well. In one embodiment, the system 500 may be used to select a set of features, and then a wrapper method may be used to select a subset of these features as the final set of features.

[0054] Figure 7 is a flow chart of an example embodiment of selecting features for training a mathematical model to diagnose a medical condition. In Figure 7 and other flow charts herein, the order of steps is exemplary; other orders are possible, not all steps are required, steps may be combined (in whole or in part) or subdivided, and some steps may be omitted or other steps may be added in some embodiments. Any of the methods described by the flow charts described herein may be implemented, for example, by any of the computers or systems described herein. It can be implemented by:

[0055] In step 710, a training corpus of speech data items is obtained. The training corpus may include any other suitable information, such as a representation of an audio signal of a person's speech, a medical diagnostic indication of the person from whom the speech was obtained, and any of the information described herein.

[0056] In step 720, speech recognition results are obtained for each speech data item in the training corpus. The speech recognition results may be pre-computed and stored with the training corpus, or may be stored elsewhere. The speech recognition results may be stored in any format, such as a transcript, a list of the highest scoring transcripts (e.g., a list of the N best), or a lattice of possible transcriptions. This may include any suitable information, as well as timing information such as the start and end times of words, fillers, or other speech units.

[0057] In step 730, acoustic features are computed for each speech data item in the training corpus. The acoustic features may include any features computed without using speech recognition results for the speech data item, such as any of the acoustic features described herein. The acoustic features may include or be computed from data used in the speech recognition process (e.g., mel-frequency cepstral coefficients or perceptual linear predictors), but the acoustic features do not use speech recognition results, such as information about words or fillers present in the speech data item.

[0058] Linguistic features are computed for each speech data item in the training corpus in step 740. The linguistic features may include any features computed using speech recognition results, such as any of the linguistic features described herein.

[0059] In step 750, a feature selection score is calculated for each acoustic and linguistic feature. To calculate the feature selection score for a feature, the value of the feature for each speech data item in the training corpus may be used along with other information, such as known diagnostic values ​​corresponding to the speech data item. The feature selection score may be calculated using any of the techniques described herein, such as by calculating absolute Pearson correlation. In some implementations, feature selection scores may be calculated for other features as well, such as features related to a person's demographic information.

[0060] In step 760, the feature selection scores are used to select a number of features. For example, a number of features with the highest selection scores may be selected. may compute a stability measure for each feature, and may use both the feature selection score and the stability measure to select multiple features, such as by using any of the techniques described herein.

[0061] In step 770, a mathematical model is trained using the selected features. Any suitable mathematical model may be trained, such as a neural network or a support vector machine. After the mathematical model is trained, it may be deployed in a production system, such as speech module 104, system 109, etc., of FIG. 1B, to perform diagnosis of a medical condition.

[0062] The steps of Figure 7 can be performed in a variety of ways. For example, in one embodiment, steps 730 and 740 may be performed in a loop, repeatedly performed for each speech data item in the training corpus. In a first iteration, acoustic and linguistic features may be calculated for a first speech data item, and in a second iteration, acoustic and linguistic features may be calculated for a second speech data item. Acoustic and linguistic features may be computed for the eyes, and so on.

[0063] When using the deployed model to diagnose a medical condition, a series of prompts or questions can be uttered to the person to obtain speech from the person being diagnosed. Any suitable prompts can be used, such as any of the prompts in Figure 4. After features are selected as described above, the prompts can be selected such that they provide useful information about the selected features.

[0064] For example, suppose the selected feature is pitch. Pitch is used to diagnose a medical condition. However, to obtain a useful pitch feature, Some prompts may be better than others: very short utterances (e.g., yes / no answers) may not provide enough data to accurately calculate tone, so prompts that generate longer responses can be more useful in obtaining information about tone.

[0065] As another example, suppose the selected feature is word difficulty. Although word difficulty has been determined to be a useful feature for diagnosing medical conditions, some prompts may be better than others at obtaining useful word difficulty features. A prompt that asks a user to read a presented passage generally results in the words in the passage being spoken, and therefore the word difficulty feature will have the same value each time the prompt is presented. This means that the prompt is not useful for obtaining information about word difficulty. In contrast, an open-ended question such as "Tell me about your day" will result in greater vocabulary diversity in the responses and therefore may provide more useful information about word difficulty.

[0066] Selecting a set of prompts can also improve the performance of a system for diagnosing a medical condition and provide a better experience for the person being assessed. Using the same set of prompts for each person can enable a system for diagnosing a medical condition to achieve more accurate results because data collected from multiple people is easier to compare than if different prompts were used for each person. Furthermore, using a fixed set of prompts makes it easier to predict a person's ratings and the desired duration of the rating appropriate for assessing a medical condition. For example, to assess whether a person has Alzheimer's disease, it is acceptable to use more prompts to collect a greater amount of data. However, to assess whether a person suffered a concussion at a sporting event, it may be necessary to use fewer prompts to obtain results more quickly.

[0067] In some implementations, prompts may be selected by calculating a prompt selection score. The training corpus may have multiple speech data items for a prompt, or even many speech data items. For example, the training corpus may include examples of prompts used by different people, or the same prompt may be used multiple times by the same person.

[0068] Figure 8 is a flow chart of an example embodiment of selecting a prompt to use with a deployed model to diagnose a medical condition. Steps 810 through 840 may be performed for each prompt (or subset of prompts) in the training corpus to calculate a prompt selection score for each prompt.

[0069] In step 810, a prompt is obtained, and in step 820, a speech data item corresponding to the prompt is obtained from the training corpus. A medical diagnostic score is calculated for each audio data item corresponding to the prompt, in step 830. For example, the medical diagnostic score for an audio data item may be a numerical value output by a mathematical model (e.g., the mathematical model trained in FIG. 7) that indicates the likelihood that a person has a medical condition and / or the severity of that condition.

[0070] In step 840, the calculated medical diagnostic score is used to calculate a prompt selection score for the prompt. Calculating the prompt selection score may be similar to calculating the feature selection score, as described above. For each audio data item corresponding to a prompt, a pair of numerical values ​​may be obtained. For each pair, the first numerical value of the pair may be the medical diagnostic score calculated from the audio data item, and the second numerical value of the pair may be a known medical condition diagnosis for the person (e.g., known that the person has a medical condition or is indicative of the severity of the condition). Plotting these pairs of numerical values ​​results in a plot similar to FIG. 6A or 6B, where, depending on the prompt, there may or may not be a pattern or correlation between the pairs of numerical values.

[0071] The prompt selection score for a prompt may include any score that indicates a pattern or correlation between the calculated medical diagnostic score and a known medical condition diagnosis. For example, the prompt selection score may include a Rand index, an adjusted Rand index, mutual information, adjusted mutual information, a Pearson correlation, an absolute Pearson correlation, a Spearman correlation, or an absolute Spearman correlation.

[0072] In step 850, it is determined whether any more prompts remain to be processed. If so, processing can proceed to step 810 where additional prompts can be processed. If all prompts have been processed, processing can proceed to step 860.

[0073] In step 860, the prompt selection scores are used to select a number of prompts, e.g., a number of prompts with the highest prompt selection scores. In some implementations, a stability judgment may be calculated for each prompt, and both the prompt selection score and the prompt stability score may be used to select multiple prompts, such as by using any of the techniques described herein.

[0074] The selected prompts are used with the deployed medical condition diagnosis service in step 870. For example, when diagnosing a person, the selected prompts can be presented to the person and the person's voice can be obtained in response to each of the prompts.

[0075] In some implementations, other techniques, such as wrapper methods, may be used for prompt selection or may be used in combination with the prompt selection techniques presented above. In some implementations, a set of prompts may be selected using the process of Figure 8, and then a subset of these prompts may be selected using wrapper methods as the final set of features.

[0076] In some embodiments, a person involved in creating the medical condition diagnostic service may assist in the selection of prompts. This person can use their knowledge or experience to select prompts based on the selected characteristics. For example, if the selected characteristic is word difficulty, this person can review the prompts and select those that are more likely to provide useful information about word difficulty. This person can then select useful information for each selected characteristic. Simply select one or more prompts that are likely to provide the relevant information.

[0077] In one embodiment, the person can review the prompts selected by the process of Figure 8 and add or remove prompts to improve the performance of the medical condition diagnosis system. For example, two prompts may each provide useful information about the difficulty of a word, but the information provided by these two prompts may be so redundant that using both prompts may not provide a significant benefit over using only one of them.

[0078] In some embodiments, after prompt selection, a second mathematical model appropriate to the selected prompt can be trained. The mathematical model trained in FIG. 7 can process a single utterance (in response to the prompt) and generate a medical diagnostic score. The process of performing a diagnosis may include processing multiple utterances corresponding to multiple prompts, and then processing each of the utterances through the mathematical model of Figure 7 to generate multiple medical diagnostic scores. It may be necessary to combine the multiple medical diagnostic scores in some way to determine an overall medical diagnosis. Thus, the mathematical model trained in Figure 7 may not be appropriate for the selected set of prompts.

[0079] When the selected prompts are used in a session to diagnose a person, each of the prompts can be presented to the person to obtain an utterance corresponding to each of the prompts. Instead of processing the utterances separately, the utterances can be processed simultaneously by the model to generate a medical diagnostic score. Thus, the model can adapt to the selected prompts because it is trained to simultaneously process utterances corresponding to each of the selected prompts.

[0080] 9 is a flow chart of an example embodiment for training a mathematical model appropriate for a set of selected prompts. In step 910, a first mathematical model is obtained, such as by using the process of FIG. 7. In step 920, a plurality of prompts are selected using the first mathematical model, such as by the process of FIG. 8.

[0081] In step 930, a second mathematical model is trained to simultaneously process multiple speech data items corresponding to multiple selected prompts to generate a medical diagnosis score. When training the second mathematical model, a training corpus including a session with speech data items corresponding to each of the multiple selected prompts can be used. When training this mathematical model, inputs to the mathematical model may be fixed to the speech data items from the session and corresponding to each of the selected prompts. The output of the mathematical model may be fixed to a known medical diagnosis.

[0082] The parameters of this model can then be trained to optimally process audio data items to simultaneously produce a medical diagnostic score. Any suitable training technique can be used, such as stochastic gradient descent.

[0083] The second mathematical model can then be deployed as part of a medical condition diagnosis service, such as the speech module 104, the service of Figure 1. Because the second mathematical model is trained to process the utterances simultaneously, rather than individually, the second mathematical model can outperform the first mathematical model, i.e., the training can combine information from all the utterances to produce a more accurate medical condition diagnosis score.

[0084] FIG. 10 illustrates components of one embodiment of a computing device 1000 for implementing any of the techniques described above. In FIG. 10, the components include: Although shown as being on one computing device, components may be distributed across multiple computing devices, such as in a system of computing devices that includes end-user computing devices (e.g., smart phones or tablets) and / or server computing devices (e.g., cloud computing).

[0085] The computing device 1000 may include any components typical of a computing device, such as volatile or non-volatile memory 1010, one or more processors 1011, and one or more network interfaces 1012. The computing device 1000 may also include any input and output components, such as a display, a keyboard, and a touch screen. The computing device 1000 may also include various components or modules that provide specific functionality, and these components or modules may be implemented in software, hardware, or a combination thereof. Various example components are described below as an example implementation; however, other implementations may include additional components or may exclude some of the components described below.

[0086] The computing device 1000 may have an acoustic feature computation component 1021 that can compute acoustic features for an audio data item as described above. The computing device 1000 may have a linguistic feature computation component 1022 that can compute linguistic features for an audio data item as described above. The computing device 1000 may have a speech recognition component 1023 that can generate speech recognition results for an audio data item as described above. The computing device 1000 may have a feature selection score computation component 1031 that can compute selection scores for features as described above. The computing device 1000 may have a feature stability score computation component 1032 that can perform or compute stability scores as described above. The computing device 1000 may have a feature selection component 1033 that can select features using the selection scores and / or stability determinations as described above. The computing device 1000 may have a prompt selection score computation component 1041 that can compute selection scores for prompts as described above. The computing device 1000 can have a prompt stability score calculation component 1042 that can make a stability determination or calculate a stability score as described above. The computing device 1000 can have a prompt selection component 1043 that can select a prompt using the selection score and / or the stability determination as described above. The computing device 1000 can have a model training component 1050 that can train a mathematical model as described above. The computing device 1000 can have a medical condition diagnosis component 1060 that can process the audio data items to determine a medical diagnosis score as described above.

[0087] Computing device 1000 may include or have access to various data stores, such as training corpus data store 1070. The data stores may use any well-known storage technology, such as files, relational or non-relational databases, or any non-transitory computer-readable medium.

[0088] The methods and systems described herein may be implemented in part or in whole by a computer. The data software, program code, and / or instructions may also be deployed by a machine that executes the data software, program code, and / or instructions on a processor. As used herein, "processor" is meant to include at least one processor, and the plural and singular should be understood as interchangeable unless the context clearly indicates otherwise. Any aspect of the disclosure may be a method on a machine, a system or apparatus as part of or relating to a machine, or a computer-readable medium executing on one or more machines. The present invention may be realized as a computer program product embodied in a removable medium. The processor may be part of a server, client, network infrastructure, mobile computing platform, stationary computing platform, or other computing platform. The processor may be any type of computing or processing device capable of executing program instructions, code, binary instructions, etc. The processor may be or include a single processor, digital processor, embedded processor, microprocessor, or any variation such as a coprocessor (e.g., math coprocessor, graphics coprocessor, communications coprocessor, etc.) that can directly or indirectly facilitate the execution of stored program code or program instructions. Additionally, the processor may enable the execution of multiple programs, threads, and code. Multiple threads may be executed simultaneously to improve processor performance and facilitate concurrent processing of applications. In one embodiment, the methods, program codes, program instructions, etc. described herein may be implemented in one or more threads. A thread may spawn other threads, which may then be processed by the threads. The threads may be associated with one another and assigned priorities, and the processor may execute these threads based on priority or any other order based on instructions provided in the program code. The processor may include memory that stores methods, codes, instructions, and programs as described herein and elsewhere. The processor may access a storage medium through an interface that may store methods, codes, and instructions as described herein and elsewhere. Storage media associated with the processor for storing methods, programs, codes, program instructions, or other types of instructions that can be executed by a computing or processing device may include, but are not limited to, one or more of a CD-ROM, DVD, memory, hard disk, flash drive, RAM, ROM, cache, etc.

[0089] A processor may include one or more cores, which can increase the speed and performance of a multiprocessor. In embodiments, a process may be a dual-core processor, a quad-core processor, or other chip-level multiprocessor that combines two or more independent cores (called a die).

[0090] The methods and systems described herein may be deployed, in part or in whole, by machines executing computer software on servers, clients, firewalls, gateways, hubs, routers, or other such computer and / or networking hardware. Software programs may be associated with servers, which may include file servers, print servers, domain servers, Internet servers, intranet servers, and other variants such as secondary servers, host servers, distributed servers, etc. Servers may include one or more of the following: memory, processors, computer-readable media, storage media, ports (physical and virtual), communication devices, and interfaces that allow access to other servers, clients, machines, and devices through wired or wireless media. Methods, programs, or code as described herein and elsewhere may be executed by servers. In addition, methods as described herein may be implemented by a server. Other devices required for the performance of the method may be considered part of the infrastructure associated with the server.

[0091] A server may provide an interface to other devices, including, but not limited to, clients, other servers, printers, database servers, print servers, file servers, communication servers, distributed servers, etc. Additionally, this coupling and / or connection may facilitate remote execution of programs across a network. Networking some or all of these devices may facilitate parallel processing of a program or method in one or more locations without departing from the scope of this disclosure. Additionally, any device attached to a server via an interface may include at least one storage medium capable of storing methods, programs, code, and / or instructions. A central repository may provide program instructions for execution on different devices. In this embodiment, a remote repository may act as a storage medium for program code, instructions, and programs.

[0092] Software programs may also be associated with clients. Clients may include file clients, print clients, domain clients, Internet clients, intranet clients, and other variations such as secondary clients, host clients, distributed clients, etc. Clients may include one or more of memory, processors, computer-readable media, storage media, ports (physical and virtual), communication devices, and interfaces that can access other clients, servers, machines, and devices through wired or wireless media. Methods, programs, or code as described herein and elsewhere may be executed by a client. Additionally, other devices required for execution of methods as described herein may be considered part of the infrastructure associated with the client.

[0093] A client may provide an interface to other devices, including, but not limited to, servers, other clients, printers, database servers, print servers, file servers, communication servers, distributed servers, etc. Additionally, this coupling and / or connection may facilitate remote execution of programs across a network. Networking some or all of these devices may facilitate parallel processing of a program or method in one or more locations without departing from the scope of this disclosure. Additionally, any device attached to a client via an interface may include at least one storage medium capable of storing methods, programs, applications, code, and / or instructions. A central repository may provide program instructions for execution on different devices. In this embodiment, a remote repository may act as a storage medium for program code, instructions, and programs.

[0094] The methods and systems described herein may also be deployed, in part or in whole, over a network infrastructure. The network infrastructure may include elements such as computing devices, servers, routers, hubs, firewalls, clients, personal computers, communication devices, routing devices, and other active and passive devices, modules, and / or components known in the art. The computing and / or non-computing device(s) associated with the network infrastructure may include, among other components, storage media such as flash memory, buffers, stacks, RAM, ROM, etc. The processes, methods, program codes, and instructions described herein and elsewhere may be executed by one or more of the network infrastructure elements.

[0095] The methods, program codes, and instructions described herein and elsewhere may also be implemented on a cellular network having multiple cells. The cellular network may be either a Frequency Division Multiple Access (FDMA) network or a Code Division Multiple Access (CDMA) network. The cellular network may include mobile devices, cell sites, base stations, repeaters, antennas, towers, etc. The cellular network may be a GSM, GPRS, 3G, EVDO, mesh, or other network type.

[0096] The methods, program codes, and instructions described herein and elsewhere may also be implemented on or through a mobile device. Mobile devices may include navigation devices, cell phones, mobile telephones, mobile personal digital assistants, laptops, palmtops, netbooks, pagers, e-readers, music players, etc. These devices may include, among other components, storage media such as flash memory, buffers, RAM, ROM, and one or more computing devices. A computing device associated with a mobile device may be capable of executing program code, methods, and instructions stored thereon. Alternatively, a mobile device may be configured to execute instructions in cooperation with other devices. A mobile device may be configured to communicate with a base station interfaced with a server and execute program code. A mobile device may also communicate over a peer-to-peer network, a mesh network, or other communications network. Program code may be stored on a storage medium associated with a server and executed by a computing device embedded within the server. A base station may include a computing device and a storage medium. The storage device may store program codes and instructions executed by computing devices associated with the base station.

[0097] The computer software, program code, and / or instructions may be stored on and / or accessed from a machine-readable medium. Machine-readable media can include computer components, devices, and recording media that hold digital data used for calculations at certain intervals of time; semiconductor storage known as random access memory (RAM); mass storage, typically for more permanent storage, such as optical disks, hard disks, tapes, drums, cards, and other types of magnetic storage; processor registers, cache memory, volatile memory, non-volatile memory; optical storage such as CDs and DVDs; removable media such as flash memory (e.g., USB sticks or keys), floppy disks, magnetic tape, paper tape, punch cards, standalone RAM disks, Zip drives, removable mass storage, offline, etc.; and other computer memory such as dynamic memory, static memory, read / write storage, mutable storage, read-only, random access, sequential access, position-addressable, file-addressable, content-addressable, network-attached storage, storage area networks, bar code, magnetic ink, etc.

[0098] The methods and systems described herein can transform physical and / or intangible items from one state to another. and the system can transform data representing physical and / or intangible items from one state to another.

[0099] Throughout the figures, elements included in flow charts and block diagrams and described and illustrated herein imply logical boundaries between the elements. However, depending on software and hardware design practices, the illustrated elements and their functions may be implemented on a machine having a processor via a computer-executable medium. The processor may execute program instructions stored on the medium as a monolithic software structure, as a standalone software module, or as a module employing external routines, code, services, etc., or any combination thereof; all such implementations fall within the scope of this disclosure. Examples of such machines may include, but are not limited to, personal digital assistants, laptops, personal computers, mobile phones, other handheld computing devices, medical equipment, wired or wireless communication devices, transducers, chips, calculators, satellites, tablet PCs, e-books, gadgets, electronic devices, devices with artificial intelligence, computing devices, networking equipment, servers, routers, etc. Furthermore, elements illustrated in flow charts and block diagrams, or any other logical component, may be implemented on a machine capable of executing program instructions. In other words, while the foregoing figures and description clearly set forth functional aspects of the disclosed system, the particular configuration of software for implementing those functional aspects should not be inferred from these descriptions unless expressly stated or otherwise apparent from the context. Similarly, it will be recognized that the various steps identified and described above may be varied in many ways, and that the order of steps may be altered to suit particular applications of the techniques disclosed herein. All such variations and modifications are intended to fall within the scope of this disclosure. However, Therefore, the illustration and / or description of the order of various steps should not be construed as requiring a particular order of performance of these steps unless required by a particular application or unless expressly stated or otherwise clear from the context.

[0100] The methods and / or processes described above, and their steps, may be implemented in hardware, software, or any combination of hardware and software suitable for a particular application. Hardware may include general-purpose computers and / or dedicated computing devices, or specific computing devices, or particular aspects or components of specific computing devices. The processes may be implemented in one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors, or other programmable devices, together with internal and / or external memory. Additionally or alternatively, the processes may be embodied in an application-specific integrated circuit, a programmable gate array, programmable array logic, or any other device or combination of devices that can be configured to process electronic signals. It will be appreciated that one or more of the processes may be implemented as computer-executable code that can be executed on a machine-readable medium.

[0101] The computer executable code may be written using a structured programming language such as C, an object-oriented programming language such as C++, or any other high-level or low-level programming language (including assembly language, hardware description languages, and database programming languages ​​and techniques), and may be executed on one of the above devices, and on a processor, processor architecture, or on different hardware. and software combinations, or any other machine capable of executing program instructions.

[0102] Thus, in one aspect, each of the methods and combinations described above may be embodied in computer-executable code, which, when executed on one or more computing devices, performs the steps. In other aspects, the methods may be embodied in a system that performs the steps, and may be distributed across multiple devices in a number of ways, or all of the functionality may be integrated into a dedicated, stand-alone device or other hardware. In other aspects, the means for performing the steps associated with the processes described above may include any of the hardware and / or software described above. All such permutations and combinations are intended to fall within the scope of the present disclosure.

[0103] While the present invention has been disclosed in connection with the preferred embodiment shown and described in detail above, various changes and modifications thereof will become readily apparent to those skilled in the art. Accordingly, the spirit and scope of the invention is not to be limited by the foregoing description, but is to be understood in the broadest sense permitted by law.

[0104] All documents cited herein are hereby incorporated by reference.

Claims

1. 1. A system for training a mathematical model for detecting a medical condition, comprising: The system comprises at least one computer; The at least one computer obtaining a training corpus including a plurality of speech data items, each speech data item having an associated diagnostic value, each speech data item of the training corpus corresponding to a prompt of a plurality of prompts; Computing a plurality of features for each speech data item in the training corpus; selecting, for each audio data item, a subset of the plurality of features based on a diagnostic value corresponding to the audio data item and a usefulness of the features for detecting the medical condition; for each speech data item of the training corpus, training the mathematical model for detecting the pathology using the subset of the plurality of features; computing a medical diagnostic score for each speech data item of the training corpus by processing the plurality of speech data items using the mathematical model; calculating a prompt selection score for each prompt of the plurality of prompts using the medical diagnostic score; selecting a subset of the plurality of prompts from the plurality of prompts using the prompt selection scores; deploying a computer program product or computer service for detecting the medical condition using the mathematical model and the subset of the plurality of prompts; presenting, by the computer program product or the computer service, a prompt to a person, the prompt being from the subset of the plurality of prompts; receiving, by the computer program product or the computer service, in response to the prompt, an audio data item corresponding to a human voice; computing a medical condition diagnosis by processing the received audio data items using the mathematical model; displaying the medical condition diagnosis by said computer program product or said computer service; A system that is configured to:

2. The at least one computer obtaining a speech recognition result for each speech data item of the training corpus, the speech recognition result for a speech data item including a transcription of the speech data item; computing linguistic features for each speech data item of the training corpus by processing the speech recognition results; and The system of claim 1 , wherein the plurality of features includes linguistic features.

3. The at least one computer receiving, for each prompt of the subset of the plurality of prompts, a speech data item corresponding to a speech of the person; computing a medical condition diagnosis for the person by processing the received audio data items using the mathematical model; and The system of claim 1 configured to:

4. The at least one computer computing speech features for each speech data item of said training corpus; and the speech features are calculated from the speech data item, and the calculation of the speech features does not use a speech recognition result of the speech data item; The system of claim 1 , wherein the plurality of features includes the audio features.

5. The system described in claim 4, wherein the plurality of features include linguistic features calculated from speech recognition results of speech data items.

6. The system described in claim 1, wherein the mathematical model includes a neural network or a support vector machine.

7. The system described in claim 1, wherein the plurality of features include at least one of spectral features, prosodic features, or voice quality features.

8. 1. A computer-implemented method for training a mathematical model for detecting a medical condition, comprising: obtaining a training corpus including a plurality of speech data items, each speech data item having an associated diagnostic value, each speech data item of the training corpus corresponding to a prompt of a plurality of prompts; Computing a plurality of features for each speech data item in the training corpus; selecting, for each audio data item, a subset of the plurality of features based on a diagnostic value corresponding to the audio data item and a usefulness of the features for detecting the medical condition; for each speech data item of the training corpus, training the mathematical model for detecting the pathology using the subset of the plurality of features; computing a medical diagnostic score for each speech data item of the training corpus by processing the plurality of speech data items using the mathematical model; calculating a prompt selection score for each prompt of the plurality of prompts using the medical diagnostic score; selecting a subset of the plurality of prompts from the plurality of prompts using the prompt selection scores; deploying a computer program product or computer service for detecting the medical condition using the mathematical model and the subset of the plurality of prompts; presenting, by the computer program product or the computer service, a prompt to a person, the prompt being from the subset of the plurality of prompts; receiving, by the computer program product or the computer service, in response to the prompt, an audio data item corresponding to a human voice; computing a medical condition diagnosis by processing the received audio data items using the mathematical model; displaying the medical condition diagnosis by said computer program product or said computer service; 10. A computer-implemented method comprising:

9. The computer-implemented method of claim 8, wherein the condition is a concussion or Alzheimer's disease.

10. The computer-implemented method of claim 8, wherein the plurality of features include one or more of the number of filler words over a period of time, the number of filler words over a number of words, word difficulty, or speaking rate.

11. The computer-implemented method of claim 8, further comprising calculating a feature selection score for a feature by generating a pair of numerical values ​​for each speech data item in the training corpus, a first numerical value of the pair corresponding to a feature value and a second numerical value of the pair corresponding to a diagnostic value.

12. The computer-implemented method comprising: Dividing the training corpus into a plurality of folds; calculating a statistic for each feature and each fold of said plurality of folds; The computer-implemented method of claim 8 , comprising:

13. The computer-implemented method comprising: computing a stability determination for each feature of the plurality of features using the statistics for each feature and each fold of the plurality of folds; selecting the subset of the plurality of features using the stability determination; and The computer-implemented method of claim 12, comprising:

14. one or more non-transitory computer-readable media containing computer-executable instructions that, when executed, cause at least one processor to perform a plurality of actions; The plurality of actions include: obtaining a training corpus including a plurality of speech data items, each speech data item having an associated diagnostic value, each speech data item of the training corpus corresponding to a prompt of a plurality of prompts; obtaining a plurality of features for each speech data item in the training corpus; selecting, for each audio data item, a subset of the plurality of features based on a diagnostic value corresponding to the audio data item and a usefulness of the features for detecting a medical condition; For each speech data of the training corpus, training a mathematical model for detecting the pathology using the subset of the plurality of features; computing a medical diagnostic score for each speech data item of the training corpus by processing the plurality of speech data items using the mathematical model; calculating a prompt selection score for each prompt of the plurality of prompts using the medical diagnostic score; selecting a subset of the plurality of prompts from the plurality of prompts using the prompt selection scores; deploying a computer program product or computer service for detecting the medical condition using the mathematical model and the subset of the plurality of prompts; presenting, by the computer program product or computer service, a prompt to a person, the prompt being from the subset of the plurality of prompts; receiving, by said computer program product or computer service, in response to said prompt, an audio data item corresponding to a human voice; computing a medical condition diagnosis by processing the audio data items using the mathematical model; displaying said medical condition diagnosis by said computer program product or computer service; one or more non-transitory computer-readable media, 15. The method of claim 1, wherein the first feature of the plurality of features is calculated. calculating a value for each short-time segment of the audio signal to obtain a plurality of values; calculating the first feature using the plurality of values; 15. One or more non-transitory computer-readable media according to claim 14, comprising:

16. The plurality of actions includes calculating a feature selection score for each feature of the plurality of features; The one or more non-transitory computer-readable media of claim 14 , wherein the feature selection score comprises an adjusted Rand index, an adjusted mutual information, an absolute Pearson correlation, or an absolute Spearman correlation.

17. The plurality of actions are: computing a stability measure for each feature of the plurality of features; selecting the plurality of features using the stability determination; and 15. One or more non-transitory computer-readable media according to claim 14, comprising:

18. The method of claim 17, wherein each speech data item in the training corpus corresponds to one prompt of a plurality of prompts, the plurality of prompts including the presented prompt; The plurality of actions include: computing a medical diagnostic score for each speech data item of the training corpus by processing the plurality of speech data items using the mathematical model; calculating a prompt selection score for each prompt of the plurality of prompts using the medical diagnostic score; selecting a subset of the plurality of prompts from the plurality of prompts using the prompt selection score, the subset of the plurality of prompts including the presented prompt; and deploying the computer program product or the computer service to detect the medical condition using the mathematical model and the subset of the plurality of prompts; 15. One or more non-transitory computer-readable media according to claim 14, comprising:

19. One or more non-transitory computer-readable media as described in claim 14, wherein the plurality of features includes non-audio features.

Images