Systems and methods for assessing behavioral disorders, developmental delays, and neurological disorders

Abstract

[0003] Described herein are systems and methods for use in assessing individuals, such as children, for behavioral disorders, developmental delays, and neurological disorders. An exemplary method includes receiving input data for an individual associated with a behavioral disorder, neurological disorder, or developmental delay, and evaluating the input data using an assessment module comprising at least one machine learning model to thereby generate an assessment result, the machine learning model comprising one or more decision threshold hyperparameters that distinguish between a positive assessment, a negative assessment, and an indeterminate assessment regarding the presence or absence of the behavioral disorder, neurological disorder, or developmental delay.

Description

[Technical Field]

[0001] This application claims the benefit of U.S. Patent Application No. 63 / 416,420, filed October 14, 2022, and U.S. Patent Application No. 63 / 416,422, filed October 14, 2022, each of which is incorporated by reference in its entirety. [Background technology]

[0002] Many individuals, including children, suffer from behavioral disorders, developmental delays, and neurological disorders, including attention deficit hyperactivity disorder (ADHD), autism (e.g., autism spectrum disorder, ASD), and speech disorders.

[0003] Healthcare providers typically use traditional observational methods, such as questionnaires and clinician interviews, to assess behavioral disorders, developmental delays, and neurological disorders. Summary of the Invention

[0004] Described herein are methods, devices, systems, software, and platforms used to evaluate individuals, such as children, for behavioral disorders, developmental delays, and neurological disorders. Also described herein are methods, devices, systems, software, and platforms used to improve the accuracy and efficiency of evaluating individuals with one or more behavioral disorders, developmental delays, and neurological disorders. Compared to conventional techniques, the methods, devices, systems, software, and platforms described herein can utilize input data related to an individual and generate a diagnosis with high accuracy. The methods, devices, systems, software, and platforms of the present invention described herein are designed, at least in part, to provide treatment to individuals suffering from behavioral disorders, developmental delays, and neurological disorders.

[0005] It can be difficult to accurately and efficiently evaluate behavioral disorders, developmental delays, and neurological disorders. A potential reason is that these health disorders often have overlapping symptoms, making them difficult to distinguish from one another. For example, one type of developmental delay, such as autism, has overlapping symptoms with another type of developmental delay, such as speech delay. Another example is that behavioral disorders (e.g., ADHD) have overlapping symptoms with developmental delays (e.g., autism). Therefore, patients often receive incorrect or incomplete diagnoses, and patients with multiple health disorders are diagnosed with only one disease.

[0006] Questionnaires and clinician interviews are often used to diagnose patients by collecting multiple types of data, including various test findings.Both questionnaires and clinician interviews involve a long set of questions that are administered to patients and / or their respective caregivers, which can be costly and inefficient in terms of time and resources.Questionnaire data can be of limited value because the quality of the data depends heavily on the subject's attention span and willingness to participate.Therefore, questionnaire data can be incomplete or inaccurate.

[0007] Another challenge in diagnosing patients, especially young patients suffering from behavioral disorders, developmental delays, and neurological disorders, is timing. Although a reliable autism diagnosis is possible as early as 18 months, the average age of diagnosis in the United States remains above 4 years. Multiple factors contribute to diagnostic delays, including structural inequities such as poverty, racism, and gender bias. While several innovative physician training programs have demonstrated the feasibility of diagnosing health issues (e.g., autism) in young patients in primary care settings, many primary care providers still report being under-equipped and / or understaffed. Existing diagnostic tools for autism are often difficult to use, despite reasonable accuracy and inter-rater reliability. These tools can take a long time to administer, may not be amenable to telehealth, and often require expert training.

[0008] In contrast, methods, devices, systems, software, and platforms are described herein for accurately and efficiently assessing individuals for at least one disease type selected from a plurality of diseases, including behavioral disorders, developmental delays, and / or neurological disorders. Also described herein are methods, devices, systems, software, and platforms for assessing individuals using trained models having one or more decision threshold hyperparameters that provide improved coverage, accuracy, and / or other performance metrics. The output of the trained model includes a positive, negative, or inconclusive assessment.

[0009]

[0003] In some aspects, computer-implemented methods are disclosed herein that include: (a) receiving individual input data related to a behavioral disorder, neurological disorder, or developmental delay; (b) evaluating the input data using an assessment module comprising at least one machine learning model, thereby generating an assessment result, wherein the at least one machine learning model comprises one or more decision threshold hyperparameters that distinguish between a positive assessment, a negative assessment, and an indeterminate assessment regarding the presence or absence of the behavioral disorder, neurological disorder, or developmental delay; and (c) if the assessment result includes the presence of the behavioral disorder, neurological disorder, or developmental delay, generating a personal therapeutic treatment plan for the individual based at least in part on the assessment result.

[0010] In some embodiments, at least one machine learning model comprises one or more decision threshold hyperparameters that, when evaluated using a nested cross-validation procedure, provide a positive predictive value of at least about 80%, a negative predictive value of at least about 95%, a coverage or inclusion rate of at least about 70%, or any combination thereof. In some embodiments, the one or more decision threshold hyperparameters are generated using an automated cross-validation procedure. In some embodiments, the one or more decision threshold hyperparameters define a threshold range for determining whether an evaluation result is a positive evaluation, a negative evaluation, or an indeterminate evaluation.

[0011] In some embodiments, the first classification determination of the presence or absence of a behavioral disorder, neurological disorder, or developmental delay in an individual is based on a particular sensitivity, a particular specificity, a particular negative predictive value, or a particular positive predictive value.

[0012] In some embodiments, the at least one machine learning model comprises a subset of the plurality of tunable machine learning models.

[0013] In some embodiments, the method further includes (a) requesting additional data if the assessment result includes an indeterminate assessment, and (b) generating an updated assessment result based on the additional data using the assessment module. In some embodiments, the method further includes (a) combining the scores for each of the plurality of adjustable machine learning models to generate a combined preliminary output score, and (b) comparing the combined preliminary output score to one or more decision threshold hyperparameters to generate an updated assessment result. In some embodiments, the combined preliminary output score is based on rule-based logic or a combinatorial technique for combining the scores. In some embodiments, the method further includes training at least one machine learning model using a first training dataset, a second training dataset, and a third training dataset, wherein the first training dataset includes one or more video recordings of the individual, the second training dataset includes one or more feedbacks provided by a healthcare provider to a questionnaire, and the third training dataset includes one or more feedbacks provided by a caregiver to a questionnaire. In some embodiments, at least one of the first training data set, the second training data set, or the third training data set includes at least 100 training samples.

[0014] In some embodiments, the behavioral disorder, neurological disorder, or developmental delay comprises pervasive developmental disorder (PDD), autism spectrum disorder (ASD), social-communicative disorder, restricted repetitive behaviors, interests, and activities (RRB), autism (e.g., classic autism), Asperger's syndrome (e.g., high-functioning autism), PDD not otherwise specified (PDD-NOS, atypical autism), attention deficit disorder (ADD), attention deficit hyperactivity disorder (ADHD), speech and language delay, obsessive-compulsive disorder (OCD), depression, schizophrenia, Alzheimer's disease, dementia, intellectual disability, or learning disability. In some embodiments, the behavioral disorder, neurological disorder, or developmental delay is autism spectrum disorder or autism.

[0015] In some embodiments, the method further includes generating an individualized therapeutic treatment plan for the individual based on the assessment results. In some embodiments, the individualized therapeutic treatment plan is generated using a therapeutic module comprising at least one statistical or machine learning model. In some embodiments, the method further includes receiving feedback data based on performance of the individualized therapeutic treatment plan and updating the individualized therapeutic treatment plan based on the feedback data. In some embodiments, the feedback data includes at least one of efficacy, compliance, or response to the individualized therapeutic treatment plan. In some embodiments, the individualized therapeutic treatment plan includes a drug therapy, a non-drug therapy, or both. In some embodiments, the non-drug therapy includes a digital therapeutic. In some embodiments, the drug therapy includes the use of one or more drugs to treat a behavioral disorder, a neurological disorder, or a developmental delay. The therapeutic module generates a recommendation for one or more drugs for the individual, as well as their respective dosages and timing, based at least in part on the individual's assessment results. For example, for an individual with a positive assessment for a behavioral disorder, the treatment module generates a medication recommendation, including one or more of stimulants (e.g., mixed amphetamine salts, methylphenidate), non-stimulant ADHD medications (e.g., atomoxetine, guanfacine ER), anticonvulsants (e.g., divalproex), and antipsychotics (e.g., aripiprazole, risperidone, ziprasidone). In some embodiments, non-pharmacological therapies include cognitive therapy, behavioral therapy, occupational therapy (e.g., sensory integration therapy), physical therapy, speech and language therapy, lifestyle modifications, physiotherapy, and pain management. For example, behavioral therapy may include discrete trial training (DTT), which uses step-by-step instructions to teach desired behaviors or responses. Lessons are broken down into their simplest parts, and desired responses and behaviors are rewarded. Undesirable responses and behaviors are ignored.As another example, or in conjunction with DTT, pivotal response training (PRT) may be used, which takes place in a natural setting rather than a clinical setting. The goal of PRT is to improve a few "pivotal skills" that help individuals learn many other skills. One example of a pivotal skill is initiating communication with others.

[0016] In some embodiments, the digital therapeutic includes single or multiple therapeutic activities or interventions that can be performed by the individual and / or their respective caregivers / healthcare providers. The digital therapeutic includes predetermined interactions with third-party devices, including sensors, computers, medical devices, and therapy delivery systems. The digital therapeutic can support FDA-approved medical billing, a set of diagnostic codes, or a single diagnostic code. In some other embodiments, the digital therapeutic includes instructions, feedback, activities, or interactions provided to the individual and / or their respective caregivers / healthcare providers. Examples include suggested behaviors, activities, games, or interactive sessions with third-party devices (e.g., Internet of Things (IoT)-enabled therapy devices). Further descriptions and examples of digital therapeutics are included in PCT application PCT / US2018 / 017354, which is incorporated by reference in its entirety for all purposes.

[0017] In some embodiments, the treatment module comprises a recommendation engine that provides an individualized treatment plan based at least in part on the assessment results. The recommendation engine receives the assessment result generated from the assessment module that the individual has a behavioral disorder and generates the individualized treatment plan. The individualized treatment plan includes a therapeutic treatment plan, which includes, but is not limited to, a recommended course of behavior, a treatment for the individual to engage in, and a daily activity for the individual to engage in.

[0018] In other embodiments, in addition to the evaluation results, the recommendation engine receives data related to the individual to generate a personalized treatment plan. Additional data about the individual includes, but is not limited to, demographic data, metabolic data, pharmacokinetic data, clearance data, and microbiome data. Demographic data includes the individual's age, sex, height, weight, diagnostic status for one or more disorders, and / or any other relevant demographic data. Metabolic data may be relevant to evaluating the effectiveness of using a therapy to treat a behavioral disorder. Metabolic data may include measurements of one or more metabolites (e.g., creatinine, xanthine, hypoxanthine, inosine) related to using a therapy to treat a behavioral disorder. The pharmacokinetics of an individual can be determined in response to administering a known amount of a therapeutic agent to the individual at a first time point and determining the amount of the therapeutic agent at a second time point. For example, a known amount of a therapeutic agent can be administered to the individual at the start of treatment while monitoring the individual's physiological parameters and metabolic data. If physiological parameters and metabolic data indicate severe side effects of the therapeutic agent, a lower dosage and / or less frequent timing may be recommended for later time points during treatment. The measured pharmacokinetic data may be selected from the group consisting of alpha elimination half-life and beta elimination half-life. The individual's clearance data may include the clearance rate of the therapeutic agent in the individual's body. The individual's microbiome data may include data selected from fecal samples, intestinal lavage fluid, or other samples of the individual's intestinal flora.

[0019] In some embodiments, the recommendation engine receives an assessment result generated from the assessment module that the individual has a behavioral disorder and generates a recommendation of one or more types of medication for the individual, hi other embodiments, the recommendation engine generates a recommendation for the timing and / or dosage of the medication.

[0020] In some embodiments, the method further comprises administering a treatment to the individual if the assessment result includes the presence of a behavioral disorder, neurological disorder, or developmental delay. The treatment may be selected from the group consisting of psychotherapy, behavioral therapy, pharmacological therapy, and surgical intervention.

[0021] In some aspects, disclosed herein is a computer-implemented method comprising: (a) receiving input data comprising a plurality of features associated with a behavioral disorder, a neurological disorder, or a developmental delay; (b) dividing the input data into a training dataset and a test dataset; (c) training a model using one of the training datasets; (d) determining at least one decision threshold for the model using a corresponding test dataset; (e) repeating steps (b)-(d) at least once using a cross-validation procedure to generate a plurality of decision thresholds; (f) determining one or more decision threshold hyperparameters using the plurality of decision thresholds; and (g) training a final model using the input data, wherein the final model comprises the one or more decision threshold hyperparameters.

[0022] In some aspects, a computer-implemented method includes: (a) receiving input data including a plurality of features associated with a behavioral disorder, a neurological disorder, or a developmental delay; (b) dividing the input data into a calibration data set and a validation data set; (c) dividing the calibration data set into a training data set and a test data set; (d) training a model using one of the training data sets; (e) determining at least one decision threshold for the model using a corresponding test data set; (f) repeating steps (c)-(e) at least once using a cross-validation procedure to generate a plurality of decision thresholds; and (g) dividing the calibration data set into a training data set and a test data set. Disclosed herein is a computer-implemented method comprising: (a) training a new model using one of the input data sets; (h) evaluating the new model using a corresponding validation dataset to generate at least one decision threshold based on the plurality of decision thresholds; (i) repeating steps (a)-(h) at least once using a cross-validation procedure to calculate one or more performance metrics; (j) determining one or more decision threshold hyperparameters using the plurality of decision thresholds; and (k) training a final model using the input data, wherein the final model includes the one or more decision threshold hyperparameters.

[0023] Disclosed herein in some aspects is a computer-implemented method comprising: (a) receiving individual input data associated with a behavioral disorder, neurological disorder, or developmental delay; (b) evaluating the input data using a machine learning model to generate an assessment result comprising a positive assessment, a negative assessment, and an indeterminate assessment regarding the presence or absence of the behavioral disorder, neurological disorder, or developmental delay; and (c) if the assessment result comprises the presence of the behavioral disorder, neurological disorder, or developmental delay, generating an individualized therapeutic treatment plan based at least in part on the assessment result.

[0024] In some aspects, a system is disclosed herein that includes a processor and a non-transitory computer-readable medium that includes executable instructions configured to cause the processor to perform a computer-implemented method according to any one of the disclosed embodiments.

[0025] In some aspects, disclosed herein is a non-transitory computer-readable medium comprising executable instructions configured to cause a processor to perform a computer-implemented method according to any one of the disclosed embodiments.

[0026]

[0013] Further aspects and advantages of the present disclosure will become readily apparent to those skilled in the art from the following detailed description, wherein only illustrative embodiments of the present disclosure are shown and described. As will be realized, the present disclosure is capable of other and different embodiments, and its several details are capable of modifications in various obvious respects, all without departing from the present disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature, and not as restrictive.

[0027] Incorporation by Reference All publications, patents, and patent applications mentioned herein are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. In the event that the publications and patents or patent applications incorporated by reference conflict with the disclosure contained herein, the present specification is intended to supersede and / or supersede any such conflicting material. [Brief explanation of the drawings]

[0028] The novel features of the present disclosure are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present disclosure will be obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the disclosure are utilized, and the accompanying drawings. [Figure 1]Figure 1 shows the workflow for assessing an individual for the presence of autism spectrum disorder (ASD) using caregiver input by completing a questionnaire, video input of the individual in their home environment, and healthcare provider input by completing a questionnaire. [Figure 2] Figure 2 shows a diagram of the lower and upper thresholds indicating the cutoffs for predicting the absence and presence of ASD, respectively. Any score between the lower and upper thresholds results in abstention. [Figure 3] Figure 3 shows a chart of the neurodevelopmental composition of children in the combined dataset. [Figure 4] Figure 4 shows a map showing that children from 43 states were represented in the combined data set. [Figure 5A] Figures 5A and 5B provide an illustration of the effect of the model's threshold hyperparameter on the performance metrics, with the algorithm ("Algorithm V2") having a 66.5% success rate after optimization. [Figure 5B] Figures 5A and 5B provide an illustration of the effect of the model's threshold hyperparameter on the performance metrics, with the algorithm ("Algorithm V2") having a 66.5% success rate after optimization. [Figure 6] Figure 6 shows the Canvas DX™ performance metrics using algorithm V2. [Figure 7A] Figure 7A shows the abstention rate and neurodevelopmental configuration using algorithm V1. [Figure 7B] Figure 7B shows the abstention rate and neurodevelopmental configuration using algorithm V2. [Figure 8] FIG. 8 illustrates a computer system that is programmed or otherwise configured to implement the methods provided herein. DETAILED DESCRIPTION OF THE INVENTION

[0029] Described herein are systems and computer-implemented methods used to evaluate individuals, including children, for behavioral disorders, developmental delays, and neurological disorders. In various embodiments, the computer-implemented methods described herein are configured to run on one or more computing devices, within one or more computing systems, or on one or more platforms.

[0030] Disclosed herein, in some aspects, is a computer-implemented method comprising: (a) receiving input data of an individual associated with a behavioral disorder, neurological disorder, or developmental delay; and (b) evaluating the input data using an assessment module comprising at least one machine learning model, thereby generating an assessment result, wherein the at least one machine learning model comprises one or more decision threshold hyperparameters that distinguish between a positive assessment, a negative assessment, and an indeterminate assessment regarding the presence or absence of the behavioral disorder, neurological disorder, or developmental delay.

[0031] Individual input data related to behavioral disorders, neurological disorders, or developmental delays may be collected directly from the individual or from their respective caregivers and / or healthcare providers. In some embodiments, the input data may comprise one or more video recordings of the individual performing tasks and / or interacting with others. In some other embodiments, a trained analyst may review the video recordings, observe the individual's behavior, and complete a questionnaire. The questionnaire may include approximately 5-50 questions, such as, for example, "How often does the child try to get people's attention?" and "How would you describe the child's activity level?" The individual's video recordings and / or responses to the questionnaire provided by the analyst may be used as input data for processing by the assessment module. In other embodiments, the caregiver and / or healthcare provider may also complete one or more questionnaires, which may be used as input data to the assessment module.

[0032] In some embodiments, the method may include training at least one machine learning model using a first training dataset (e.g., input data including feedback from a caregiver), a second training dataset (e.g., input data including ratings of video recordings), and a third training dataset (e.g., input data including feedback from a healthcare provider), where the first training dataset comprises one or more video recordings of the individual, the second training dataset comprises one or more feedbacks provided by a healthcare provider in a questionnaire, and the third training dataset comprises one or more feedbacks provided by a caregiver in a questionnaire. In some embodiments, at least one of the first training dataset, the second training dataset, or the third training dataset comprises at least 100 training samples.

[0033] The questionnaire may include a variety of questions aimed at assessing an individual's development, behavior, and / or other characteristics. The number and / or type of questions in the questionnaire may vary depending on the individual's age (e.g., 18-47 months, 48-72 months) and the role of the person completing the questionnaire (e.g., caregiver, healthcare provider, analyst reviewing the individual's video recording). In some embodiments, a questionnaire for healthcare providers of children aged 18-47 months may include approximately 5-30 questions, such as, for example, "Does the child have repetitive whole body movement?" and "How is the child's level of eye contact?" A questionnaire for healthcare providers of children aged 48-72 months may include approximately 5-30 questions, such as, for example, "Does the child spontaneously imitate parents or other family members?" In some other embodiments, a questionnaire targeted to caregivers of children aged 18-47 months may include approximately 5-30 questions, such as, for example, "Does your child typically share his / her excitement or enjoyment with you or others?" and "Consider when he / she is excited about a new toy or about going somewhere, or anything else that gets him / her excited."A questionnaire for caregivers of 48-72 month old children may include approximately 5-30 questions, such as, for example, "Does your child try to comfort others without being told to or without any sort of prompting?" In some other embodiments, a questionnaire for analysts reviewing video recordings of 18-47 month old children may include approximately 5-50 questions, such as, for example, "Does the child exhibit any self-harming behaviors?" A questionnaire for analysts reviewing video recordings of 48-72 month old children may include approximately 5-50 questions, such as, for example, "How would you describe the child's use of gestures?"

[0034] In some embodiments, the questionnaire may provide categorical response options. For example, for the question "How would you describe the child's vocalizations toward others?", the questionnaire may include options such as "Excellent: Numerous examples of social communication or vocalizations that clearly express interest or make needs known," which may correspond to a score of 0, "Good: Examples of social communication are sometimes observed and clearly express interest and make needs known," which may correspond to a score of 1, "Satisfactory: An example of social communication is observed that clearly expresses interest or makes a need known," which may correspond to a score of 2, "Poor: No examples of social communication are observed or the child does not vocalize," which may correspond to a score of 3. A variety of response options may be provided, including "The footage doesn't provide sufficient opportunity to assess reliably," which may correspond to a score of 9, and "The footage doesn't provide sufficient opportunity to assess reliably."These answers, or scores corresponding to the answers, may be used by the assessment module to calculate an assessment result for the individual. In some embodiments, the scores may be used to train a machine learning model. In some embodiments, the assessment module may rank or weight the questions and / or answers provided depending on the individual's age, the type of behavioral disorder, neurological disorder, or developmental delay diagnosed, and / or other factors relevant to the individual's diagnosis.

[0035] In some embodiments, the questionnaire may include a predetermined or set number of questions. For example, the assessment module may generate a questionnaire customized for the individual's illness. The assessment module may generate an individualized questionnaire with a predetermined or set number of questions, taking into account the individual's demographic information (e.g., age, gender) and current medical condition. In other embodiments, the assessment module may dynamically adjust the number and / or type of questions in the questionnaire based on answers provided by a caregiver, healthcare provider, and / or video analyst to previous questions. For example, if answers to questions related to developmental delay indicate a significant developmental delay in the individual, the assessment module may include additional questions directed to developmental delay assessment. In some embodiments, the assessment module may comprise one or more natural language processing models configured to generate questions directed to the individual's assessment and process freeform answers provided by caregivers and healthcare providers. The natural language processing models may provide greater flexibility and precision in the questionnaires and answers provided, thereby facilitating the individual's diagnosis and treatment.

[0036] In some embodiments, at least one machine learning model including one or more decision threshold hyperparameters provides a positive predictive value of at least about 80%, a negative predictive value of at least about 95%, a coverage or inclusion rate of at least about 70%, or any combination thereof, when evaluated using a nested cross-validation procedure. The use of one or more decision threshold hyperparameters can result in improved performance compared to scenarios in which multiple models are used for prediction, with each model processing only input data collected from individuals within a narrower age group. In some embodiments, the positive predictive value is at least about 70%, about 75%, about 80%, about 85%, about 90%, about 95%, or about 99%. In some embodiments, the negative predictive value is at least about 70%, about 75%, about 80%, about 85%, about 90%, about 95%, or about 99%. In some embodiments, the coverage or inclusion rate (e.g., the evaluation results after subtracting indeterminate results) is at least about 50%, about 55%, about 60%, about 65%, about 70%, about 75%, about 80%, about 85%, about 90%, about 95%, or about 99%.

[0037] Various procedures can be used to select the decision threshold. For example, the decision threshold can be determined based on a feasibility study, including the distribution of numerical outputs generated by the machine learning model, the distribution of responses to survey questions, and sample metadata and covariates. Clinical assumptions can also be incorporated into the likely composition of the target population when determining the decision threshold.

[0038] Disclosed herein are systems and methods that utilize a novel decision threshold selection process that generates stable performance estimates for future uses of the model, mitigates the risk of overfitting, and minimizes the influence of human bias in the loop. Use of the systems and methods described herein by healthcare providers can occur remotely or in person, without the need for specialist training.

[0039] In some embodiments, systems and methods utilize machine learning to assist healthcare providers in diagnosing or ruling out behavioral disorders, neurological disorders, or developmental delays (e.g., autism) in children aged 18 to 72 months. The systems and methods described herein can collect data from caregivers and healthcare providers and integrate it with structured observations of children based on video recorded in the home environment. Machine learning models can be trained to generate one of three outputs: positive, negative, or indeterminate for behavioral disorders, neurological disorders, or developmental delays (e.g., autism). The indeterminate category can be a safeguard that allows for abstention when insufficient data for a definitive classification is presented. Abstention in cases of high uncertainty is a beneficial machine learning-based medical device for safeguarding and minimizing the risk of erroneous clinical outcomes. Machine learning models can be configured for iterative learning and performance improvement through exposure to additional real-world data. Efficiently learning new tasks and incorporating new training data, sometimes referred to as "lifelong learning," presents several challenges. Training data selection should be appropriate to the clinical problem of interest and represent diverse phenotypes of the intended population to avoid amplifying gender, racial, socioeconomic, or other demographic biases, for example. Obtaining reliable labels for diseases without objective diagnostic tests can present other challenges, as can implementing procedures for robust model validation against relevant covariates.

[0040] Interest in the potential of both supervised and unsupervised machine learning approaches for predicting behavioral, neurological, or developmental delays and their respective treatment delivery tasks continues to grow. Artificial intelligence / machine learning software as medical devices (SaMDs) are often "locked" at the time of regulatory approval or require full clinical trials before any proposed changes. Locking a SaMD can pose a risk of preventing the SaMD from rapidly evolving and improving its performance over time with exposure to new data.

[0041] The Predetermined Change Control Plan (PCCP) was proposed by the Food and Drug Administration (FDA) in 2019 to regulate artificial intelligence / machine learning software as medical devices. The PCCP outlines a process by which approved algorithms can be intermittently "unlocked" under certain circumstances, with specific guardrails in place, to leverage new data to improve performance or address concerns. Both types of approaches are considered: anticipated software modifications (e.g., SaMD pre-specification) and approaches that ensure devices remain safe and effective after algorithm modifications (e.g., algorithm change protocols). Non-adaptive artificial intelligence / machine learning-based algorithms risk becoming outdated and degraded over time if the data used for training no longer reflects the real-world conditions in which they are applied. By providing a mechanism that allows manufacturers to intermittently "unlock" algorithms and expose models to additional data, the PCCP can help prevent algorithm drift.

[0042] In some embodiments, the systems and methods described herein use abstention thresholds that are modified under PCCP. The modifications increase the accuracy rate of machine learning models while maintaining confidence in their predictions, without changing their intended use. This novel real-world threshold update under PCCP fills a research gap in best practice regulation of SaMDs with the potential for iterative learning when exposed to new data.

[0043] In some embodiments, the systems and methods described herein include a nested cross-validation procedure in which the following steps are performed:

[0044] 1. Split the input data into a training data set and a validation data set.

[0045] 2. Split the training dataset into a training dataset and a test dataset.

[0046] 3. Train the model on one of the training datasets.

[0047] 4. Identify the optimal decision threshold using the corresponding test data set.

[0048] 5. Repeat steps 2-4 via cross-validation.

[0049] 6. Retrain the model on the training dataset - trained on the full training dataset - no longer split into subsets.

[0050] 7. Evaluate the model trained in step 6 on the validation dataset using the mean of the decision threshold specified in the inner cv loop.

[0051] 8. Repeat steps 1-7 via cross-validation.

[0052] In some embodiments, the training data may be randomly split into, for example, 90% as a training dataset, 10% as a test dataset, 80% as a training dataset, 20% as a test dataset, 70% as a training dataset, 50% as a test dataset, 60% as a training dataset, 40% as a test dataset, 50% as a training dataset and 50% as a test dataset, or 40% as a training dataset and 60% as a test dataset. The training data may have ground truth labels. A machine learning model may be trained using a portion of the training dataset and generate one or more decision thresholds using a corresponding test dataset. For example, prediction results generated by the machine learning model may be compared with the ground truth labels to determine whether the decision threshold is optimal. These steps may be repeated iteratively, during which the training data may be shuffled, subdivided into a training dataset and a test dataset, and used by the machine learning model to generate prediction results.

[0053] In some embodiments, this procedure may generate quantifiable performance estimates that ensure the machine learning model will perform as designed when used in real-life environments. A final training procedure may then optionally be performed to further fine-tune the model.

[0054] 9. Split the input data into training and testing datasets that may be different from the original training and testing datasets.

[0055] 10. Train the model on one of the training datasets.

[0056] 11. Identify the optimal decision threshold using the corresponding test data set.

[0057] 12. Repeat steps 9-11 via cross-validation.

[0058] 13. Determine the final decision threshold as the average of the decision thresholds identified in the step 12 loop.

[0059] 14. Train the final model on the full dataset.

[0060] In some embodiments, when a full nested cross-validation procedure is performed, it results in a trained machine learning model with one or more optimized decision threshold hyperparameters (e.g., thresholds indicating between positive and indeterminate assessments, and between indeterminate and negative assessments) for use in a computing device or program for assessing individuals for behavioral disorders, developmental delays, or neurological disorders. In addition, the nested cross-validation procedure provides a stable and reliable estimate of the model's real-world performance.

[0061] The evaluation system, device, and method described herein can classify at least one behavioral disorder, developmental delay, or neurological disorder with improved sensitivity and specificity.In addition, the evaluation system, device, and method can be continuously improved as more data from target populations become available for use in the model building process.

[0062] In some embodiments, individuals are assessed by a series of prompts in the form of questions that are displayed on the screen of a computing device.

[0063] In some embodiments, individuals are assessed by recorded video and / or audio data of the individual interacting with other people, performing tasks, and / or answering questions. In some embodiments, individuals are recorded answering questions asked by a human interviewer or caregiver. In some embodiments, a video analyst completes a questionnaire based on one or more recorded videos of the individual.

[0064] In some embodiments, the questionnaire is completed by the individual or their respective caregiver or healthcare provider on a mobile or stationary computing device. In some embodiments, the video and / or audio recording is made on a mobile device. In some embodiments, the mobile device is a smartphone, tablet, smartwatch, or any device with a mobile camera or recording capability. In some embodiments, the video and / or audio recording is taken using a stationary camera and / or microphone. For example, the individual can be questioned in a clinician's office and the responses recorded with a camera on a tripod with an attached microphone.

[0065] Some non-limiting examples of illnesses classified as behavioral disorders include attention deficit hyperactivity disorder (ADHD), postural disorder (ODD), autism spectrum disorder (ASD), anxiety disorder, depression, bipolar disorder, learning or performance disorder, or tract disorder. In some embodiments, attention deficit hyperactivity disorder (ADHD) includes predominantly inattentive ADHD, predominantly hyperactive-impulsive ADHD, or combined hyperactive-impulsive and inattentive ADHD. In some embodiments, autism spectrum disorder (ASD) includes autistic disorder (classic autism), Asperger's syndrome, pervasive developmental disorder (atypical autism), or childhood disintegrative disorder. In some embodiments, anxiety disorders include panic disorder, phobia, social anxiety disorder, obsessive-compulsive disorder, separation anxiety disorder, health anxiety disorder (hypochondria), or post-traumatic stress disorder. In some embodiments, depression includes major depression, persistent depressive disorder, bipolar disorder, seasonal affective disorder, psychotic depression, postpartum (postpartum) depression, premenstrual dysphoric disorder, "situational" depression, or typical depression. In some embodiments, bipolar disorder includes bipolar I disorder, bipolar II disorder, cyclozygosity disorder, or bipolar disorder caused by another medical or substance abuse disorder. In some embodiments, learning disorders include reflex disorder, dyscalculia, dysgraphia, sensory integration disorder, ataxia / dystonia, auditory processing disorder, or visual processing disorder. In some embodiments, behavioral disorders are disorders defined in any edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM).

[0066] Some non-limiting examples of diseases classified as developmental delay include autism spectrum disorder, mental retardation, cerebral palsy, Down syndrome, failure to thrive, muscular dystrophy, hydrocephalus, developmental coordination disorder, cystic fibrosis, fetal alcohol syndrome, homocystinuria, tuberous sclerosis, abetalipoproteinemia, phenylketonuria, speech delay, gross motor delay, micromotor delay, social delay, emotional delay, or cognitive delay. In some embodiments, mental retardation includes adrenoleukodystrophy, Ito syndrome, acrodysostosis, Huntington's disease, Alskog syndrome, Ikadi syndrome, or Tay-Sachs disease. In some embodiments, cerebral palsy includes spastic cerebral palsy, dyskinetic cerebral palsy, hypotonic cerebral palsy, ataxic cerebral palsy, or mixed cerebral palsy. In some embodiments, the autism spectrum disorder comprises autistic disorder (classic autism), Asperger's syndrome, pervasive developmental disorder (atypical autism), or childhood disintegrative disorder. In some embodiments, the Down's syndrome comprises trisomy 21, mosaicism, or translocation. In some embodiments, the muscular dystrophy comprises Duchenne muscular dystrophy, Becker muscular dystrophy, congenital muscular dystrophy, myotonic dystrophy, musculoscapular-humeral muscular dystrophy, oculopharyngeal muscular dystrophy, distal muscular dystrophy, or Emery-Dryffs muscular dystrophy.

[0067] Some non-limiting examples of diseases classified as neurological disorders include amyotrophic lateral sclerosis, arteriovenous malformation, cerebral aneurysm, brain tumor, dural arteriovenous fistula, epilepsy, headache, memory loss, multiple sclerosis, Parkinson's disease, peripheral neuropathy, postherpetic neuralgia, spinal cord tumor, stroke, Alzheimer's disease, cortical basal degeneration, Creutzfeldt-Jakob disease, frontotemporal dementia, mild cognitive impairment, progressive supranuclear palsy, or vascular dementia.

[0068] In some embodiments, the methods described herein are used in conjunction with known techniques for diagnosing behavioral disorders, developmental delays, or neurological disorders. In some embodiments, the methods disclosed herein are used to aid in the diagnosis of behavioral disorders, developmental delays, or neurological disorders. In some embodiments, the methods described herein can increase the accuracy of known diagnostic methods or reduce the time or recourse required for an accurate diagnosis.

[0069] The software described herein, in some embodiments, is located within a computing device used to receive input to the software, hi some embodiments, the software described herein is located on a server that is communicatively coupled to a computing device used by the individual being evaluated.

[0070] Machine Learning Software Module As described above, some embodiments of the methods, devices, systems, software, and platforms described herein utilize an assessment module to evaluate an individual's input data and generate an assessment for at least one behavioral disorder, developmental delay, or neurological disorder. In some embodiments, the assessment module comprises one or more trained models or machine learning algorithms that process the individual's input data and generate an output indicating a positive, negative, or indeterminate assessment of at least one behavioral disorder, developmental delay, or neurological disorder. In some other embodiments of the methods, devices, systems, software, and platforms described herein, a treatment module is utilized to generate a personalized therapeutic treatment plan for the individual based on the assessment results. The treatment module comprises at least one statistical or machine learning model.

[0071] A given trained machine learning model may have one or more decision threshold hyperparameters that indicate the boundaries between positive, negative, and indeterminate outputs. For example, the output generated by the model may be a score that is compared to the decision threshold hyperparameters to determine the final evaluation result. It should be understood that machine learning encompasses numerous architectures and configurations of data, and the teachings herein are not limited to any one single type of machine learning.

[0072] The machine learning model described herein can be trained using a dataset from an individual with a known positive or negative diagnosis of one or more of behavioral disorders, developmental delays, or neurological disorders. The individual may have previously been evaluated for one or more symptoms of behavioral disorders, developmental delays, or neurological disorders. In some embodiments, input data can be divided into a training dataset and a test dataset. The machine learning model can be trained using one of the training datasets, and the training dataset can be used to determine at least one decision threshold for the model using a corresponding test dataset. The division of input data, training of the machine learning model, and determination of at least one decision threshold can be repeated using a cross-validation procedure.

[0073] In some embodiments, the machine learning model includes one or more supervised, semi-supervised, self-supervised, or unsupervised machine learning techniques. For example, the machine learning model may be a training model trained by supervised learning (e.g., various parameters are determined as weights or scaling coefficients).

[0074] Training the machine learning model may optionally include selecting one or more untrained data models to train using the training dataset. The selected untrained data models may include any type of untrained machine learning model for supervised, semi-supervised, self-supervised, or unsupervised machine learning. The selected untrained data models are specified based on input (e.g., user input) that specifies relevant parameters for use as predictors or other variables for use as potential explanatory variables. For example, the selected untrained data models may be specified to generate an output (e.g., a prediction) based on the input. Conditions for training the machine learning model from the selected untrained data models may also be selected, such as limits on the complexity of the machine learning model or limits on the improvement of the machine learning model beyond a certain point. The machine learning model may be trained using the training dataset (e.g., via a computer system such as a server). In some cases, a first subset of the training dataset may be selected to train the machine learning model. The selected untrained data models may then be trained on the first subset of the training dataset using appropriate machine learning techniques based on the type of machine learning model selected and any conditions specified for training the machine learning model. In some cases, due to the processing power requirements of training a machine learning model, the selected untrained data model may be trained using additional computing resources (e.g., cloud computing resources). Such training may continue in some cases until at least one aspect of the machine learning model is validated and meets the selection criteria to be used as a predictive model.

[0075] In some cases, one or more aspects of the machine learning model may be validated using a second subset of the training dataset (e.g., different from the first subset of the training dataset) to determine the accuracy and robustness of the machine learning model. Such validation may include applying the machine learning model to the second subset of the training dataset to make predictions derived from the second subset of the training dataset. The machine learning model may then be evaluated to determine whether performance is sufficient based on the derived predictions. The sufficiency criteria applied to the machine learning model may vary depending on the size of the training dataset available for training, the performance of previous iterations of the trained model, or user-specified performance requirements. If the machine learning model does not achieve sufficient performance, additional training may be performed. The additional training may include improving the machine learning model or retraining on a different first subset of the training dataset, after which the new machine learning model may be validated and evaluated again. If the machine learning model achieves sufficient performance, in some cases, the machine learning model may be stored for current or future use. The machine learning model may be stored as a set of parameter values ​​or weights for analysis of additional inputs (e.g., additional relevant parameters for use as additional predictor variables, additional explanatory variables, additional user interaction data, etc.), which may also, in some instances, include analytical logic or an indication of model validity. In some cases, multiple machine learning models may be stored for generating predictions under different sets of input data conditions. In some embodiments, the machine learning models may be stored in a database (e.g., associated with a server).

[0076] The machine learning model may include one or more of regression analysis, regularization, classification, dimensionality reduction, ensemble learning, meta-learning, association rule learning, cluster analysis, anomaly detection, deep learning, or ultra-deep learning. Machine learning (ML) may include k-means, k-means clustering, k-nearest neighbors, learning vector quantization, linear regression, nonlinear regression, least squares regression, partial least squares regression, logistic regression, stepwise regression, multivariate adaptive regression splines, ridge regression, principal component regression, least absolute shrinkage and selection operation, least angle regression, canonical correlation analysis, factor analysis, independent component analysis, linear discriminant analysis, multidimensional scaling, non-negative matrix factorization, principal component analysis, principal coordinate analysis, projection pursuit, Sammon mapping, t-distributed stochastic neighbor embedding, Adabas boosting, boosting, gradient boosting, bootstrap aggregation, ensemble averaging, decision trees, conditional decision trees, boosted decision trees, gradient boosted decision trees, alternating decision trees, best-first decision trees, random The neural network may include, but is not limited to, a neural forest, a stacked generalization, a Bayesian network, a Bayesian belief network, a naive Bayes, a Gaussian naive Bayes, a polynomial naive Bayes, a hidden Markov model, a hierarchical hidden Markov model, a support vector machine, an encoder, a decoder, an autoencoder, a stacked autoencoder, a perceptron, a multilayer perceptron, an artificial neural network, a feedforward neural network, a convolutional neural network, a recurrent neural network, a long short-term memory, a deep belief network, a deep Boltzmann machine, a deep convolutional neural network, a deep recurrent neural network, or a generative adversarial network.

[0077] In some embodiments, the machine learning model may implement a decision tree. A decision tree may be a supervised machine learning algorithm that can be applied to both regression and classification problems. A decision tree may mimic the decision-making process of the human brain. For example, a decision tree may grow from a root (base condition) and split into multiple branches if conditions (internal nodes / features) are met. The end of a branch that does not split further may be the result (leaf). A decision tree may be generated using a training dataset according to the following operations: (1) starting from the root node (the entire dataset), the algorithm may split the dataset into two branches using a decision rule or branching criterion; (2) each of these two branches may generate a new child node; (3) for each new child node, the branching process may be repeated until the dataset is no longer split; (4) each branching criterion may be selected to maximize information gain (e.g., a quantification of how much the branching criterion reduces or a quantification of how mixed the labels are in the child node). The labels may be the data or classification predicted by the decision tree.

[0078] Random forest regression is an extension of decision tree models that tends to produce more stable predictions by expanding the use of training data partitions. While decision trees can make a single pass through the data, random forest regression can bootstrap 50% of the data (e.g., with replacement) and build many trees. Rather than using all explanatory variables as candidates for splits, a random subset of candidate variables can be used for splits, allowing for trees with completely different data and different variables (hence the term "random"). Predictions from the trees, collectively referred to as a "forest," can then be averaged together to produce a final prediction. Many trees (e.g., 100 trees) can be included in a random forest model, with a number of sampled terms per split (e.g., 3, 6, 10, etc.), a minimum number of splits per tree (e.g., 1, 2, 4, 10, etc.), and a minimum split size (e.g., 16, 32, 64, 128, 256, etc.). Random forests can be trained similarly to decision trees. Specifically, training a random forest may involve the following operations: (1) randomly selecting k features from the total number of features; (2) generating a decision tree from these k features using the same operations as generating a decision tree; and (3) repeating the first two operations until the target number of trees is created.

[0079] Systems and Devices The present disclosure provides a computerized device programmed to implement the methods of the present disclosure. Figure 8 shows a computerized device 801 suitable for use with the software described herein. The computerized device 801 can process various aspects of the information of the present disclosure, such as questions and answers, responses, statistical analysis, etc. The computerized device 801 can be a user's electronic device or a computerized device located remotely from the electronic device. The electronic device can be a mobile electronic device.

[0080] The computing device 801 includes a central processing unit (CPU, further referred to herein as "processor" and "computer processor") 805, which may be a single-core or multi-core processor, or multiple processors for parallel processing. The computing device 801 also includes memory or memory locations 810 (e.g., random access memory, read-only memory, flash memory), electronic storage 815 (e.g., hard disk), a communication interface 820 (e.g., network adapter) for communicating with one or more other devices, and peripheral devices 825, such as cache, other memory, data storage, and / or electronic display adapters. The memory 810, storage 815, interface 820, and peripheral devices 825 communicate with the CPU 805 via a communication bus (solid lines), such as a motherboard. The storage 815 may be a data storage device (or data repository) for storing data. The computing device 801 can be operatively coupled to a computer network ("network") 830 with the aid of the communication interface 820. Network 830 may be the Internet, an Internet and / or extranet, or an intranet and / or extranet in communication with the Internet. Network 830 is, in some cases, a telecommunications and / or data network. Network 830 may include one or more computer servers that may enable distributed computing, such as cloud computing. Network 830 may implement a peer-to-peer network, possibly with the aid of computing device 801, that may enable devices coupled to computing device 801 to act as clients or servers.

[0081] The CPU 805 may execute sequences of machine-readable instructions, which may be embodied in a program or software. The instructions may be stored in a memory location, such as the memory 810. The instructions may be directed to the CPU 805, which may then program or otherwise configure the CPU 805 to implement the methods of the present disclosure. Examples of operations performed by the CPU 805 may include fetch, decode, execute, and writeback.

[0082] The CPU 805 may be part of a circuit, such as an integrated circuit. One or more other components of the device 801 may be included in the circuit. In some cases, the circuit is an application specific integrated circuit (ASIC).

[0083] The storage device 815 may store files such as drivers, libraries, and saved programs. The storage device 815 may store user data, such as user preferences and user programs. The computing device 801 may optionally include one or more additional data storage devices external to the computing device 801, such as located on a remote server that communicates with the computing device 801 via an intranet or the Internet.

[0084] Computing device 801 can communicate with one or more remote computing devices via network 830. For example, computing device 801 can communicate with a remote computing device of a user (e.g., a parent). Examples of remote computing devices and mobile communication devices include a personal computer (e.g., a portable PC), a slate or tablet PC (e.g., an Apple® iPad®, a Samsung® Galaxy Tab), a telephone, a smartphone (e.g., an Apple® iPhone®, an Android-enabled device, a Blackberry®), or a personal digital assistant. A user can access computing device 801 via network 830.

[0085] The methods described herein can be implemented by machine (e.g., computer processor) executable code stored on an electronic storage location of computing device 801, such as on memory 810 or electronic storage 815. The machine-executable or machine-readable code can be provided in the form of software. During use, the code can be executed by processor 805. In some cases, the code can be retrieved from storage 815 and stored on memory 810 for easy access by processor 805. In some situations, electronic storage 815 can be eliminated, and machine-executable instructions are stored in memory 810.

[0086] The code may be pre-compiled and configured for use with a machine having a processor adapted to execute the code, or may be compiled during run-time. The code may be supplied in a programming language that may be selected to allow the code to be executed in a pre-compiled or as-compiled manner.

[0087] Aspects of the devices and methods provided herein, such as computing device 801, can be embodied in programming. Various aspects of the present technology may be considered a “product” or “articles of manufacture,” typically in the form of machine (or processor) executable code and / or associated data carried on or embodied in some type of machine-readable medium. The machine-executable code may be stored in electronic storage, such as memory (e.g., read-only memory, random-access memory, flash memory) or a hard disk. “Storage” type media may include any or all of the tangible memory of a computer, processor, or its associated modules, such as various semiconductor memories, tape drives, disk drives, etc., which can provide non-transitory storage for software programming at any time. All or portions of the software may, from time to time, be communicated via the Internet or various other telecommunications networks. Such communication may, for example, enable loading of the software from one computer or processor to another, for example, from an administrative server or host computer to the computer platform of an application server. Thus, another type of medium that may carry software elements includes light waves, radio waves, and electromagnetic waves, such as those used across physical interfaces between local devices, through wired and optical landline networks, and over various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., may also be considered software-bearing media. As used herein, unless limited to non-transitory, tangible "storage" media, terms such as computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

[0088] Thus, a machine-readable medium such as a computer-executable code may take many forms, including, but not limited to, a tangible storage medium, a carrier wave medium, or a physical transmission medium. Non-volatile storage media include optical or magnetic disks, such as any of the storage devices in any computer, such as may be used to implement the databases, etc., shown in the figures. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables, copper wire, and optical fiber, including the wires that comprise a bus within a computing device. Carrier-wave transmission media may take the form of electric or electromagnetic signals, or acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Thus, common forms of computer-readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, a DVD or DVD-ROM, any other optical medium, punch cards, paper tape, any other physical storage medium with a pattern of holes, RAM, ROM, PROM and EPROM, FLASH-EPROM, any other memory chip or cartridge, a carrier wave carrying data or instructions, a cable or link carrying such a carrier wave, or any other medium from which a computer can read programming code and / or data. Many of these forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0089] The computing device 801 may include or communicate with an electronic display 835 that provides a user interface (UI) for providing, for example, questions and answers, analysis results, and recommendations. Examples of UIs include, but are not limited to, graphical user interfaces (GUIs) and web-based user interfaces.

[0090] The methods and devices of the present disclosure may be implemented by one or more algorithms and using instructions provided by one or more processors disclosed herein. The algorithms may be implemented by software when executed by the central processing unit 805. The algorithms may be, for example, random forests, graphical models, support vector machines, etc.

[0091] The above steps illustrate a method of the device according to the example embodiment, but one skilled in the art will recognize many variations based on the teachings provided herein. Steps may be completed in a different order. Steps may be added or removed. Some of the steps may include sub-steps. Many of the steps may be repeated as often as is beneficial to the platform.

[0092] Each of the examples described herein may be combined with one or more other examples. Additionally, one or more elements of one or more examples may be combined with other examples.

[0093] Digital Processing Device In some embodiments, the software described herein is located on a digital processing device and / or configured to cause a processor of the digital processing device to perform specific tasks. In further embodiments, the digital processing device includes one or more hardware central processing units (CPUs) or general-purpose graphics processing units (GPGPUs) that perform the functions of the device. In still further embodiments, the digital processing device further comprises an operating system configured to execute executable instructions. In some embodiments, the digital processing device is optionally connected to a computer network. In further embodiments, the digital processing device is optionally connected to the Internet to access the World Wide Web. In still further embodiments, the digital processing device is optionally connected to a cloud computing infrastructure. In other embodiments, the digital processing device is optionally connected to an intranet. In other embodiments, the digital processing device is optionally connected to a data storage device.

[0094] In accordance with the description herein, suitable digital processing devices include, by way of non-limiting example, server computers, desktop computers, laptop computers, notebook computers, subnotebook computers, netbook computers, netpad computers, set-top computers, media streaming devices, handheld computers, Internet appliances, mobile smartphones, tablet computers, personal digital assistants, video game consoles, and vehicles. Those skilled in the art will recognize that many smartphones are suitable for use with the systems described herein. Those skilled in the art will also recognize that selected televisions, video players, and digital music players, with optional computer network connectivity, are suitable for use with the systems described herein. Suitable tablet computers include those having booklet, slate, and convertible configurations known to those skilled in the art.

[0095] In some embodiments, the digital processing device includes an operating system configured to execute executable instructions. An operating system is software, including, for example, programs and data, that manages the device's hardware and provides services for the execution of applications. Those skilled in the art will recognize that suitable server operating systems include, by way of non-limiting example, FreeBSD, OpenBSD, NetBSD®, Linux®, Apple® Mac OS X® Server®, Oracle® Solaris®, Windows Server®, and Novell® NetWare®. Those skilled in the art will recognize that suitable personal computer operating systems include, by way of non-limiting example, UNIX-like operating systems such as Microsoft® Windows®, Apple® Mac OS X®, UNIX®, and GNU / Linux®. In some embodiments, the operating system is provided by cloud computing. Those skilled in the art will also recognize that suitable mobile smartphone operating systems include, by way of non-limiting example, Nokia® Symbian® OS, Apple® iOS®, Research In Motion® BlackBerry® OS®, Google® Android®, Microsoft® Windows Phone® OS, Microsoft® Windows Mobile® OS, Linux®, and Palm® WebOS®.Those skilled in the art will also recognize that suitable media streaming device operating systems include, by way of non-limiting example, Apple TV®, Roku®, Boxee®, Google TV®, Google Chromecast®, Amazon Fire®, and Samsung® HomeSync®. Those skilled in the art will also recognize that suitable video game console operating systems include, by way of non-limiting example, Sony® PS3®, Sony® PS4®, Microsoft® Xbox 360®, Microsoft Xbox One, Nintendo® Wii®, Nintendo® Wii U®, and Ouya®.

[0096] In some embodiments, the device comprises a storage and / or memory device. A storage and / or memory device is one or more physical devices used to temporarily or permanently store data or programs. In some embodiments, the device is volatile memory and requires power to maintain stored information. In some embodiments, the device is non-volatile memory and retains stored information when power is not applied to the digital processing device. In further embodiments, the non-volatile memory comprises flash memory. In some embodiments, the non-volatile memory comprises dynamic random access memory (DRAM). In some embodiments, the non-volatile memory comprises ferroelectric random access memory (FRAM®). In some embodiments, the non-volatile memory comprises phase change random access memory (PRAM). In other embodiments, the device is a storage device, including, by way of non-limiting example, a CD-ROM, a DVD, a flash memory device, a magnetic disk drive, a magnetic tape drive, an optical disk drive, and cloud computing-based storage. In further embodiments, the storage and / or memory device is a combination of devices such as those disclosed herein.

[0097] In some embodiments, the digital processing device includes a display for transmitting visual information to a user. In some embodiments, the display is a liquid crystal display (LCD). In further embodiments, the display is a thin film transistor liquid crystal display (TFT-LCD). In some embodiments, the display is an organic light emitting diode (OLED) display. In various further embodiments, the OLED display is a passive matrix OLED (PMOLED) or an active matrix OLED (AMOLED) display. In some embodiments, the display is a plasma display. In other embodiments, the display is a video projector. In still other embodiments, the display is a head-mounted display in communication with the digital processing device, such as a VR headset. In further embodiments, suitable VR headsets include, by way of non-limiting example, HTC Vive, Oculus Rift, Samsung Gear VR, Microsoft HoloLens, Razer OSVR, FOVE VR, Zeiss VR One, Avegant Glyph, Freefly VR headsets, and the like. In still further embodiments, the display is a combination of devices, such as those disclosed herein.

[0098] In some embodiments, the digital processing device includes an input device for receiving information from a user. In some embodiments, the input device is a keyboard. In some embodiments, the input device is a pointing device, including, by way of non-limiting example, a mouse, trackball, trackpad joystick, game controller, or stylus. In some embodiments, the input device is a touchscreen or multi-touchscreen. In other embodiments, the input device is a microphone for capturing voice or other sound input. In other embodiments, the input device is a video camera or other sensor for capturing movement or visual input. In further embodiments, the input device is a Kinect, Leap Motion, or the like. In still further embodiments, the input device is a combination of devices, such as those disclosed herein.

[0099] Non-transitory computer-readable storage medium In some embodiments, a computing device used with the software described herein further includes one or more non-transitory computer-readable storage media encoded with a program including instructions executable by an operating system of the networked digital processing device. In further embodiments, the computer-readable storage medium is a tangible component of the digital processing device. In still further embodiments, the computer-readable storage medium is optionally removable from the digital processing device. In some embodiments, computer-readable storage media include, by way of non-limiting example, CD-ROMs, DVDs, flash memory devices, solid-state memory, magnetic disk drives, magnetic tape drives, optical disk drives, cloud computing systems and services, and the like. In some cases, the programs and instructions are encoded on the medium permanently, substantially permanently, semi-permanently, or non-permanently.

[0100] computer program In some embodiments, software as described herein comprises sequences of instructions written to perform specified tasks that are executable by a processor, such as a CPU, of a digital processing device. The computer-readable instructions may be implemented as program modules, such as functions, objects, application programming interfaces (APIs), data structures, etc., that perform particular tasks or implement particular abstract data types. In light of the disclosure provided herein, those skilled in the art will recognize that computer programs may be written in a variety of languages ​​and versions.

[0101] The functionality of the computer-readable instructions can be combined or distributed as desired in various environments. In some embodiments, a computer program includes one sequence of instructions. In some embodiments, a computer program includes multiple sequences of instructions. In some embodiments, a computer program is provided from one location. In other embodiments, a computer program is provided from multiple locations. In various embodiments, a computer program includes one or more software modules. In various embodiments, a computer program includes part or all of one or more web applications, one or more mobile applications, one or more standalone applications, one or more web browser plug-ins, extensions, add-ins, or add-ons, or combinations thereof.

[0102] Software Module In some embodiments, the software described herein includes modules, or uses thereof. In light of the disclosure provided herein, software modules are created by techniques known to those of skill in the art using machines, software, and languages ​​known to those of skill in the art. The software modules disclosed herein are implemented in numerous ways. In various embodiments, a software module comprises a file, a selection of code, a programming object, a programming structure, or a combination thereof. In further various embodiments, a software module comprises multiple files, multiple selections of code, multiple programming objects, multiple programming structures, or a combination thereof. In various embodiments, one or more software modules comprise, by way of non-limiting examples, a web application, a mobile application, and a standalone application. In some embodiments, a software module is within one computer program or application. In other embodiments, a software module is within multiple computer programs or applications. In some embodiments, a software module is hosted on one machine. In other embodiments, a software module is hosted on multiple machines. In further embodiments, a software module is hosted on a cloud computing platform. In some embodiments, a software module is hosted on one or more machines in one location. In other embodiments, a software module is hosted on one or more machines in one or more locations.

[0103] Web Applications In some embodiments, the software described herein includes a web application. In light of the disclosure provided herein, those skilled in the art will recognize that web applications, in various embodiments, utilize one or more software frameworks and one or more database systems. In some embodiments, the web application is created on a software framework such as Microsoft® .NET or Ruby on Rails (RoR). In some embodiments, the web application utilizes one or more database systems, including, by way of non-limiting example, relational, non-relational, object-oriented, associative, and XML database systems. In further embodiments, suitable relational database systems include, by way of non-limiting example, Microsoft® SQL Server, mySQL™, and Oracle®. Those skilled in the art will also recognize that web applications, in various embodiments, are written in one or more versions of one or more languages. Web applications may be written in one or more markup languages, presentation definition languages, client-side scripting languages, server-side coding languages, database query languages, or combinations thereof. In some embodiments, the web application is written to some extent in a markup language such as Hypertext Markup Language (HTML), Extensible Hypertext Markup Language (XHTML), or Extensible Markup Language (XML). In some embodiments, the web application is written to some extent in a presentation definition language such as Cascading Style Sheets (CSS). In some embodiments, the web application is written to some extent in a client-side scripting language such as Asynchronous Javascript and XML (AJAX), Flash® Actionscript, Javascript, or Silverlight®.In some embodiments, the web application is written to some extent in a server-side coding language such as Active Server Pages (ASP), ColdFusion®, Perl, Java™, JavaServer Pages (JSP), Hypertext Preprocessor (PHP), Python™, Ruby, Tcl, Smalltalk, WebDNA®, or Groovy. In some embodiments, the web application is written to some extent in a database query language such as Structured Query Language (SQL). In some embodiments, the web application integrates an enterprise server product such as IBM® Lotus Domino®. In some embodiments, the web application includes a media player element. In various further embodiments, the media player element utilizes one or more of many suitable multimedia technologies, including, by way of non-limiting example, Adobe® Flash®, HTML5, Apple® QuickTime®, Microsoft® Silverlight®, Java™, and Unity®.

[0104] Mobile Applications In some embodiments, the software described herein comprises a mobile application provided to the mobile digital processing device. In some embodiments, the mobile application is provided to the mobile digital processing device at the time of manufacture. In other embodiments, the mobile application is provided to the mobile digital processing device via a computer network described herein.

[0105] Given the disclosure provided herein, mobile applications are created using hardware, languages, and development environments known to those skilled in the art and with techniques known to those skilled in the art. Those skilled in the art will recognize that mobile applications can be written in a number of languages. Suitable programming languages ​​include, by way of non-limiting example, C, C++, C#, Objective-C, Java™, Javascript™, Pascal, Object Pascal, Python™, Ruby, VB.NET, WML, and XHTML / HTML with or without CSS, or combinations thereof.

[0106] Suitable mobile application development environments are available from several sources. Commercially available development environments include, but are not limited to, Airplay SDK, alcheMo, Appcelerator®, Celsius, Bedrock, Flash Lite, .NET Compact Framework, Rhomobile, and WorkLight Mobile Platform. Other development environments are available free of charge, but are not limited to, Lazarus, MobiFlex, MoSync, and Phonegap. Mobile device manufacturers also distribute software developer kits, but are not limited to, the iPhone® and iPad® (iOS) SDK, Android™ SDK, BlackBerry® SDK, BREW SDK, Palm® OS SDK, Symbian SDK, webOS SDK, and Windows® Mobile SDK.

[0107] Those skilled in the art will recognize that several commercial forums are available for the distribution of mobile applications, including, by way of non-limiting example, the Apple® App Store, Google® Play, Chrome WebStore, BlackBerry® App World, App Store for Palm devices, App Catalog for webOS, Windows® Marketplace for Mobile, Ovi Store for Nokia® devices, Samsung® Apps, and Nintendo® DSi Shop.

[0108] Standalone Applications In some embodiments, the software described herein comprises a standalone application, which is a program that runs as an independent computer process rather than an add-on to an existing process, e.g., not a plug-in. Those skilled in the art will recognize that standalone applications are often compiled. A compiler is a computer program that converts source code written in a programming language into binary object code, such as assembly language or machine language. Suitable compiled programming languages ​​include, by way of non-limiting example, C, C++, Objective-C, COBOL, Delphi, Eiffel, Java™, Lisp, Python™, Visual Basic, and VB.NET, or combinations thereof. Compilation is often performed, at least in part, to create an executable program. In some embodiments, a computer program comprises one or more executable compiled applications.

[0109] Web browser plugin In some embodiments, the software described herein includes or works in conjunction with a web browser plug-in (e.g., extension, etc.). In computing, a plug-in is one or more software components that add specific functionality to a larger software application. Software application manufacturers support plug-ins to allow third-party developers to create the ability to extend the application, help easily add new features, and reduce the size of the application. When supported, plug-ins allow customization of the software application's functionality. For example, plug-ins are commonly used in web browsers to play video, generate interactivity, scan for viruses, and display specific file types. Those skilled in the art are familiar with several web browser plug-ins, including Adobe® Flash® Player, Microsoft® Silverlight®, and Apple® QuickTime®.

[0110] Given the disclosure provided herein, one of ordinary skill in the art will recognize that several plug-in frameworks are available that allow for the development of plug-ins in a variety of programming languages, including, by way of non-limiting example, C++, Delphi, Java™, PHP, Python™, and VB.NET, or combinations thereof.

[0111] A web browser (also referred to as an Internet browser) is a software application designed for use with networked digital processing devices to search, present, and traverse information resources on the World Wide Web. Suitable web browsers include, by way of non-limiting example, Microsoft® Internet Explorer®, Mozilla® Firefox®, Google® Chrome, Apple® Safari®, Opera Software® Opera®, and KDE Konqueror. In some embodiments, the web browser is a mobile web browser. Mobile web browsers (also referred to as microbrowsers, minibrowsers, and wireless browsers) are designed for use on mobile digital processing devices, including, by way of non-limiting example, handheld computers, tablet computers, netbook computers, subnotebook computers, smartphones, music players, personal digital assistants (PDAs), and handheld video game systems. Suitable mobile web browsers include, by way of non-exhaustive example, Google® Android® browser, RIM BlackBerry® Browser, Apple® Safari®, Palm® Blazer, Palm® WebOS® Browser, Mozilla® Firefox® for mobile, Microsoft® Internet Explorer® Mobile, Amazon® Kindle® Basic Web, Nokia® Browser, Opera Software® Opera® Mobile, and Sony® PSP™ browser.

[0112] Database In some embodiments, the software described herein operates in conjunction with, or uses of, one or more databases. In various embodiments, suitable databases include, by way of non-limiting example, relational databases, non-relational databases, object-oriented databases, object databases, entity-relationship model databases, associative databases, and XML databases. Further non-limiting examples include SQL, PostgreSQL, MySQL®, Oracle, DB2, and Sybase. In some embodiments, the database is internet-based. In further embodiments, the database is web-based. In yet further embodiments, the database is cloud-computing-based. In other embodiments, the database is based on one or more local computer storage devices. [Example]

[0113] Example 1 - Artificial intelligence based medical device with improved detection of autism in children.

[0114] Despite evidence that early intervention for autism spectrum disorder (ASD) improves long-term outcomes and quality of life, significant diagnostic delays persist. User-friendly, data-driven ASD diagnostic tools suitable for deployment in time-pressured care settings can support streamlined diagnoses and help meet the rapidly increasing demand for assessments that exceed the capabilities of specialists.

[0115] In response to this need, Example 1 describes an artificial intelligence-based medical device with improved detection of autism in children. Canvas Dx™ is an artificial intelligence-based medical device that assists healthcare providers in diagnosing or ruling out autism in young children with concerns about developmental delay. Canvas Dx™ can be used to diagnose autism and received FDA marketing approval in 2021.

[0116] Canvas Dx™ used a gradient-boosted decision tree machine learning model, which received behavioral features selected through machine learning techniques as the best predictors of autism in 18-72 month-old children across a range of phenotypic presentations to avoid amplification of gender, racial, socioeconomic, or other demographic biases, resulting in a prediction score.

[0117] Canvas Dx™ integrates data from multiple sources, including input from caregivers and healthcare providers, as well as structured observations of the child. Figure 1 illustrates the workflow for assessing an individual for the presence of ASD using caregiver input via completing a questionnaire, video input of the individual in their home environment, and healthcare provider input via completing a questionnaire. As shown, caregivers can use their smartphones to complete a brief questionnaire (e.g., 18 or 21 items) about the child's behavior. Caregivers can also record and upload one or more videos and / or audio recordings of the child in their home environment. The videos can be securely transmitted to a portal, where trained analysts identify key characteristics about the child in a 28- or 33-item questionnaire. A healthcare provider, such as the child's physician, can independently complete a questionnaire (e.g., 13 or 15 items) about the child's behavior and / or health status. One or more of these inputs (e.g., caregiver input, video input, and healthcare provider input) can be used by Canvas Dx™ to generate a predicted assessment. For example, these inputs can be combined into vectors for machine learning analysis and classification. The classification may include a positive assessment of ASD, a negative assessment of ASD, or an indeterminate assessment. Canvas Dx™ uses a gradient-boosted decision tree machine learning model that receives behavioral features selected through machine learning techniques as best predictive of ASD in children aged 18-72 months across a range of phenotypic presentations and generates a prediction score.

[0118] Figure 2 shows a diagram of the lower and upper thresholds, indicating the cutoffs for predicting the absence and presence of ASD, respectively. Any score between the lower and upper thresholds results in an abstention. If the prediction score is above the predetermined threshold, a positive ASD output is generated. If the prediction score is below the predetermined threshold, a negative ASD output is generated. The thresholds were selected through an optimization procedure on the training data to allow the device to abstain if insufficient information is presented, a key safety feature that helps ensure the device's efficacy. The decision threshold optimization process reduced the abstention rate for Canvas Dx™ without changing its intended use. Compared to the version of the algorithm granted marketing clearance by the FDA (Algorithm V1), the optimized algorithm (Algorithm V2) maintained comparable predictive value.

[0119] Decision threshold optimization was performed under a prescribed change control plan (PCCP), which was part of the device's de novo classification request granted by the FDA. This includes both the anticipated modifications, software as medical device (SaMD) predesignation, based on a retraining and model update strategy, and the associated methodology and algorithm change protocol used to implement these changes in a controlled manner that manages risk to patients in accordance with FDA Good Machine Learning Practices. The PCCP covers model adjustment and optimization given new data from the intended use population of children aged 18 to 72 months who are symptomatic or suspected of being at risk for developmental delay, including autism, based on observations from their respective caregivers and / or healthcare providers. The optimization approach involved threshold optimization, specifically, the selection of an optimal threshold, to reduce the probability of device abstention while maintaining comparable predictive value.

[0120] Two data sets were used in the decision threshold optimization process.

[0121] Dataset 1: n = 425, ClinicalTrials.gov Identifier NCT04151290 (mean age 3.33 years, 36.4% girls). Reference standard diagnoses were based on Diagnostic and Statistical Manual of Mental Disorders (DSM-5) criteria, supported by independent review by expert clinicians. Expert clinicians were board-certified child psychiatrists, child neurologists, developmental-behavioral pediatricians, or psychologists with at least 5 years of experience diagnosing autism. Expert clinicians recorded both autism diagnoses and other non-autistic developmental delays. In the absence of an expert documenting a participant's neurodevelopmental delays, neurotypicality was assumed. All study participants and assessors were blinded to the device results and the diagnostic decisions of other expert clinicians. Participants were identified by expert clinicians as having 29.0% autism and 71.0% without autism (61.4% of participants had one or more non-autistic developmental delays, and 9.6% of participants had neurotypical development). Participants in this dataset reflected US demographics across race, ethnicity, and socioeconomic status. Analysis of Dataset 1 is described in "Evaluation of an Artificial Intelligence-based Medical Device for Diagnosis of Autism Spectrum Disorder," npj Digital Medicine (2022) 5:57, which is incorporated herein by reference in its entirety.

[0122] Dataset 2: n = 297 (ClinicalTrials.gov Identifier NCT03871179) (mean age 3.97 years, 42.9% girls). This is a prospectively collected dataset of children with developmental delay concerns. Reference standard ASD diagnoses were based on DSM-5 criteria by expert clinicians. Expert clinicians recorded both autism diagnoses and other non-autistic developmental delays. Neurotypicality was assumed in participants without expert documentation of neurodevelopmental delays. All participants and assessors were blinded to device results. Participants were identified by expert clinicians as having autism (27.6%) and not having autism (72.4%) (33.3% of subjects had one or more non-autistic developmental delays, and 39.1% of subjects were neurotypical).

[0123] For the purposes of the threshold optimization procedure, Datasets 1 and 2 were combined. The combined dataset included 722 children who met the Canvas Dx™ labeling requirements. The children were 18–72 months of age and had concerns about developmental delay. The children in the combined dataset were from 43 U.S. states, had a mean age of 3.6 years, and 39% were girls. Figure 3 shows a chart of the neurodevelopmental composition of the children in the combined dataset. The neurodevelopmental distribution of the children in the combined dataset represented the intended use population, with 28% having autism, 50% having one or more non-autistic developmental delays, and 22% having typical development, as defined above through expert clinician evaluation. Figure 4 shows a map showing the representation of children from the 43 states in the combined dataset.

[0124] The algorithm's decision threshold was optimized using an iterative training / test validation procedure. From a combined dataset of 722 samples, 70% of the samples were randomly selected for threshold optimization (the "training set") and 30% were selected for evaluation (the "test set") in 1000 iterations. Each iteration was constructed to ensure that data used in training were not used to test the model. 504 samples in the training set (mean age = 3.6 + / - 1.2 years, mean 39.1% girls) were used for threshold selection, and 218 (mean age = 3.7 + / - 1.2 years, mean 39.0% girls) were used for testing. The composition of the training and test sets remained representative of the intended use population.

[0125] Threshold optimization was performed on the training set at each iteration by finding the optimal threshold pair that maximized the positive predictive value (PPV) and negative predictive value (NPV) for evaluation using a holdout test set while reducing the device's rejection rate. Optimization focused on reducing the rejection rate while ensuring that predictions generated using algorithm V2 remained comparable to those generated using algorithm V1. Out-of-sample performance was estimated by evaluating the selected threshold pair on the test set and comparing the performance metrics of the new pair with the corresponding V1 metrics on the same test set. Performance was determined by examining the difference in NPV, PPV, and confirmed rejection rate between the two algorithms (V2-V1) and the 95% confidence interval calculated from the quantiles of the iterations.

[0126] Out-of-sample performance was estimated by evaluating the selected threshold pair on the test set and comparing the performance metric of the new pair with the corresponding VI metric on the same test set.

[0127] Figures 5A and 5B provide illustrations of the impact of the model's threshold hyperparameter on performance metrics. After optimization, the algorithm ("Algorithm V2") had a 66.5% accuracy rate. The NPV and PPV were comparable to those of Algorithm V1.

[0128] Table 1 lists the performance metrics for devices using algorithms V1 and V2. The confirmation rate was the percentage of children receiving a positive or negative output for autism.

[0129] [Table 1]

[0130] Figure 6 shows Canvas DX™ performance metrics (using algorithm V2). Both the positive predictive value (PPV) and negative predictive value (NPV) exceeded the minimum requirements provided by the FDA.

[0131] Table 2 shows the performance metrics of devices using algorithms V1 and V2 measured on the combined dataset.

[0132] [Table 2]

[0133] To compare Algorithm V2 with Algorithm V1, the differences in NPV, PPV, and confirmed output rates between the two algorithms were examined, and 95% confidence intervals were calculated from the quantiles of the iterations. Confidence intervals overlapping zero indicated no significant difference in performance between Algorithms V1 and V2. Confidence intervals entirely above zero indicated superior V2 performance. Confidence intervals entirely below zero indicated inferior V2 performance. As shown in Tables 1 and 2, devices using Algorithm V2 produced significantly higher confirmed rates (autism positive or negative) than devices using Algorithm V1. Devices using Algorithm V2 maintained predictive values ​​comparable to those achieved by devices using Algorithm V1. The confirmed output rate for devices using Algorithm V1 was 21% higher than the confirmed rate for devices using Algorithm V2. On average across iterations, 50 subjects were converted from indeterminate to confirmed when using devices using Algorithm V2. Of these, on average, 4 / 50 (8%) were incorrectly predicted as positive (mean = 1) or negative (mean = 3).

[0134] Figures 7A and 7B show the relative abstention rates and neurodevelopmental profiles of children in the abstention group across both algorithm versions based on expert assessment. Figure 7A shows the abstention rate and neurodevelopmental profile using Algorithm V1. Figure 7B shows the abstention rate and neurodevelopmental profile using Algorithm V2. When using the optimized algorithm (i.e., Algorithm V2), children who received an indeterminate output were observed to be more likely to have one or more neurodevelopmental abnormalities (including autism) as determined by expert assessment compared to those using a device with Algorithm V1. Specifically, children who received an indeterminate output from a device with Algorithm V2 were 3.4 times less likely to be neurodeveloped than children who received an indeterminate output from a device with Algorithm V1 (V1: 8:1 odds of neurotypical vs. V2: 28:1 odds of neurotypical).

[0135] In a prospective, multisite, double-blind, clinical validation study, Canvas Dx™ output was also compared with consensus expert diagnosis. The device output PPV for all study completers was 80.8% (70.3%-88.8%), and the NPV was 98.3% (90.6%-100%). Approximately one-third (31.8%) of participants received a definite output (ASD positive or negative). The remaining 68% of participants received an indeterminate (expected withdrawal) output.

[0136] The algorithm threshold modification procedure described herein resulted in higher definitive rates without a reduction in negative or positive predictive value. The majority (92%) of decision outputs received under Algorithm V2 for subjects classified as indeterminate under Algorithm V1 were correctly aligned with expert diagnoses. This finding highlights the potential of optimized devices to assist healthcare providers in clinical practice in accurately detecting or ruling out autism in younger children. Furthermore, if a previously indeterminate patient is reclassified with a future algorithm update to a definitive output, the definitive output is likely to be correct. The modifications do not affect how the device is actually used, do not affect the instructions for use, or introduce any new risks or significantly modify existing risks.

[0137] The flexible machine learning design approach employed in this threshold optimization procedure allowed us to utilize training data that reflects U.S. demographics across race, ethnicity, and socioeconomic status. All children in the combined dataset fell within the intended use population (developmental delay concern, 18-72 months of age). Good machine learning practices were followed throughout the design and execution of the study, including separation between the training and test sets in each iteration and the use of best practice data management and handling practices, in accordance with FDA recommendations. Additional real-world data will help clarify the extent to which the prevalence of autism and other assumptions incorporated into our model reflect real-world use populations. Planned and ongoing real-world evidence studies may also focus on how the device can be integrated into primary care practice environments and how its use may impact time to diagnosis and treatment initiation.

[0138] The above is an example of the practical use of a prescribed change control plan under the FDA's proposed regulatory framework. Regulatory mechanisms such as PCCPs may play a critical role in product development as artificial intelligence or machine learning-based technologies grow at a rapid pace. Non-adaptive artificial intelligence or machine learning-based algorithms may be at risk of becoming outdated and deteriorating over time if the data used to train them no longer reflects the real-world conditions in which they are applied.

[0139] The systems and methods described herein allow manufacturers to intermittently "unlock" algorithms and expose models to new data, thereby preventing algorithm drift when deployed clinically. The modified decision threshold significantly reduced device rejection rates while maintaining predictive value comparable to that achieved by devices using Algorithm V1. Devices with optimized decision thresholds may assist providers in efficiently evaluating more children with concerns about developmental delays, including diagnosing or ruling out ASD. The application of a data-driven approach addressed some of the inherent limitations of current ASD diagnostic approaches. Enhanced diagnostic capabilities could allow more children to access ASD services during the critical early years of high neuroplasticity, when intervention has the greatest impact. Further details are described in "Optimizing a de novo artificial intelligence-based medical device under a predetermined change control plan: Improved ability to detect or rule out pediatric autism," Intelligence-Based Medicine 8 (2023) 100102, which is incorporated herein by reference in its entirety.

[0140] In conclusion, implementation of the algorithm optimization process significantly reduced device rejection rates while maintaining predictive values ​​comparable to those observed with Algorithm V1. Devices with optimized decision thresholds have the potential to assist healthcare providers in diagnosing or ruling out autism in a large proportion of young children for whom developmental delay is a concern.

Claims

1. 1. A computer-implemented method comprising: (a) receiving individual input data associated with a behavioral disorder, neurological disorder, or developmental delay; (b) evaluating the input data using an evaluation module comprising at least one machine learning model to thereby generate an evaluation result, wherein the at least one machine learning model comprises one or more decision threshold hyperparameters that distinguish between a positive evaluation, a negative evaluation, and an indeterminate evaluation regarding the presence or absence of the behavioral disorder, the neurological disorder, or the developmental delay; (c) if the assessment results include the presence of the behavioral disorder, the neurological disorder, or the developmental delay, generating a personal therapeutic treatment plan for the individual based at least in part on the assessment results.

2. 2. The computer-implemented method of claim 1, wherein the at least one machine learning model comprises the one or more decision threshold hyperparameters that provide a positive predictive value of at least about 80%, a negative predictive value of at least about 95%, a coverage or envelopment rate of at least about 70%, or any combination thereof, when evaluated using a nested cross-validation procedure.

3. 3. The computer-implemented method of claim 1, wherein the one or more decision threshold hyperparameters are generated using an automated cross-validation procedure.

4. 4. The computer-implemented method of claim 1, wherein the one or more decision threshold hyperparameters define a threshold range for determining whether an evaluation result is a positive evaluation, a negative evaluation, or an indeterminate evaluation.

5. 5. The computer-implemented method of claim 1, wherein a first classification determination of the presence or absence of the behavioral disorder, the neurological disorder, or the developmental delay in the individual is based on a particular sensitivity, a particular specificity, a particular negative predictive value, or a particular positive predictive value.

6. 6. The computer-implemented method of claim 1, wherein the at least one machine learning model comprises a subset of a plurality of tunable machine learning models.

7. (a) requesting additional data if the assessment result includes the indeterminate assessment; (b) generating an updated assessment result based on the additional data using the assessment module; The computer-implemented method of any one of claims 1 to 6, further comprising:

8. 8. The computer-implemented method of claim 1, further comprising: training the at least one machine learning model with a first training dataset, a second training dataset, and a third training dataset, wherein the first training dataset comprises one or more video recordings of an individual, the second training dataset comprises one or more feedbacks provided by a healthcare provider in a questionnaire, and the third training dataset comprises one or more feedbacks provided by a caregiver in a questionnaire.

9. 9. The computer-implemented method of claim 8, wherein at least one of the first training data set, the second training data set, or the third training data set includes at least 100 training samples.

10. 10. The computer-implemented method of any one of claims 1 to 9, wherein the behavioral disorder, neurological disorder, or developmental delay comprises Pervasive Developmental Disorder (PDD), Autism Spectrum Disorder (ASD), Social Communication Disorder, Restricted Repetitive Behaviors, Interests, and Activities (RRB), Autism ("Classic Autism"), Asperger's Syndrome ("High Functioning Autism"), PDD Not Otherwise Specified (PDD-NOS, "Atypical Autism"), Attention Deficit Disorder (ADD), Attention Deficit Hyperactivity Disorder (ADHD), Speech and Language Delay, Obsessive-Compulsive Disorder (OCD), Depression, Schizophrenia, Alzheimer's Disease, Dementia, Intellectual Disability, or Learning Disability.

11. 11. The computer-implemented method of any one of claims 1 to 10, wherein the behavioral disorder, the neurological disorder, or the developmental delay is an autism spectrum disorder or autism.

12. The computer-implemented method of any one of claims 1 to 11, further comprising administering an individualized therapeutic treatment plan to the individual via a healthcare provider or caregiver of the individual.

13. 13. The computer-implemented method of claim 12, wherein the personalized therapeutic treatment plan is generated using a therapy module comprising at least one statistical or machine learning model.

14. 14. The computer-implemented method of claim 12 or 13, further comprising receiving feedback data based on performance of the individualized therapeutic treatment plan; and updating the individualized therapeutic treatment plan based on the feedback data.

15. 15. The computer-implemented method of claim 14, wherein the feedback data includes at least one of efficacy, compliance, or response to the individualized therapeutic treatment plan.

16. 16. The computer-implemented method of any one of claims 12 to 15, wherein the personalized therapeutic treatment plan includes pharmacological therapies, non-pharmacological therapies, or both.

17. 17. The computer-implemented method of claim 16, wherein the non-drug therapy comprises a digital therapeutic.

18. 18. The computer-implemented method of any one of claims 1 to 17, wherein the at least one machine learning model comprises a gradient boosted classifier model.

19. 1. A computer-implemented method comprising: (a) receiving input data comprising a plurality of features associated with a behavioral disorder, a neurological disorder, or a developmental delay; (b) dividing the input data into a training data set and a test data set; (c) training a model using one of the training datasets; (d) determining at least one decision threshold for said model using a corresponding test data set; (e) repeating steps (b)-(d) at least once using a cross-validation procedure to generate multiple decision thresholds; (f) determining one or more decision threshold hyperparameters using the plurality of decision thresholds; (g) training a final model using the input data, the final model comprising the one or more decision threshold hyperparameters.

20. The final model is (h) receiving individual input data associated with the behavioral disorder, the neurological disorder, or the developmental delay; (i) evaluating the individual's input data to generate an evaluation result, distinguishing between a positive evaluation, a negative evaluation, and an indeterminate evaluation regarding the presence or absence of the behavioral disorder, the neurological disorder, or the developmental delay according to the one or more decision threshold hyperparameters; and 20. The computer-implemented method of claim 19, further configured to: (j) if the assessment results include the presence of the behavioral disorder, the neurological disorder, or the developmental delay, generate an individualized therapeutic treatment plan for the individual based at least in part on the assessment results.

21. 1. A computer-implemented method comprising: (a) receiving input data comprising a plurality of features associated with a behavioral disorder, a neurological disorder, or a developmental delay; (b) dividing the input data into a training data set and a validation data set; (c) dividing the training data set into a training data set and a test data set; (d) training a model using one of the training datasets; (e) determining at least one decision threshold for said model using a corresponding test data set; (f) repeating steps (c)-(e) at least once using a cross-validation procedure to generate multiple decision thresholds; (g) training a new model using one of the training datasets; (h) evaluating the new model using a corresponding validation data set according to at least one decision threshold based on the plurality of decision thresholds; (i) repeating steps (a)-(h) at least once using a cross-validation procedure to calculate one or more performance metrics; (j) determining one or more decision threshold hyperparameters using the plurality of decision thresholds; (k) training a final model using the input data, the final model including one or more decision threshold hyperparameters.

22. The final model is (l) receiving individual input data associated with the behavioral disorder, the neurological disorder, or the developmental delay; (m) evaluating the individual's input data to generate an evaluation result, distinguishing between a positive evaluation, a negative evaluation, and an indeterminate evaluation regarding the presence or absence of the behavioral disorder, the neurological disorder, or the developmental delay according to the one or more decision threshold hyperparameters; and 22. The computer-implemented method of claim 21, further configured to: (n) if the assessment results include the presence of the behavioral disorder, the neurological disorder, or the developmental delay, generate an individualized therapeutic treatment plan for the individual based at least in part on the assessment results.

23. A system comprising a processor and a non-transitory computer-readable medium containing executable instructions configured to cause the processor to perform the computer-implemented method of any one of claims 1 to 22.

24. A non-transitory computer-readable medium comprising executable instructions configured to cause a processor to perform the computer-implemented method of any one of claims 1 to 22.

Images