Systems and methods for mitigating diversity, equity and inclusion bias in clinical artificial intelligence models

Abstract

A framework is provided to assess and mitigate bias in clinical artificial intelligence and machine learning. The framework includes features that demonstrate usefulness in indicating the potential causes of bias and ways to mitigate them. The framework measures classification parity, as well as the effect of biased words on predictions. Techniques are provided that reduce bias in training data (derived from electronic health records) by filtering the salient clinical information and neutralizing biased words. In an embodiment, training data for a machine learning engine is rewritten to mitigate bias in the training dataset, where the re-writing step includes a step of information density filtering on electronic health record training data and / or replacing biased words in electronic health record training data with relevant neutral versions of the biased words.

Description

TITLE: Systems and Methods for Mitigating Diversity, Equity and Inclusion Bias in Clinical Artificial Intelligence ModelsCROSS REFERENCE TO RELATED APPLICATIONS

[0001] The current application claims priority to provisional application Ser. No. 63 / 583.707, filed September 19, 2023, the entire disclosure of which is incorporated herein by reference. The cunent application is also related to International Application No. PCT / US2024 / 025223, the entire disclosure of which is incorporated herein by reference.BACKGROUND

[0002] The global pandemic has acted as a catalyst and highlighted the changes required in health and social care systems to ensure the ongoing well-being of the population, with an emphasis on mental health. These changes should address the challenges in early detection, prevention, and persistent disparities in closely related health outcomes, such as anxiety, depression, and suicidal ideation.

[0003] The use of clinical Artificial Intelligence (Al) has the potential to enhance healthcare by enabling informed decisions that lead to better patient outcomes. Al has already begun to influence medical diagnostics in primary care through structured data (Ref. 1). Mitigating bias in clinical Al models is crucial to ensure accurate and equitable healthcare outcomes.

[0004] The primary source of information on mental health care is free text (clinical notes). Despite the extensive and promising research in detecting mental illness using Al, an important challenge in building trustworthy models is the availability of high-quality data in sufficient volumes (Ref. 2). The available data are sparse and biased because their selection process reflects an intrinsically unfair healthcare system (selection bias) (see Fig. 1). Unfortunately, Al models are prone to exaggerating this bias in a vicious circle: when the model was fed data that reflect historical inequalities, it tends to perform worse for these groups and cause inequality in treatment outcomes amplify ing those existing inequalities. A recent study by Yates Coley et al. (Ref. 3) found poor performance in predicting the risk of suicide for underrepresented demographic groups: Black, Native American, and Alaskan patients.

[0005] Annotation bias is caused by the limitations of annotation, i.e., the difficulties of hiring a sufficient number of annotators or annotators with relevant expertise and the different views and perspectives that caregivers may have. When dealing with text, bias can be introduced through the language itself (textual bias or linguistic bias). This bias is caused bydifferences in word usage in texts describing different population subgroups. For example, the word "doctor” may occur more frequently in texts describing males, and the word "nurse” may be more often used in association with females. Finally, bias can occur due to overamplification created when Al models leam a spurious correlation between the patterns found in the data and the labels instead of relying on relevant information. For example, in the medical domain, chat bots have exhibited discrete sex and race bias when prescribing pain medications (Ref. 5), or associating women with cooking in semantic role labeling (Ref. 5).

[0006] Seeing the growing impact of Al on our lives, the consequences of bias present in models cannot be underestimated. There has been extensive research in the domain of bias assessment of Al algorithms in machine learning (ML) (Refs. 6-8) and natural language processing (NLP) (Refs. 9-11) as part of broader research on fairness (achieving equitable and non-discriminatory results). However, the problem of bias is non-trivial and heavily embedded in the social context. Multiple definitions, measures, and mechanisms already exist; additionally, there have been some attempts to unify them in frameworks such as Al Fairness 360 for structured data (Ref. 12) or the approach of Shah et al. for text (Ref. 13).

[0007] In contrast, the estimation of bias in the clinical text remains understudied. One of the few works on the topic of Roosli et al. (Ref. 14) estimates selection bias by assessing the performance and calibration of models built with structured data on in- and out-of-domain test distributions (generalization scenario), as well as within subgroups (demographic scenario). They start with a descriptive analysis of the cohort variables and missing data extracted. Following (Ref. 8), they measure selection bias in terms of classification parity (equality of classification error between groups) and calibration (difference between the predicted risk and the factual risk).

[0008] Healthcare analytics and Artificial Intelligence (Al) have the potential to transform healthcare. However, there is a technical problem in that Al models are prone to perpetuate the bias found in the data, which can increase the health inequities faced by minority populations. Estimating and combating bias has been a subject of active research in recent years. Bias in Al healthcare models has become a key focus of attention, mainly directed at structured data as a more straightforward goal. In the mental health domain, there is a focus on free text (clinical notes) as a significant source of information. For these unstructured data, the technical problems of data sparsity and bias are particularly pronounced.

[0009] Mental illness is one of the leading causes of disability worldwide. Children are most commonly affected by anxiety disorders, followed by behavior disorders, mood disorders, and substance use disorders. Alarmingly, fewer than half of youth diagnosed with mentaldisorders receive the requisite specialized mental health care, and often identification comes too late. Studies have shown that most adult mental disorders begin during childhood and adolescence, and Al has the potential to support clinical care as a sentinel for early identification (Ref. 15). By developing reliable algorithms based on large amounts of data, Al can predict the onset of mental illness long before it becomes symptomatic (Refs. 16-18). However, the precision of these predictions depends on valid, reliable, and objective mental health data in the EHR.

[0010] Historically, transformative change in health care has often been facilitated by technological advancement. The power of big data offers the opportunity to improve the assessment of population health and disparities in care and outcomes by race / ethnicity, socioeconomic status, language, immigrant status, and other dimensions. Although Al models have immense potential to raise healthcare standards, it is imperative to understand and address data biases. It is unlikely that all bias can be identified, assessed, or “corrected’’, but frameworks to address bias are sorely needed amid the rapid application of new healthcare analytics and Al.

[0011] In practice, in the challenging pediatric primary care setting (multiple informants, variety in interpretation of risk factors and symptoms, overlapping symptoms) (Refs. 19-21), a bias-aware screening Al system is necessary to produce more fair and accurate decisions via the bias reminder mechanism (displaying bias measure).SUMMARY

[0012] The current disclosure introduces a framework to assess and mitigate bias in clinical Al, which can be widely applied to different models built with free text. Once extracted from this clinical text, the framework includes features that demonstrate usefulness in indicating the potential causes of bias and ways to mitigate them. The framework measures classification parity', as well as the effect of biased words on predictions. Moreover, the current disclosure provides a technique that reduce bias by filtering the salient clinical information and neutralizing biased words. In an embodiment, the current disclosure presents the utility of the framework for the anxiety detection task in pediatric patients for the sex demographic subgroups. On this task and given the disclosed bias definition, it is detected that female patients get systematically under-diagnosed due to differences in information density and textual distributions of patient notes. Using the disclosed mitigation methodology, this bias can be reduced by up to 27% in an embodiment.

[0013] The current disclosure identifies and mitigates textual bias with respect to demographic subgroups found in the text of electronic healthcare records (EHR) related to apsychiatric condition such as pediatric anxiety. The current disclosure considers the situation where the textual bias in the training data causes predictive bias (which refers to the bias that arises from the predictions made by an ML model) as a scenario where models produce consistently different predictions of population subgroups and potentially for wrong reasons (i.e., relying on biased words). The current disclosure anticipates that this type of bias will have a significant impact on models operating in production. Embodiments of the current disclosure considers textual bias within sex subgroups and assess the outcome parity across those subgroups. Embodiments of the current disclosure also look into the effect of gender-related words (first and last names, gender-related pronouns) on the predictions.

[0014] An objective of the current disclosure is to conceptualize an adaptive bias measure for automated patient screening to estimate bias in the incoming data together with predicting outcomes. Screening is difficult to implement in the pediatric primary care setting, with traditional approaches that rely on multiple informants often producing discrepant findings (Ref. 19). Beyond this, clinicians in the primary care setting vary in how they consider risk factors and interpret symptoms (Ref. 20), and clinicians may struggle to disentangle overlapping symptoms (Ref. 21). In addition, clinicians are already charged with screening for growth and development concerns, trauma, educational concerns, safety, sleep, and dietary concerns, as well as screening for mental health problems and social determinants of health, in addition to the “hand-on-door’ questions (Ref. 22). These challenges may often hinder comprehensive screening for mental health concerns. Merely having access to standard Al screening assistance is insufficient in those conditions, emphasizing the necessity for bias- aware Al to mitigate unconscious bias and enhance the accuracy and objectivity of decisions. The framework of the current disclosure is a step towards the creation of such bias-aware Al.

[0015] The current disclosure provides: (a) a methodology for a detailed analysis of textual distributions of demographic subgroups in mental health text that hints towards causes of bias and ways to mitigate them; (b) a definition of bias for mental health text involving the measures of classification parity and of the effect of biased words on predictions involving the usage of interpretability techniques; and (c) widely applicable bias mitigation methodology that normalizes information density to reduce bias and neutralizes biased words to reduce reliance on them.

[0016] In a specific embodiment, the task of predicting pediatric anxiety, the current disclosure analyzes intrinsic differences in textual properties of notes for male vs. female population subgroups (such as information density and dissimilarities in word distributions, see subsection Descriptive Analysis in Results). It is observed that these differences causeperformance decreases (bias) for the non-privileged subgroup (females) in predictive Al models built using these textual data, as well predictions that rely on biased words.

[0017] Taking one step forward, the current disclosure provides a de-biasing technique that balances information density by filtering out irrelevant content and replaces biased words (such as first and last names, as well as gender pronouns) with their neutral versions. In this specific embodiment, for the sex subgroups, the disclosed information density normalization method improves classification parity by up to 27% and reduces the reliance on biased words.

[0018] As such, in an aspect of the current disclosure, a method is provided for training and preparing a machine learning model for use in diagnosing or predicting a psychiatric or behavioral condition, or an event related to a psychiatric or behavioral condition. The method includes, collecting first electronic health records from previous patients who have been diagnosed with a psychiatric or behavioral condition or have experienced an event related to the psychiatric or behavioral health condition and a second electronic health records from previous patients who have not been diagnosed with the psychiatric or behavioral condition or have not experienced a related event: processing the first and second collected electronic health records into a training dataset; and training a machine learning model with the training dataset; where the processing step includes re-writing at least a portion of the first or second electronic health records to mitigate bias in the training dataset. In a more detailed embodiment, the rewriting step includes a step of information density filtering on the portion of the first or second electronic health records. In a further detailed embodiment, the re-writing step includes a step of replacing biased words in the portion of the first or second electronic health records with relevant neutral versions of the biased words. In a further detailed embodiment, the biased words are gender-specific words and the neutral versions of the biased words are gender-neutral counterparts to the gender-specific words. In a further detailed embodiment, the genderspecific words and gender-neutral counterparts to the genders specific words include proper nouns. Alternatively or in addition, the gender-specific words and gender-neutral counterparts to the genders specific words include pronouns. Alternatively or in addition, the step of information density filtering on the portion of the first or second electronic health records includes: sconng salient words as a multiple the frequency of a word in a given health record with respect to the word’s rarity across other given health records, to provide a word-level TF- IDF score for each word in the given health record; computing an averaged sum of word-level TF-IDF scores for each sentence in the given health record; and removing a predetermined percentage of the lowest scoring sentences. Alternatively, or in addition, the processing stepfurther includes organizing data from the first and second collected electronic health records into a set of time limited bins.

[0019] In another aspect, a method is provided for training and preparing a machine learning model for use in diagnosing or predicting a psychiatric or behavioral condition, or an event related to a psychiatric or behavioral condition. The method includes the step of: collecting electronic health records from previous patients who have been diagnosed with a psychiatric or behavioral condition ; processing the collected electronic health records into a training dataset; and training a machine learning model with the training dataset; where the processing step includes detecting the presence of bias in the training dataset by, at least in part, analyzing textual distributions in the electronic health records. In a more detailed embodiment, the step of analyzing textual distributions in the electronic health records includes: evaluating skewness of vocabulary frequency distribution in data from the electronic health records; evaluating burstiness or one or more words in data from the electronic health records; determining an average length of a patient note in words from the electronic health records; determining unique word vocabulary overlap in a first subset of the electronic health records versus a second, larger subset of the electronic health records; determining a topic count in the first subset of the electronic health records; and / or determining a percentage of the first subset of the electronic health records that does not pertain to identified topic. In a further detailed embodiment, the step of evaluating skewness includes detecting the presence of rare w ords in the data from the electronic health records. Alternatively, or in addition, the step of evaluating burstiness includes calculating an average probability that the one or more words will appear in the data from the electronic health records. Alternatively, or in addition, the first subset of the electronic health records is a first time limited bin of the data from the electronic health records and the second, larger subset of the electronic health records is a plurality of time limited bins of the data from the electronic health records. Alternatively, or in addition, the method further includes the step of re-w riting at least a portion of the electronic health records to mitigate detected bias in the training dataset

[0020] Another aspect is a method for training and preparing a machine learning model for use in predicting bias in electronic health records. The method includes: collecting a plurality of previous electronic health records; for each of the plurality7of previous electronic health records, calculating a balanced error rate (BER), which is a measure of model bias using classification parity (equality of classification error between groups); and training a machine learning model with the electronic health records and each corresponding calculated balancederror rate (BER). In a more detailed embodiment, the balanced error rate (BER) is calculated according to the following equation.FP FN )BER=( FP+TN )+( FN+TP2 where FP is the number of false positives; FN is the number of false negatives; TN is the number of true negatives and TP is the number of true positives. Alternatively, or in addition, the method includes inputting a new electronic health record to the machine learning model to predict a balance error rate (BER) for a new electronic health record. Alternatively, or in addition, the method may include re-writing at least a portion of the new electronic health record if the predicted balance error rate (BER) exceeds a predetermined value.

[0021] Another aspect provides a method for diagnosing or predicting a patient's psychiatric or behavioral condition, or for predicting an event related to a patient’s psychiatric or behavioral condition that includes: providing a machine learning engine trained according to any of the above aspects; collecting a patient's one or more electronic health records; processing the patient’s one or more electronic health records to generate input data for the machine learning engine; feeding the input data to the machine learning engine, and receiving from the machine learning engine (a) a diagnosis or a prediction of the patient’s a psychiatric or behavioral condition, or (b) a prediction of an event related to the patient’s psychiatric or behavioral condition. Another aspect provides a method for treating a patient’s psychiatric or behavior condition, compnsing: (a) diagnosing or a predicting a patient’s a psychiatric or behavioral condition, or (b) predicting an event related to the patient’s psychiatric or behavioral condition using a machine learning engine trained according to any of the above aspects; and providing treatment to the patient. In a more detailed embodiment, the step of providing the treatment to the individual includes treating the individual with cognitive behavioral therapy or another similar evidence based behavioral therapy. Alternatively, or in addition, the step of providing the treatment to the individual includes treating the individual with a medication selected from: (a) a selective serotonin reuptake inhibitor; (b) serotonin-norepinephrine reuptake inhibitor; (c) benzodiazepines; (d) beta blockers; (e) tricyclic antidepressants; (I) monoamine oxidase inhibitors; (g) atypical antidepressants; (h) lithium; (i) antiepileptics; (j) first, second, or third generations antipsychotics; (k) stimulants; (1) non-stimulant, such as atomoxetine, clonidine, guanfacine, and / or modafinil; and (m) non-benzodiazepine anxiolytics.

[0022] Another aspect of the current disclosure includes a machine learning engine trained according to any of the above aspects.

[0023] Another aspect of the current disclosure includes one or more non-transitory memory devices including computer instructions configured to direct one or more processors to perform any of the methods as set forth in any of the above aspects.

[0024] These and other aspects and advantages of the current disclosure will be apparent from the following description, the appended claims and the attached drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described below are for illustrative purposes only. The drawings are not intended to limit the scope of the present teachings in any way.

[0026] Fig. 1 a diagram illustrating how bias, such as linguistic bias, in electronic health records may adversely impact accuracy of clinical artificial intelligence based on such electronic health records.

[0027] Fig. 2 is an exemplary flow diagram for training a machine learning model for use in diagnosing or predicting a patient’s health condition, or predict an event related to a patient’s health condition.

[0028] Fig. 3 is an exemplary flow diagram for using a machine learning model, such as trained in Fig. 2. to diagnose or predict a patient’s health condition, or predict an event related to a patient’s health condition, and then providing treatment to the patient.

[0029] Fig. 4 is an image representing ty pes of electronic health record topics that are over- represented in certain cultural subgroups versus other cultural subgroups.

[0030] Fig. 5 is an image representing types of electronic health record topics that are over- represented in certain gender subgroups versus other gender subgroups.

[0031] Fig. 6 is an image representing keywords most frequent in electronic health records associated with over-represented topics for certain cultural subgroups.

[0032] Fig. 7 is an exemplary flow diagram for training a machine learning model for use in predicting balanced error rates (or associated bias) in certain electronic health records.

[0033] Fig. 8 is a histogram chart of BER ratios calculated for a subgroup of data.

[0034] Fig. 9 is a histogram chart of BER ratios calculated for a Male / Female subgroup of data before rewriting according to an exemplary method.

[0035] Fig. 10 is a histogram chart of BER ratios calculated for a Male / Female subgroup of data after rewriting according to an exemplary method showing a decrease in bias.DETAILED DESCRIPTION

[0036] The methods may comprise, consist of, or consist essentially of the elements of the methods as described herein, as well as any additional or optional element described herein orotherwise useful in methods for training and preparing a machine learning model for use in diagnosing a psychiatric condition.

[0037] Turning first to FIG. 2, this figure illustrates a method that may be used to create software and / or used to train and prepare one or more machine learning models for use in the early identification and (earlier) treatment of a psychiatric condition, such as an anxiety disorder. As shown in FIG. 2, such a method may begin with obtaining 101, a set of electronic health records (EHRs) associated with the psychiatric condition. This may include using a computer to query an EHR repository for records of patients diagnosed with an anxiety disorder (e.g., as shown through the presence of ICD9 and / or ICD10 diagnosis codes) and then anonymizing those records to obtain an anonymized anxiety disorder dataset. This may also include applying other parameters in obtaining the dataset as described herein.

[0038] Once the psychiatric disorder records have been obtained 101, they may be separated 102 into cohorts. This may be done, for example, by dividing the EHRs based on age at the time of diagnosis. Following the separation of 102, the cases may be matched to 103 with corresponding controls. This may be done in several ways. For instance, EHRs for patients with ages and genders matching those from the case group but who had not received a diagnosis of the psychiatric condition may be retrieved from EHR system, anonymized, and then paired with the matching cases. Data that would be relevant to the psychiatric condition (e.g., anxiety7diagnoses) could then be extracted 104 from the records. Optionally, the extracted data could be organized 105 into bins, such as time bins. At any point in the process, the EHR data may be preprocessed 106 to prepare it for use in training 107 one or more machine learning model(s). The preprocessing step 106 may include the framework for identitying and mitigating bias in the data set as disclosed herein.

[0039] Once the data set is preprocessed 106, it could be used to train 107 one or more machine learning model(s). This may be done, for example, by individually training a model for each age cohort. This could be done using a variety of machine learning model types, such as classic and deep learning methods, which would directly output a prediction, classifiers which would specify potential delays until diagnosis as various classes with various probabilities, decision trees such as XG Boost, or other types of machine learning models. These machine learning models could then be subjected to some level of validation (e.g., 10- fold cross-validation) and, once validated, could be incorporated into softw are stored in a non- transitory computer-readable medium and used to configure the processor of a computer to diagnose a patient's specific psychiatric condition and / or future risk for developing such psychiatric condition.

[0040] A method of treatment that may be performed using such trained machine learning model is illustrated in FIG. 3. Initially, in that method, an electronic health record would be received 201 for an individual. This may be done, for example, by a treating physician retrieving a medical record for a patient under his or her care from a medical record system. After the electronic health record had been retrieved 201, it could be used to determine 202, a plurality of data values. This data value determination 202 may be performed, for example, by using natural language processing software to extract unstructured data or otherwise using techniques such as those described herein in the context of training to retrieve information from the electronic health record. Once the data values had been determined 202, the method of FIG. 3 continues with the generation 203 of diagnosis input data. This may include processing the data from the electronic health record like that discussed herein in the context of training (e.g., normalizing and organizing the data into time-limited bins) to put it in the same form as the data used to train the machine learning model. Once the data was in the form that the machine learning model had been trained to use as input, it could be provided 204 to the trained machine learning model, and, based on the output of that model, the individual could be diagnosed 205 for a psychiatric condition and / or a future risk of developing a psychiatric condition and / or a future risk of some other related occurrence (e.g. emergency room visit, hospitalization, suicide attempt). Finally, if the patient is identified as needing treatment, they could be provided 206 with treatment. For example, providing treatment to the individual may comprises treating the individual with a medication selected from appropriate medications for the psychiatric condition which could include (a) a selective serotonin reuptake inhibitor; and (b) serotoninnorepinephrine reuptake inhibitor; and (c) benzodiazepines; and (d) beta blockers; and (e) tricyclic antidepressants; and (f) monoamine oxidase inhibitors; and (g) atypical antidepressants; and (h) lithium; and (i) antiepileptics; and (j) first, second, or third generations antipsychotics; and (k) stimulants; and (1) non-stimulant (atomoxetine, clonidine, guanfacine, and modafinil); and (m) non-benzodiazepine anxiolytics. Treatment may also include providing cognitive behavioral therapy and other types of psychotherapies.

[0041] The current disclosure provides a definition of textual bias as discrepancies in prediction performance across to demographic group traceable to differences in information density and differences in word distributions in the training data, as well as an interpretability method to investigate how much model predictions rely on biased words. The disclosed debiasing methodology balances information density to reduce bias and improve classification outcomes. By neutralizing biased words the cunent methodology reduces reliance on those words and improve generalization. The disclosed bias mitigation methodology is widelyapplicable and generalizable to other datasets and types of bias (given a dictionary of biased keywords).

[0042] For the exemplary task of predicting pediatric anxiety, it is observed that intrinsic differences in textual properties of notes across demographics can lead to performance decrease (bias) for non-privileged subgroups. Word substitution techniques can reduce those differences while preserving the original content. In the exemplary case, they helped achieve a bias reduction of up to one third in terms of equal possibilities for female patients which were systematically under-diagnosed.

[0043] Using accurate calculations and reducing data bias can positively affect both patients and the broader population. By minimizing biases, healthcare providers can more effectively match the disease phenotype with appropriate care. Biased Al models can perpetuate health disparities in minority' and sex groups, creating a harmful feedback cycle. However, by aggregating data extracted from clinical text, population-based reports can become more precise and reliable.Anxiety Prediction Models

[0044] Exemplary anxiety prediction models were built by fine-tuning the state-of-the-art Transformer-based Clinical-BigBird model (Ref. 23) as imported from HuggingF ace (Ref. 24). This model is not only pre-trained on clinical text but can also handle long input sequences (up to 4,096 tokens).

[0045] Best practices in the domain were followed and fine-tuned the model for 2 epochs. Input length of notes was limited to 1 ,000 tokens (since considering longer inputs did not result in further improvement). AdamW optimizer (Ref. 25) was used with a learning rate of le-5 and a batch size of 8 (these were the best-performing set of parameters out of the sets suggested for fine-tuning by the model authors (Ref. 23)).Explainability

[0046] In addition to measuring performance of the models with the standard accuracy measure, qualitative analysis of words the models rely upon were performed on while making predictions. The Local Interpretable Model-Agnostic Explanations (LIME) technique (Ref. 26) enables this qualitative analysis by offering local explanations, pinpointing specific words that significantly influenced the model decisions. These explanations are “local” because they relate to the model’s behavior for each specific incoming note. Note that globally important features (for example, weights that a model assigns to words in its vocabulary) might not be precise enough for the local prediction context.

[0047] In the example study, LIME methodology was used to highlight influential words to verify if the prediction models are functioning correctly. LIME is designed to uncover undesirable behaviors in Al models that might seem efficient based on standard metrics. For example, Ribeiro et al. (Ref. 26) showed that a model, despite a 94% accuracy in differentiating documents on Atheism versus Christianity, relied on irrelevant words such as “posting”, “host” and “re”. These words were wrongly associated with Atheism due to their frequent appearance in the training data.Text De-biasing Methods

[0048] Motivated by such observations over the data as differences in information density and textual distributions across demographic subgroups, the current disclosure provides two bias mitigation methods:

[0049] 1. Information density filtering (td-idfjilf): Normalization of content is performed which involves filtering sentences from concatenated notes using their importance scores. Those importance scores are computed as averaged sum of word-level TF-IDF scores per sentence. TF-IDF scores help identify most salient words in a document by multiplying the frequency of the word in that document by its rarity across all documents (the more rare is the word the higher is the score). So that document keywords (e g., term “myopia”) that may appear multiple times in one document but not in the others receive higher score than auxiliary words such as articles (“the”, “a”) or verbs (“am”, “have”).

[0050] This exemplary method keeps a percentage (e.g. 80%) of the most salient sentences (threshold defined empirically). This allows the models to take into the most informative data into account. This approach is compared to the baseline approach where 20% of sentences chosen at random (rnd Jilt) were removed.

[0051] 2. Gender-word debiasing (gen sub) Following the best de-biasing practices from the NLP domain (Ref. 27), this method focuses on names and pronouns as gender-biased attributes in text. This method automatically detects those biased words and replace them with relevant neutral versions. In particular, this method detects proper nouns (first and last names) using the off-shelf Stanza tool (Ref. 28). This method extracts unique names and group them based on the character similarity, ensuring that variations of names are considered equivalent (for example, “Jonathan”, “Johnathan” and “Johnatan”). This grouping facilitates the creation of a mapping system where each name group is replaced by a generic identifier, such as “personl”, “person2”, etc. (enumeration is maintained per note).

[0052] Following the name replacement, the text undergoes pronoun substitution, where gender-specific pronouns from a dictionary (for example, “she” and “he”) are replaced withtheir gender-neutral counterparts using a predefined pronoun mapping (for example, “she” and “he” are replaced with “they”). Note that this approach maintains the integrity and coherence of the original text while achieving certain gender neutrality.

[0053] These two exemplary7methods could be used separately or combined. Additionally, substitution of biased words could be applied to the original text before or after the removal of the least informative sentences.EXAMPLE 1Datasets

[0054] As part of this example study, foundational database was created consisting of Electronic Health Record (EHR) data from the Cincinnati Children’s Hospital Medical Center Epic Link, an online platform allowing providers real-time access to patient charts and results. This database has approximately 1.3 million unique patients seen at CCHMC between January 1, 2009, and March 31, 2022, with 63 million clinical notes. Patients diagnosed with at least one ICD 9 / 10 code related to an anxiety disorder were included in this study as patients with anxiety7. (Refer to Table 18 below for a list of anxiety-related ICD 9 / 10 codes).

[0055] For 1.3 million patients in the CCHMC EHR, inclusion and exclusion criteria were applied as explained in the Methods section to obtain the analytic anxiety cohort of 66,299 patients with 5.1 million notes.

[0056] This cohort contains various demographic data (gender and race) and the timeline of clinical textual notes provided by different care providers (progress notes, telephone encounters, plan of care notes, patient instructions, etc.). These data were initially cleaned as described in the Data Cleaning subsection of the Methods section below. A representative Bin of age 5 was used and tested on the neighboring age Bins of 4 and 6. There was a focus on progress notes and telephone encounters as deemed the most informative by internal experts. Timelines were kept with at least one record of one of those ty pes and maintained the 1:1 proportion of cases and matches.

[0057] After removing duplicates (see the Data Cleaning subsection in Methods), each Bin contained the training set of around 4.3-4.6K cases and controls, and the testing set included around 1.1-1.2K cases and controls.Descriptive Analysis

[0058] Before conducting the study, the demographics (race and gender) of the dataset and the properties of relevant texts were carefully analyzed. The following traditional statisticalcharacteristics of textual distributions (the choice of statistics was largely inspired by Chan et al. (Ref. 29)) in particular were analyzed:

[0059] 1. Skewness of the vocabulary frequency distribution, which characterizes its asymmetry, meaning the higher the skewness, the longer the tail of rare words in the distribution.

[0060] 2. Burstiness as the average probabili ty that a word will appear in a document with the frequency of the highest quartile for the corpus (probability was normalized by the frequency of words in the corpus to favor less frequent words; we largely followed the definition of Serrano et al. (Ref. 30)). The metric largely reflects whether single notes tend to be devoted to one topic (word occurrences burst within a particular note causing a higher burstiness value) or to be devoted to a general topic (word occurrences are even across notes causing a lower burstiness value).

[0061] 3. Average length of a patient note in words (without tokenization, i.e., separation of punctuation). We measured the maximum length of the note per patient as the most representative for the calculation of this statistic.

[0062] 4. Unique word vocabulary overlap between the distribution of the whole bin and the subgroup distribution (computed for demographic subgroups).

[0063] Table 1. Demographic statistics and properties of textual notes across Age bins 4, 5, and 6 (measured for the training data). Topics are extracted with the help of the unsupervised semantic BERTopic (Ref. 31) technique. No topic size indicates the percentage of examples in the data that were not assigned to any topic (general patient information).

[0064] Table 2. Class statistics and properties of textual notes across race demographic subgroups in Bins 4, 5, and 6 (as measured for the training data). Vocabulary overlap is measured with respect to the vocabulary of the entire bin. Topics are extracted with the help of the unsupervised semantic BERTopic (Ref. 31) technique. No topic size indicates the percentage of examples in the data that were not assigned to any topic (general patient information).

[0065] 5. Topic count in the bin / subgroup as extracted by BERTopic (Ref. 31). We used this metric as a proxy to the thematic diversity (see Subsection Topic Models of the Methods section)

[0066] 6. Percentage of examples in the data not assigned to any topic (No topic size) byBERTopic. We used this metric as a proxy for the number of examples that describe more general information.

[0067] The results of the analysis are presented in Table 1 between the bins first and then in Tables 2 and 3 between demographic groups within the bins. In addition to the White vs. Black / Afro-American subgroups we also report the statistics for the other races subgroup for better comparative analysis.

[0068] Initial observations were that there are no notable changes in demographics and text properties across age bins 4-6 (Table 1). There are similar ratios of males and females across bins (60 / 40 ratio), as well as similar ratios across race subgroups (70 / 20 / 10 for White, Black / Afro- American and Others, respectively), and the length of Bin 4 notes is slightly lower than the average of 1.1K tokens for the other two bins. Each of the three bins contains approximately 60 topics.

[0069] In contrast, the differences between demographic groups are more drastic (see Tables 2 and 3). The percentage of cases is equally spread among race subgroups (50 / 49 / 50 on average for the White, Black / Afro-American and Others, respectively). The most obvious differences lie in the length of the notes, with the notes for the White subgroup being around 130 takes longer than the notes for the other race and, on average, 160 words longer than the notes for the Black / Afro- American subgroup. Notes of the other race subgroups also contain,on average, three times fewer topics than those of the White subgroup, covering only around 1 / 3 of the whole bin vocabulary. The reduced percentage of patients from non-white subgroups and the length of the notes could explain these observations. The distributions of the non-white subgroups are also less skew ed (-9 on average) and more bursty (+0.02 on average), indicating that the notes for those subgroups tend to focus on a specific topic with certain keywords repeated.

[0070] The percentages of cases are equal across gender subgroups as well, which are predefined by the cohort design. For the other features, similar differences are observed between gender subgroups.

[0071] Table 3. Class statistics and properties of textual notes across gender demographic subgroups for Bins 4, 5, and 6 (as measured for the training data). Vocabulary overlap is measured with respect to the vocabulary of the entire bin. Topics are extracted with the help of the unsupervised semantic BERTopic (Ref. 38) technique. No topic size indicates the percentage of examples in the data that were not assigned to any topic (general patient information)

[0072] As shown in Fig. 4, topics as extracted by BERTopic for Bin 5 training data across race demographic subgroups. The percentage of sentences are shown by grey scale darkness (darker grey scale indicates a higher percentage) for the 50 most frequent topics.

[0073] As shown in Fig. 5, topics as extracted by BERTopic for Bin 5 training data across gender demographic subgroups. The percentage of sentences are show n by grey scale darkness (darker grey scale indicates a higher percentage) for the 50 most frequent topics.

[0074] As shown in Fig. 6, topics present for the Black / Afro-American subgroup of the Bin 5 training data represented by their most frequent words. The topics of poisoning, cardiovascular problems, skin conditions, and medical examination w ere observed present for this race subgroup. These topics are less frequently detected for the other race subgroups on average 175 words shorter than those for the male subgroup. Like race minority groups, they cover around 1 / 3 of the whole bin vocabulary. Topics covered in the female subgroup are morediverse: the female notes contain only 1.5 times less topics than the male notes (compare to the reduction of three times for the race subgroups). Skewness and burstiness differences are twice as extreme compared to race subgroups (-4 on average and +0.01, respectively). Given that the female subgroup is fairly well-represented (40% of the examples), the differences observed cannot be explained by scarcity and suggest qualitative differences in content of the notes across demographic subgroups.

[0075] Indeed, taking a closer look at the topics qualitatively (Fig. 4). it is observed that the Black / Afro-American subgroup is mainly represented with topics that are almost not present for the White and Others subgroups, indicating that the content of those notes is very7different from the content of the notes for the other race subgroups. Such drastic differences in topic distribution are not observed across the gender demographic subgroups (Fig. 5). However, some topics are better represented for males than for females. Fig. 6 presents the keywords of the four topics that are most frequent for the Black / Afro- American subgroup: for example, poisoning and skin conditions are often discussed. Similar topic differences were observed within demographic subgroups for the other two bins.

[0076] It is concluded that the cohort matching procedure addressed the problem of severe differences. However, statistical differences for race demographic subgroups remain drastic, and it is expected that the bias across those subgroups to be the most pronounced for predictive Al.Bias Measure

[0077] Best practices in Machine Learning (Ref. 8) measure model bias in terms of anticlassification (effect of protected attributes on predictions), classification parity7(equality7of classification error between groups), and calibration (difference between predicted risk and factual risk). When applied to text, the anti-classification and calibration measures are particularly suitable to assess linguistic bias and model over-amplification. For example, predictive models can erroneously rely in their predictions on words in the training data that are directly associated with protected demographic attributes (e.g., relying on the pronoun “she” indicating gender to predict the increased probability7of depression or on the word “teacher” carrying the semantic of the female gender in its pre-trained representation (Ref. 32)).

[0078] Since there is an aim to assess selection bias in test data, there is focus on classification parity7. Several metrics are commonly used in the fairness literature to evaluate classification parity. These are accuracy equality7(equal accuracy across groups), equal opportunity7(equal false negative rate (FNR), orientation towards recall) or predictive equality (equal false positive rate (FPR) across groups, orientation towards precision)( Refs. 33, 34).Following Feldman et al. (Ref. 33), the balanced error rate (BER) is used (see, Eq. 1 below), which is the unweighted average of FPR and FNR. BER was chosen as the primary metric because there was a focus on Fl -score as a performance metric, which involves both FPR (precision) and FNR (recall). The unweighted average was used to emphasize the importance of both precision and recall (enough relevant patients should be monitored, but there is a desire to avoid too many). To assess bias, the ratio was computed comparing the non-white subgroup (such as race minority) with the white subgroup (such as race majority). Hence, a ratio of >1 hints towards better performance for the white subgroup measured by BER, while a ratio <1 indicates that a model performs better for the non-white group.

[0079] BER ratio measure was used to assess the bias of the state-of-the-art Transformerbased Clinical-BigBird (Ref. 23) prediction model built using Bin 5 training data (see subsection Anxiety Prediction Models in Methods for more details). The model was built using the maximum length note in the patient’s history because, during empirical evaluations, this set-up gave the best performance (see Table 10 for similar empirical conclusions). The results are shown in Table 4 across bins and Table 5 across demographic subgroups.

[0080] The model achieves the performance of 0.56 Fl on Bin 5 test data (this performance is significantly above random). When the Bin 5 model is applied to test sets from Bin 4, the BER ratio remains around 0.99 with a minor performance increase of -0.01 Fl. When the model is applied to test sets from Bin 6, a significant performance increase of +0.04 Fl is observed towards these data. These observations indicate no tendency towards bias in non-white bins (bins 4 and 6) (Table 4).

[0081] Table 4 Classification parity results across bins for Clinical-BigBird trained on Cohort 5. We report macro Fl-scores and Balanced Error Rates (BER). f and * mark statistically significant differences to Fl macro scores of class probability baseline and Fl macro scores of Bin 5 as measured by the 2S-KS test. Bold highlights are statistically significant improvements. Results per bin are averaged over 10 test sets sampled from all the bin data respecting the gender balance for this bin.

[0082] Table 5. Classification parity results across demographic subgroups for Clinical- BigBird trained on Cohort 5. We report macro Fl -scores and Balanced Error Rates (BER). Bold highlights the BER ratios > 1.1. * marks statistically significant differences in Fl macro scores between white and non-white subgroups as measured by the 2S-KS test. Results per bin are averaged over 10 test sets sampled from all the bin data respecting the gender balance for this bin.

[0083] Looking closer at the bias within demographic subgroups, it is observed that the model is slightly more biased towards White patients with an average BER ratio of 1. 13 than towards Male patients (BER ratio of 1.06) (see Table 5). However, even this slight bias results in a significant performance drop for the Black / Afro- American subgroup of around -0.05 Fl across bins, a similar significant drop of -0.06 Fl is also observed for Bin 5 Female subgroup where the bias is the strongest (1.12 BER ratio).

[0084] In summary, bias, as monitored by BER ratio, correlates well with differences in textual statistics from the Descriptive Analysis subsection: it is slightly biased towards the White subgroup, demonstrating a performance decrease for the Black / Afro- American subgroup. There is no clear bias pattern when Male / Female subgroups are looked into: depending on the bin, performance may be better for either of the subgroups or equally good. Although the bias may not be substantial, it is recognized that a small bias can have a significant impact on model performance.Bias Prediction

[0085] When testing models with new data there is a need to be able to reliably predict the presence of bias, which is essential to evaluate the trustworthiness of the anxiety predictions. For this, prediction model was built that is trained on data consisting of test sets and their BER ratios (see Fig. 7).

[0086] Table 6 Statistics over 17 textual features in the bias predictor training data sampled from Bin 6. Vocabulary overlap is measured with respect to the vocabulary of the bin. Topics are extracted using BERTopic.

[0087] Fig. 7 provides the prediction model pipeline. The prediction model 700 was constructed by training it on a dataset of test sets 702 and their corresponding BER ratios 704. These test sets are represented by 17 textual features. The trained model can predict BER ratios 706 for new, unseen test data 708.

[0088] As shown in Fig. 8, BER ratios in 1.558 test sets sampled from Bin 6 data. During sampling, equally-sized batches were considered for race and gender subgroups. Within each batch, the original demographic group proportions were considered and included an equal number of examples containing patient notes ranging from note 5 to note 24 in the timeline, instead of solely considering the longest note. This was done to introduce more variety into the bias prediction training data.

[0089] Table 7. Bias prediction results. Results are averaged over 10 runs with different random seeds. * mark statistically significant differences to Fl macro scores of the class probability baseline as measured by the 2S-KS test. Bold highlights statistically significant increases.

[0090] Following the previous observations regarding textual properties and related bias (subsections Descriptive Analysis and Bias Measure above), it was decided to rely on similartextual features to build the predictor. In particular, for each demographic subgroup separately, the six features mentioned in the subsection Descriptive Analysis (12 in total) were extracted.

[0091] In addition, five more features in total were extracted. After building a vector indicating the presence of the topics for each subgroup, we compare them to topic vectors built for Bin 5 cases and controls subgroups using cosine similarity (four features). This is to see whether the subgroups are closer to cases or controls in the model training data.

[0092] The line feature directly relates to how our test sets from Bin 6 data were built. We sampled 1,558 test sets. Half of these test sets were sampled for the race subgroups, and half of those sets were sampled for the gender subgroup while respecting original subgroup balances. Additionally, instead of always choosing the maximum length note, we picked notes at different steps of the patient timeline from note 5 up to note 24 inclusive. The intuition behind this setup was to increase representatives of our bias scores as patient data are appearing progressively in the timeline in the real-life setting.

[0093] In general, 17 features were extracted from Bin 6 data (see Table 6 for statistics). These statistics mirror previous observations and provide a good range of values across features. Notes of the white subgroup are in general more skewed, less bursty, longer and contain a larger variety of topics no matter the note location in the timeline. As for the topic similarity7, the white group has on average 25% higher similarity7to the training data topics.

[0094] Analy sis of the BER ratio scores for the test sets shows that in 14% of the test sets, bias toward the non-white class (for BER ratios between 0.80 and 0.95) is observed, 49% of the examples contain almost no bias (for BER ratios between 0.95 and 1.05) and 37% of the examples with bias towards the white class (for BER ratios between 1.05 and 1.20) (see Fig. 6). Hence the task is cast as a three-class prediction problem: bias to non-white subgroup, nobias and bias to white subgroup.

[0095] As detailed in the subsection Bias Prediction Models of Methods, the BER ratio prediction task is addressed by building a decision tree model (Random Forests, RFs (Ref. 35)). RFs are chosen as a particularly good model for sparse classification tasks such as this. Out of the 1,558 test sets sampled, 70% were used for building the model and 30% for testing this model.

[0096] The results of the experiments are shown in Table 7. It can be seen that the Fl scores for the no-bias and white bias class show statistically significant improvements compared to the random (class probability ) baseline and are quite good in general, with an average of 0.59 Fl. The most important features contributing to this performance are the skewness and burstiness of the non-white class topic similarity7of the non-white class to the case topics.However, the performance for the non-white bias is unsatisfactory' due to sparsity and is significantly lower than random.

[0097] Furthermore, it is argued that the most important distinction needing to be drawn whilst predicting bias is between white bias and the rest of the bias types. Hence, the non-white bias and no bias classes are merged together and perform a binary classification task. This boosts the Fl macro across classes up to 0.63 Fl and provides statistically significant improvements over the random performance across classes. The most important features stay roughly the same for this binary setup, with the topic count for the non-white class gaining importance.Bias Mitigation via Rewriting

[0098] As shown in the Bias Measure subsection above, even non-substantial bias can have a significant impact on the model performance. Also, there are substantial differences in textual distributions of notes for different demographic subgroups: in particular regarding the vocabulary used and the length of notes. State-of-the-art text generation techniques demonstrate an increased capacity to generate valid high quality outputs for science and medicine (Refs. 36-38). Therefore, rewnting clinical notes using generative techniques will help mitigate those differences while preserving the content of the notes.

[0099] The Pegasus model (Ref. 39) was used to paraphrase notes in Bin 6 data (see subsection Text Rewriting Models in the Methods section below). To protect the confidentiality, all data were installed on local air-gaped systems and rewritten locally using a locally installed version of Pegasus.

[0100] All notes per patient were used as input to the paraphrasing model. Paraphrasing notes of maximum length will reduce the length of the textual examples, which may result in drastic information loss. See an example of a hypothetical note in Table 8 where the original text length of 224 words was reduced to 177 words.

[0101] Statistics of textual features for Bin 6 texts after rew riting are presented in Table 9. Variations in those features across different demographic groups are generally maintained and intensified compared to the original texts without rewriting. For example, while the rewritten notes of the Black / Afro-American subgroup are roughly the same length as the notes for the White subgroup, notes for the Other subgroups are approximately 370 tokens shorter (this difference w as around 100 w ords for the original notes). The rew ritten notes of the non-white subgroups are more bursty (+0.03. compare to +0.01 before rewriting). There are also 4 times fewer topics in the rewritten minority subgroup notes than in the majority ones (the reduction was roughly 2.5 times for the original notes).

[0102] It is worth noting that the vocabulary' overlap differences, though, stayed the same (with the exception of the vocabulary overlap for the Black / Afro-American subgroup where the gap to the White subgroup reduced twice from 60% difference to 30%). As the differences in length and topics grow, an increase in the overlap is expected. The fact that those overlap differences stay the same or reduce suggests that our language unification procedure was successful and the rewriten notes tend to the use the same vocabulary’.

[0103] Table 8. Example of a hypothetical patient history and its rewriting by the Pegasus model. The rewritten text has 177 words instead of 224 words in the original text.

[0104] Table 9 Class statistics and properties of rewriten textual notes across demographic subgroups for Bin 6 test set. Vocabulary overlap is measured with respect to the vocabulary of the original text. Topics are extracted using the BERTopic built for original texts.

[0105] Table 10 shows the results of the experiments where we take different types of text notes to perform classification and observe how this influences performance and bias. Using rewriten notes instead of original ones does not result in significant performance changesacross subgroups (note that using all notes may cause a significant performance drop, as for the gender subgroup). Regarding the BER ratios, no statistically significant changes were observed.

[0106] A larger sample of test sets may better showcase the benefits of rewriting. Therefore, 300 test sets per demographic setup were sampled from all the examples in Bin 6. There was focus again on the important white bias cases. For race bias, 59% out of 300 samples (178 samples) are biased towards the White subgroup. For gender bias, 74% out of 300 samples (223 samples) are biased towards the white Male subgroup. Statistics over BER ratios for these white bias cases are in Table 11. It can be seen that the rewriting procedure significantly reduces white bias across subgroups. The reduction is 10% for the gender subgroups and 3% for the race subgroups.

[0107] Figs. 9 and 10 show the distribution of bias types in the sample for the Male / Female subgroup before and after rewriting. Note that the size of the No Bias class increased from 25% to 73% after rewriting, and the size of the White Bias class reduced from 74% to 22%.

[0108] Table 10. Classification parity results across type of text notes in Bin 6 for Clinical- BigBird trained on Bin 5. This investigates the performance when choosing only the note of a maximum length, concatenating all the notes, or rewriting all the notes. * marks statistically significant differences in Fl macro scores to the maximum length note setup as measured by the 2S-K.S test. Results per bin are averaged over 10 test sets sampled from all the bin data respecting the gender balance for this bin.

[0109] Table 11 Statistics over BER ratios before and after rewriting for the white bias test sets (178 test sets for race subgroups, 223 for the gender subgroups) 300 test set sampled from Bin 6. * marks a statistically significant difference in BER ratios to original texts before rewriting as measured by the 2S-KS test. Bold highlights a statistically significant decrease.

[0110] Fig. 9 shows bias type statistics for the Gender subgroup, Bin 6 before rewriting. The statistics for 300 test sets sampled from Bin 6 data are reported while respecting the race demographic group statistic.

[0111] Fig. 10 shows BER ratios for the Gender subgroup, Bin 6 after rewriting. Ratios over 300 test sets sampled from Bin 6 data are reported while respecting the race demographic group statistic. Ratios are computed using the predictions from the model trained with Bin 5 data.Data Cleaning

[0112] The initial pre-processing of textual notes involved removing special characters, HTML tags, quotes, and brackets, and replacing periods, tabs, and specialized line breaks with spaces.

[0113] To remove duplicates in progress notes and telephone encounters, the notes were first tokenized by removing punctuation and stopwords using the NLTK toolkit (Ref. 40). The notes were then vectorized using the CountV ectoriser from the Scikit-leam toolkit (Ref. 41). Finally, duplicate notes with > 0.8 cosine similarity were removed. After removing duplicates, the 25 most recent notes in each patient’s history (average minimum count of notes per patient across the bins) were selected.Topic Models

[0114] To estimate the number of topics in the notes, the state-of-the-art BERTopic (Ref. 31) methodology was used. This methodology transforms text into numerical representations using the Transformer-based BERT mode (Refs. 31, 42). Transformers (Ref. 43) are deep learning models able to efficiently capture long-range dependencies in text. The BERT version used for the topic model (S- PubMedBert-MS-MARCO) was pre-trained on in-domain biomedical data. Standard clustering techniques are used to cluster text representations. At the next step, topic keywords are extracted from clusters using the standard TF-IDF approach. The clusters are then refined to maintain relevant groups of keywords.Anxiety Prediction Models

[0115] An exemplar ’ anxiety prediction model was built by fine-tuning the state-of-the-art Transformer-based Clinical-BigBird model (Ref. 23) as imported from HuggingF ace (Ref. 24). This model is not only pre-trained on clinical text but can also handle long input sequences (up to 4K tokens).

[0116] The best practices in the domain and fine-tuned the model for 3 epochs (we noticed no further improvement for subsequent epochs) were followed. The input length of notes was limited to 1,000 tokens. AdamW optimizer (Ref. 25) was used with a learning rate of le-5 andan epsilon of le-7, and a batch size of 16. To compute class probability baseline the DummyClassifier with the stratified strategy’ from the Scikit-leam toolkit was used.Text Rewriting Models

[0117] For an exemplary rewriting clinical text stage, the Pegasus Transformer-based model pre-trained for text summarization and fine-tuned for paraphrasing as provided by the HuggingFace library was used. The model was used without additional fine-tuning and with the following decoding parameters: max length of 60 and greedy decoding with a temperature of 0.7.Bias Prediction Models

[0118] For the exemplary' prediction of bias, the Random Forests decision tree model (Ref. 39) was used. Decision trees are machine learning models that make predictions based on a sequence of if-else statements. Random Forests are an ensemble of decision trees, where each tree is built using a random subset of the training data and a random subset of features. Random Forests are hence less prone to overfitting as compared to single decision trees. The implementation provided by the Scikit-leam toolkit was used. Grid search with 5-fold cross- validation was used to tune the following parameters to optimize accuracy: the number of estimators and the maximum depth. All other parameters are those provided by default.EXAMPLE 2Datasets

[0119] In another example study, a foundational database was created consisting of EHR data from the Cincinnati Children’s Hospital Medical Center Epic Link, an online platform that allows providers real-time access to patient charts and results. This database has approximately 1.3 million unique patients seen at CCHMC between January 1, 2009, and March 31, 2022, with 63 million clinical notes. Patients diagnosed with at least one ICD 9 / 10 code related to an anxiety disorder were included in this study as patients with anxiety (refer to the Table 18 for a complete list of anxiety-related ICD 9 / 10 codes).

[0120] In summary', there were 1,383,145 total patients in the CCHMC EHR, 84,426 total anxiety cases that pass our selection criteria, 77,187 total anxiety cases with at least 1 note, 73.288 total anxiety cases with at least I note >30 days before their first anxiety diagnosis, and 7,810,849 notes for these 73k patients. This cohort contains various demographic data (sex and race) and the timeline of clinical textual notes provided by different care providers (progress notes, telephone encounters, plan of care notes, patient instructions, etc.).

[0121] After applying an additional selection criteria on note ty pes (selecting Progress Notes and Telephone Encounters as deemed the most informative by internal experts), this is reduced further to 4.3 million notes.

[0122] Bins of age 5, 8, 10, 12 and 15 were used (see the Age Binning subsection below). This selection of bins is not common for pediatrics but gives a comprehensive selection of datasets with diverse percentages of female patients varying from 36% to 69% (see section Descriptive Analysis in Results)

[0123] These data were initially cleaned as described in the Data Cleaning subsection below. The timelines were kept with at least one record and maintained the 1: 1 proportion of cases and controls.

[0124] Each final bin contained the training set of ~3,700-5,064 cases and controls, and the testing set included ~ 852-1, 278 cases and controls.Age binning

[0125] For this example, anxiety patients are defined as any patients who have ever received any of the diagnostic codes listed in Table 18. Additional selection criteria are that the patient had at least one encounter in the EHR in the 18 months prior to the anxiety diagnosis. The patients were grouped into age ranges by the age (in years) at the time of their anxiety diagnosis. For example, the 5-year-old age group consists of patients who received an anxietydiagnosis between their fifth and sixth birthdays. The controls were matched one-to-one with cases by age and sex. Matched controls 1) were bom within 30 of the case and 2) were of the same sex. Additional criteria for controls were that they had never received one of the anxiety diagnoses at the time of the matched case’s anxiety diagnosis and that they had had at least one encounter in the EHR within the 18-month window prior to the case’s anxiety- diagnosis. Only data prior to the first anxiety diagnosis are used for our analysis.Data Cleaning

[0126] For this example, to remove duplicates in progress notes and telephone encounters, notes were tokenized by removing punctuation and stopwords using the NLTK toolkit (Ref. 40). The notes were then vectorized using the CountVectoriser from the Scikit-leam toolkit (Ref. 41). Finally, duplicate notes with cosine similarity- > 0.8 were removed. Finally, the 25 most recent notes were selected in each patient’s history- (average minimum count of notes per patient across the bins). These notes are concatenated together per patient resulting in the datasets as described in the Datasets subsection above.Descriptive Analysis

[0127] This example analyzed primarily the sex subgroups of the dataset and the properties of the relevant texts. In particular, this example considered the following characteristics of textual distributions commonly applied in the NLP domain:

[0128] 1. Average length of a patient note in words (without tokenization, i.e., separation of punctuation). Concatenation was measured for all the valid notes per patient.

[0129] 2. Percentage of medical terms, in each note on average. Biomedical named entities were extracted using the state-of-the-art Stanza tool (Ref. 44). Those in-domain entities are extracted for ten standard categories such as Observation. Treatment, Anatomy, Procedure, etc.

[0130] 3. Percentage of gender-biased words, in each patient note on average. Using the best practices from NLP (Ref. 21), all the proper nouns (first and last names) were extracted using an off-the-shelf tool (Ref. 28), as well as gender pronouns (he, she, his, her, him, hers). This example focuses on this semantic group as the most relevant to cohort design involving matching by age and sex.

[0131] 4. Jaccard distance measures the similarity between two vocabularies by estimating the portion of common words. It is computed by dividing the number of words shared by both vocabularies by the total number of words in both vocabularies combined. Vocabularies for male / female were compared.

[0132] 5. Familiarity score assesses the ratio of common words to unique words. It is computed dividing the proportion of words which occur in both sets of words by the uniqueness ratio, the ratio of words that occur only in one of the sets. Again, this example mainly compared vocabularies male / female.

[0133] The results of this example analysis across bins are presented in Table 12 and across demographic groups within each bin in Table 13.

[0134] Observations of this example indicate notable changes in demographics and text properties across age bands (seen in Table 12). The ratio of females is consistently growing (from 36% for Bin 5 to 69% for Bin 15), while the percentage of the other races is stable (around 30%). The average concatenated note length remains constant at around 4K tokens.

[0135] The differences between demographic groups within the bins are clear (see Table 13). The percentage of cases is always 50% across sex subgroups, which is predefined by the cohort design. This percentage is typically lower for the non-privileged race subgroup as race was not factored into the cohort design (43% on average for other races). There are also obvious differences in the length of the notes. Notes for the male subgroup are on average ~540 words longer (~180 words longer for white patients).

[0136] The word distributions for male / female show the similarity of 0.54 Jaccard on average (average Familiarity 2.4). The lowest similarity’ for male / female is observed for Bins 5 and 15 (average Jaccard 0.43, Familiarity index 1.75). The percentages of terms and biased words for male / female are roughly the same. The distributions of terms male / female are even more dissimilar (average Jaccard 0.34, average Familiarity 1.5). Relatively low similarity values for word distributions are also observed across race subgroups (average Jaccard index 0.34. average Familiarity index 1.52). Though seen that other races make only 30% of the patients those values are not directly comparable to the ones for sex subgroups.

[0137] Given that the female subgroup is well-represented (49% of the examples on average), the differences observed cannot be explained by scarcity and suggest qualitative differences in the content of the notes across demographic subgroups (as evidenced by rather low similarity scores). The volume of diagnostic and gender-biased content remains constant across sex groups. This suggests that the differences in the length of the notes across subgroups are caused by other than diagnostic content and may be filtered. These content differences maybe caused by the fact that notes of female / male patients tend to come from different care sites following different reporting standards. For example, the male notes in bin 5 come from more than 400 care sites, while the female notes come from 337 sites. The female notes come mostly from General, Developmental and Behavioral Pediatrics, while male notes very’ often come from more specialized departments such as Neurology and Gastroenterology.

[0138] In summary, the example cohort matching procedure considering age / sex ensures there is equivalent representation of each between cases / controls. However, statistical differences in textual distributions for sex subgroups persist. Note that those differences are naturally occurring. They have been demonstrated to be crucial for classifier performance (Ref. 24). They are difficult to control (it is difficult to control lengths of notes across care sites or choose only the records without gender-related words or names), but could be more easily manipulated post-hoc. The current disclosure provides two techniques like this for bias mitigation in subsection Bias Mitigation below.

[0139] Table 12. Demographic statistics and properties of textual notes across Age bins (measured for the training data). Average values are reported per concatenated patient note.

[0140] Table 13. Class statistics and properties of textual notes across sex and race demographic subgroups in Bins. Average values per concatenated patient note are reported. M indicates Males, F indicates Females, W and O indicate White and Other races respectively. Distributional similarities (Jaccard and Familiarity indices) are reported between demographic subgroups within each bin.

[0141] Table 14. Classification parity results for models trained on original and de-biased notes. Accuracy (acc) and the percentage of uncertain predictions (unc), percentage of model predictions are reported with probabilities from the borderline probability zone [0.4, 0.6], Bias Measure

[0142] Best practices in Machine Learning (Ref. 8) measure model bias in terms of classification parity' (equality of classification error between groups), anti-classification (effect of protected attributes on predictions) and calibration (difference between predicted risk and factual risk).

[0143] In this work with predictive models, a focus is on classification parity. Several metrics are commonly used in the fairness literature to evaluate classification parity. These are accuracy equality (equal accuracy between groups), equal opportunity (equal false negative rate (FNR), orientation towards recall), or predictive equality (equal false positive rate (FPR) between groups, orientation towards precision) (Refs. 33, 34). Following Feldman et al., the balanced error rate (BER) is used, which is the unweighted average of FPR (precision) and FNR (recall):Eq. 1 where FP is the number of false positives, FN is the number of false negatives, TN is the number of true negatives and TP is the number of true positives. BER can be considered to be a measure of model bias using classification parity (equality of classification error between groups).

[0144] To assess bias, the ratio of BER scores is computed comparing the non-privileged subgroup to the privileged subgroup. Hence, a ratio of >1 hints towards better performance for the privileged subgroup measured by BER, while a ratio <1 indicates that a model performs better for the non-privileged group. The ratio of >1.25 indicates significant bias towards the privileged subgroup measured by BER, while a ratio of <0.85 indicates that a model performs significantly better for the non-privileged group.

[0145] FPR, FNR and BER ratio is used to assess the bias of five state-of-the-art Transformer-based Clinical-BigBird27 prediction models built using Bins 5, 8, 10, 12 and 15 with IK tokens input sequence length (see subsection Anxiety Prediction Models in Methods for more details). Results are shown in Table 14 across bins and Tables 15 and 16 across demographic subgroups.

[0146] Anti-classification measures are also suitable to assess textual bias. For example, predictive models can erroneously rely on biased words (e.g., relying on the pronoun ’she" indicating gender to predict the increased probability of depression or on the word "teacher ’ carrying the semantic of a female in its pre-trained representation (Ref. 32)). In this example, state-of-the-art interpretability analysis is used to verity’ if the models erroneously rely ongender words (first and last names, as well as gender pronouns). For the purposes of this disclosure, this reliance is considered to be textual bias.

[0147] Overall performance of the example models is reported in Table 14. The model achieves the performance of 0.61 accuracy on average across test data (this performance is above the random 0.50 accuracy). The percentage of uncertain predictions may be an indicator of the model’s confidence in its decisions. Standard Al predictive models issue a probability distribution over their outcomes. If the predicted probability is >0.5, the instance is classified as the positive class. Uncertain predictions (from the borderline probability zone [0.4, 0.6]) are less stable and reliable. If the percentage of such predictions is high a model can hardly be considered useful.

[0148] Looking closer at the bias within demographic subgroups using BER, Bin 10 model is slightly more biased towards male patients with the BER ratio of 1.33 (see Table 15). BER ratios however do not tell the whole story. Looking at FNR scores (equal opportunity), it can be seen that female patients get consistently under-diagnosed (+0.09 FNR average higher for females than for males; an increase of +0.13 is observed for bins 5 and 15). This trend is observed across cohorts and is consistent even if the percentage of female patients changes. FPR values do not exhibit such a systematic increase for males or females and stays roughly the same. Also, the percentage of uncertain predictions is systematically higher for females than for males (+5%) confirming that our models “hesitate’’ more for female patients. Finally, accuracy differences exhibit a decrease of -0.04 point on average for female patients.

[0149] In summary, FNR (equal opportunity) is the most expressive measure revealing relevant predictive bias in the example anxiety models. It correlates well with differences in textual statistics from the Descriptive Analysis subsection. Bias is most pronounced for female patients in cohorts 5 and 15, where the similarities in textual distributions are the lowest.

[0150] For the anti-classification analysis (reliance on biased words for prediction), the top ten examples with highest model confidence (most useful predictions) were checked in a case or a control prediction (20 examples per age bin, 100 examples in total across bins). We extract five words per example that influence the model decision the most and then collate them into a frequency vocabulary per predicted class. The results of this analysis demonstrate that the models mostly rely on relevant words both for cases and controls across low and high confidence groups (Table 17). However, there is some tendency to rely on irrelevant gender- biased words: for example, for cases 10% from the collated dictionary is biased.

[0151] Note that similar tendencies are observed across race subgroups: An average increase in FNR of +0.05 is observed for the non-privileged race group. FPR values exhibit anincrease towards the privileged class (+0.05 FPR). Also, a decrease of -0.03 in accuracy is observed for other races.

[0152] Table 15 Classification parity results across sex groups for models trained with original and de-biased data. The table reports accuracy (Acc), percentage of uncertain predictions (unc) (probability in [0.4, 0.6]), False Positive Rate (FPR), False Negative Rate (FNR) and Balanced Error Rates (BER). The symbol “f highlights higher values of FNR for non-privileged group. The symbol “J” highlights reduction in FNR to benefit the nonprivileged group. Bold highlights best results. The table also highlights the BER ratio indicating bias and its improvement.

[0153] Table 16. Classification parity results across race groups for models trained with original and de-biased data. The table reports accuracy (Acc), percentage of uncertain predictions (unc) (probability in [0.4, 0.6]), False Positive Rate (FPR), False Negative Rate (FNR) and Balanced Error Rates (BER). The “f ” symbol highlights higher values of FNR for non-privileged group. The “J” symbol highlights reduction in FNR to benefit the nonprivileged group. The table also highlights the BER ratio indicating bias and its improvement.

[0154] Table 17. Top-10 most influential words and Top-10 most frequent words as extracted by the state-of-the-art interpretability LIME tool (Ref. 26) from 100 examples across bins 5, 8, 10, 12 and 15. Ten examples from each set of predictions were analyzed with the highest confidence in case or control outcomes (20 examples per bin, 100 examples in total). The five most important word per example were extracted and collated into a frequency vocabulary of unique words per confidence-outcome group. Their importance scores were averaged. The table also reports the percentage gender-biased words observed in each vocabulary, as well as their average frequency.Bias Mitigation

[0155] As can be seen in the Bias Measure subsection above, bias can have an impact on the model performance. It is also shown that there are considerable differences in information density and textual distributions of notes for sex subgroups, in particular, those differences naturally involve gender words which the models can erroneously rely on. According to the current disclosure, therefore, by eliminating less relevant sentences tf-idf Jilt) density' in the notes can be balanced to reduce bias in the models. Regarding gender words, it is shown that male and female notes contain similar percentages of such words. Hence the current disclosure also includes an approach to neutralize those words via substitution with their gender-neural versions rather than their removal (gen sub, see subsection Text De-biasing Methods in Methodology).

[0156] In an example, the de-biasing methods as described in the Methodology section below were applied to modify the training data. In particular, the rnd Jilt method was applied. Removing 20% of sentences at random was set as the baseline. This baseline was compared to the performance of the disclosed approach tf-idfjilt which removes 20% sentences according to their informativeness score. The gen sub method was also applied on the original notes, as well as on the notes filtered with tf-idfjilt to trace the effect of both models combined. Each time the training data was modified, the models were re-trained to obtain new models, which were then tested on the original test data.

[0157] The results show that the de-biased models in general maintain the performance of the original models (Table 14): both yw? sub and rnd Jilt slightly increase the performance by +0.5 accuracy, while tf-idfjilt decreases the performance on average by -1 accuracy. The mixed method tf-idf Jilt+gen sub maintains the original performance with negligible changes. While maintaining performance, our de-biasing methods exhibit positive influence on the performance by reducing the percentage of uncertain predictions by -8% on average with the highest reduction for the tf-idf Jilt+gen sub of -12 %.

[0158] In terms of the reduction of bias as measured by FNR, tf-idfjilt is a clear winner (see Table 15). It outperforms the random sentence removal baseline rnd Jilt, reducing the FNR gap by -0.024 (27%, initial average gap 0.09) point on average with the highest reduction by -0.1 1 point for bin 5 (from 0.13 to 0.02) and -0.06 point for bin 15 (from 0.13 to 0.07). md_filt baseline does not exhibit any consistent behayior and does not influence the gap across bins. The gen sub approach yvhen applied alone is not efficient to reduce the FNR gap as well. On average a small increase was observed in this gap rather than a decrease (+0.008). tf- idf fllt+ gen sub has roughly the same performance as tf-idf filt with the average decrease of - 0.022 point. Both tf-idfjilt and tf-idf Jilt+gen sub approaches manage to maintain the BERratios within the acceptable level, and even decrease the BER ratio for Bin 10 from 1.33 to the acceptable values of 1.24 and 0.98 for tf-idfjilt and tf-idfjilt+ gen sub respectively, tf-idf ilt reduces the increase in uncertain predictions for females by 50%, while tf-idfjilt-gen sub fully eliminates this dis-balance.

[0159] tf-idf Jilt + gen sub has also capacity to mitigate bias for race subgroups reducing the FNR gap by -0.034 point on average. For bin 10, the bias is also reduced towards the nonprivileged class (from BER ratio 0.74 to the acceptable level of 0.95).

[0160] A positive effect of the exemplary de-biasing methods was observed in terms of the words the classifiers rely on (anti-classification, see Table 17)). A first observation is that both tf-idf ilt and tf-idf ilt+gen_sub improve the generalizability of the models. Fewer terms appear in the statistics signifying more reliance on context (e.g., words “complaint7’, “presents”, “no”) while making predictions rather than overfitting to keywords (e.g., “anxiety”, “depression”, etc.). Furthermore, the example de-biasing techniques reduce the percentage of biased words the models rely on: for example, tf-idf ilt reduces this percentage from 10% to 3% for cases, from 4% to 2% for controls. tf-idf ilt+gen_sub further reduces twice the frequency of biased words tf-idf dt relies on. This is a positive effect since those residual biased associations become less systematic as compared to tf-idf ilt.

[0161] The computing engines, modules, machine learning modules, machine learning engines, deep learning modules / engines, training systems, architectures and other disclosed functions are embodied as computer instructions that may be installed for running on one or more computer devices and / or computer servers. In some instances, a local user can connect directly to the system; in other instances, a remote user can connect to the system via a network.

[0162] Example networks can include one or more types of communication networks. For example communication networks can include (without limitation), the Internet, a local area network (LAN), a wide area network (WAN), various types of telephone networks, and other suitable mobile or cellular network technologies, or any combination thereof. Communication within the network can be realized through any suitable connection (including wired or wireless) and communication technology or standard (wireless fidelity’ (WiFi®), 4G, 5G, longterm evolution (LTE™)), and the like as the standards develop.

[0163] The computer device(s) and / or computer server(s) can be configured with one or more computer processors and a computer memory (including transitory computer memory and / or non-transitory computer memory), configured to perform various data processing operations. The computer device(s) and / or computer server(s) also include a networkcommunication interface to connect to the network(s) and other suitable electronic components.

[0164] Example local and / or remote user devices can include a personal computer, portable computer, smartphone, tablet, notepad, dedicated server computer devices, any type of communication device, and / or other suitable compute devices.

[0165] The computer device(s) and / or computer server(s) can include one or more computer processors and computer memories (including transitory computer memory and / or non-transitory computer memory), which are configured to perform various data processing and communication operations associated with diagnosing psychiatric disorders as disclosed herein based upon information obtained / provided (such as the EHR data discussed above) over the network, from a user and / or from a storage device. In some implementations, storage device can be physically integrated to the computer device(s) and / or computer server(s); in other implementations, storage device can be a repository such as a Network- Attached Storage (NAS) device, an array of hard-disks, a storage server or other suitable repository7separate from the computer device(s) and / or computer server(s).

[0166] In some instances, storage device can include the machine-learning models / engines and other software engines or modules as described herein. Storage device can also include sets of computer executable instructions to perform some or all the operations described herein.DEFINITIONS

[0167] Unless otherwise noted, terms will be understood according to conventional usage by those of ordinary skill in the relevant art. The present document, including definitions, will control in case of conflict. Preferred methods and materials are described below, although methods and materials similar or equivalent to those described herein may be used in practice or testing of the present invention. All publications, patent applications, patents, and other references mentioned herein are incorporated by reference in their entirety. The materials, methods, and examples disclosed herein are illustrative only and not intended to be limiting.

[0168] As used herein and in the appended claims, the singular forms "a," "and," and "the" include plural referents unless the context dictates otherwise. Thus, for example, reference to "a method" includes one or more such methods, and reference to "a machine learning model" includes reference to one or more machine learning models and equivalents thereof known to those skilled in the art, and so forth.

[0169] The term "about" or "approximately" means within an acceptable error range for the particular value as determined by one of ordinary skill in the art, which will depend in part on how the value is measured or determined, e.g., the limitations of the measurement system.For example, "about" may mean within one or more than one standard deviation, per the practice in the art. Alternatively, "about" may mean a range of up to 20%, or up to 10%, or up to 5%, or up to 1% of a given value. Alternatively, particularly with respect to biological systems or processes, the term may mean within an order of magnitude, preferably within 5- fold, and more preferably within 2-fold, of a value. Where particular values are described in the application and claims unless otherwise stated, the term "about," meaning within an acceptable error range for the particular value, should be assumed.

[0170] As used herein, the term "effective amount" means the amount of one or more active components that is sufficient to show a desired effect. This includes both therapeutic and prophylactic effects. When applied to an individual active ingredient, administered alone, the term refers to that ingredient alone. When applied to a combination, the term refers to combined amounts of the active ingredients that result in the therapeutic effect, whether administered in combination, serially, or simultaneously.

[0171] The terms "individual," "host," "subject," and "patient" are used interchangeably to refer to the human that is the object of treatment, observation, and / or experiment. Generally, the term refers to a human patient. In some embodiments, the terms may refer to children.

[0172] As used herein, "based on" should be understood to mean that a thing is determined, at least in part, by that which it is indicated as being "based on." It should be understood that a statement that something is "based on" something else does not necessarily require one thing to be fully determined by the other. If one thing is required to be fully determined by another, this may be indicated by stating that it is "based EXCLUSIVELY on" that which it is determined by.

[0173] While certain embodiments of the present disclosure have been illustrated and described, it will be apparent to those skilled in the art that various other changes and modifications may be made without departing from the spirit and scope of the inventions as claimed.

[0174] Table 18 The list of diagnosis codes used to define an anxiety diagnosis for this disclosure. Cincinnati Children's Hospital Medical Center switched from the International Classification of Diseases (ICD) version 9 to version 10 on October 1, 2015.

[0175] It will be apparent to those of skill in the art that while the current disclosure focuses primarily on artificial intelligence and machine learning engines for diagnosing or predicting a patient's psychiatric condition or predicting events related to a patient’s psychiatric condition, the current disclosure can be applied to diagnosing or predicting other patient health conditions (or predicting an event related to another patient health condition). As such, the current disclosure may apply to mitigating bias in AI / ML training data for diagnosing or predicting health conditions (such as medical conditions, behavioral or psychiatric conditions, or a combination of medical with behavior and / or psychiatric conditions ) and related events (suchas hospital or emergency room visits, suicide attempts, episodes of violence, organ failures, treatment failures, opioid addition, and the like).

[0176] The following references are cited in this disclosure and are incorporated herein by reference. The citation of any document is not an admission that it is prior art with respect to any invention disclosed or claimed herein or that it alone, or in any combination with any other reference or references, teaches, suggests, or discloses any such invention. Further, to the extent that any meaning or definition of a term in this document conflicts with any meaning or definition of the same term in a document incorporated by reference, the meaning or definition assigned to that term in this document shall govern.1. Topol. E. J. High-performance medicine: the convergence of human and artificial intelligence. Nat. Medicine 2019 25: 1 25, 44-56, DOI: 10. 1038 / s41591-018-0300-7 (2019).2. Zhang, T., Schoene, A. M., Ji, S. & Ananiadou, S. Natural language processing applied to mental illness detection: a narrative review, npj Digit. Medicine 2022 5: 1 5. 1-13, DOI: 10.1038 / s41746-022-00589-7 (2022).3. Coley, R. Y., Johnson, E., Simon, G. E., Cruz, M. & Shortreed, S. M. Racial / ethnic disparities in the performance of prediction models for death by suicide after mental health visits. JAMA Psychiatry 78, 726-734, DOI: 10.1001 / JAMAPSYCHIATRY.2021.0493 (2021 ).4. Loge, C. et al. Q-pain: A question answering dataset to measure social bias in pain management. DOI: 10. 13026 / 6 Ixq-mj 56 (2021).5. Zhao, J.. Wang, T., Yatskar, M., Ordonez, V. & Chang, K.-W. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Palmer, M., Hwa. R. & Riedel, S. (eds.) Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2979-2989, DOI: 10. 18653AT / D17-1323 (Association for Computational Linguistics, Copenhagen, Denmark, 2017).6. Angwin, L. J. M. S., J. & Kirchner, L. Machine bias: There’s software used across the country to predict future criminals, and it’s biased against blacks. | benton institute for broadband society. (2016).7. Chouldechova, A. & Roth, A. The frontiers of fairness in machine learning. (2018).8. Corbett-Davies, S. et al. The measure and mismeasure of fairness: A critical review of fair machine learning *. (2018).Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V. & Kalai, A. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In NeurlPS (2016). Tokpo, E. K., Delobelle, P., Berendt, B. & Calders, T. How far can it go? on intrinsic gender bias mitigation for text classification. In Vlachos, A. & Augenstein, I. (eds.) Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 3418-3433, DOI: 10. 18653 / vl / 2023.eacl-main.248 (Association for Computational Linguistics, Dubrovnik, Croatia, 2023). Raza, S., Garg, M., Reji, D. J., Bashir, S. R. & Ding, C. Nbias: A natural language processing framework for bias identification in text. Expert. Syst. withAppl. 237, 121542, DOI: 10. 1016 / J.ESWA.2023. 121542 (2024). Bellamy, R. K. et al. Ai fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias. IBM J. Res. Dev. 63, DOI: 10.1147 / JRD.2019.2942287 (2019). Roosli, E., Bozkurt, S. & Hemandez-Boussard, T. Peeking into a black box, the fairness and generalizability of a mimic-iii benchmarking model. Sci. Data 20229 1 9, 1-13, DOI: 10. 1038 / s41597-021-01110-7 (2022). Kessler, R. C. et al. Age of onset of mental disorders: a review of recent literature. Curr. opinion psychiatry 20, 359 (2007). Sorter. M. et al. Addressing the pediatric mental health crisis: Moving from a reactive to a proactive system of care. J. Pediatr. 0, DOI: 10. 1016 / j.jpeds.2023. 113479 (2023). DhaubhadeL S. et al. High dimensional predictions of suicide risk in 4.2 million us veterans using ensemble transfer learning. Sci. reports 14, 1793, DOI: 10.1038 / S41598- 024-51762-9 (2024). Machado, C. D. S. et al. Prediction of suicide attempts in a prospective cohort study with a nationally representative sample of the us population. Psychol, medicine 52, 2985- 2996, DOI: 10. 1017 / S0033291720004997 (2022). Behrens, B., Swetlitz, C., Pine, D. S. & Pagliaccio, D. The screen for child anxiety related emotional disorders (scared): Informant discrepancy, measurement invariance, and test-retest reliability. Child psychiatry human development 50. 473-482, DOI:10. 1007 / S 10578-018-0854-0 (2019). Tulisiak, A. K. et al. Antidepressant prescribing by pediatricians: A mixed-methods analysis. Curr. Probl. Pediatr. Adolesc. Heal. Care 47. 15-24, DOI:10. 1016 / J.CPPEDS.2016. 11.009 (2017).Strawn, J. R., Lu, L., Peris, T. S., Levine, A. & Walkup, J. T. Research review: Pediatric anxiety disorders - what have we learnt in the last 10 years? J. child psychology psychiatry, allied disciplines 62, 114-139, DOI: 10.1 111 / JCPP. 13262 (2021). Ballard, R. & Latimer, T. Time management strategies for addressing mental health concerns in the pediatric visit. Pediatr. Annals 52, e426-e429, DOI: 10.3928 / 19382359- 20230906-02 (2023). Li. Y.. Wehbe, R. M.. Ahmad, F. S., Wang, H. & Luo. Y. Clinical-longformer and clinical-bigbird: Transformers for long clinical sequences. arXiv preprint arXiv: 2201.11838 (2022). Wolf, T. et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 38-45, DOI: 10.18653 / vl / 2020.emnlp-demos.6 (Association for Computational Linguistics, Online, 2020). Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. 7th Int. Conf, on Learn. Represent. ICLR 2019 (2017). Ribeiro. M. T., Singh. S. & Guestrin. C. "why should i trust you?": Explaining the predictions of any classifier. 1135-1144, DOI: 10.1145 / 2939672.2939778 (Association for Computing Machinery’, 2016). Kiritchenko, S. & Mohammad, S. Examining gender and race bias in two hundred sentiment analysis systems. In Nissim, M., Berant, J. & Lenci. A. (eds.) Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, 43-53, DOI: 10.18653 / vl / S 18-2005 (Association for Computational Linguistics, New Orleans, Louisiana, 2018). Qi, P.. Zhang, Y., Zhang, Y., Bolton, J. & Manning, C. D. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (2020). Chan, S. C. Y. et al. Data distributional properties drive emergent in-context learning in transformers. (2022). Serrano, M. A., Flammini, M. & Menczer, A. Modeling statistical properties of written text. PLoS ONE 4, 5372, DOI: 10. 1371 / joumal.pone.0005372 (2009). Grootendorst, M. Bertopic: Neural topic modeling with a class-based tf-idf procedure. (2022).Kurita, K.. Vyas, N., Pareek, A., Black, A. W. & Tsvetkov, Y. Measuring bias in contextualized word representations. 166-172, DOI: 10.18653 / vl / wl9-3823 (2019). Feldman, M., Friedler, S., Moeller, J., Scheidegger, C. & Venkatasubramanian, S. Certifying and removing disparate impact. Proc. 21st ACM SIGKDD Int. Conf, on Know! Discov. Data Min. (2015). Chouldechova, A. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data 5, 153-163, DOI: 10.1089 / BIG.2016.0047 (2017). Breiman, L. Random forests. Mach. Learn. 45, 5-32 (2001). Ive, J. et al. Generation and evaluation of artificial mental health records for natural language processing, npj Digit. Medicine 2020 3:1 3. 1-9, DOI: 10. 1038 / s41746-020- 0267-x (2020). Khan, R. A., Jawaid, M., Khan, A. R. & Sajjad, M. Chatgpt - reshaping medical education and clinical management. Pak. J. Med. Sci. 39, 605, DOI:10. 12669 / PJMS.39.2.7653 (2023). Else, H. Abstracts written by chatgpt fool scientists. Nature 613, 423, DOI:10. 1038 / D41586-023-00056-7 (2023). Zhang, J., Zhao, Y., Saleh, M. & Liu, P. J. Pegasus: Pre-training with extracted gapsentences for abstractive summarization (2019). Bird, S., Klein, E. & Loper, E. Natural language processing with Python: analyzing text with the natural language toolkit (" O’Reilly Media, Inc.". 2009). Pedregosa, F. et al. Scikit-leam: Machine Learning in Python. J. Mach. Learn. Res. 12, 2825-2830 (2011). Deka, P., Jurek-Loughrey, A. & Deepak, P. Improved methods to aid unsupervised evidence-based fact checking for online health news. J. Data Intell. 3, 474-504 (2022). Vaswani, A. et al. Attention is all you need. vol. 30 (Curran Associates, Inc., 2017). Zhang, Y., Zhang, Y., Qi, P., Manning, C. D. & Langlotz, C. P. Biomedical and clinical English model packages for the Stanza Python NLP library . J. Am. Med. Informatics Assoc. (2021). Graham. S. et al. Artificial intelligence for mental health and mental illnesses: an overview. Curr. Psychiatry Reports 21, 1-18, DOI: 10.1007 / S 11920-019- 1094- O / TABLES / 2 (2019).

Claims

What is claimed is:1 . A method for training and preparing a machine learning model for use in diagnosing or predicting a psychiatric or behavioral condition, or an event related to a psychiatric or behavioral condition, comprising: collecting first electronic health records from previous patients who have been diagnosed with a psychiatric or behavioral condition or have experienced an event related to a psychiatric or behavioral condition and a second electronic health records from previous patients who have not been diagnosed with the psychiatric or behavioral condition or have not experienced an event related to a psychiatric or behavioral condition; processing the first and second collected electronic health records into a training dataset; and training a machine learning model with the training dataset; wherein the processing step includes re-writing at least a portion of the first or second electronic health records to mitigate bias in the training dataset.

2. The method of claim 1, wherein the re-writing step includes a step of information density filtering on the portion of the first or second electronic health records.

3. The method of claim 2, wherein the re-writing step includes a step of replacing biased words in the portion of the first or second electronic health records with relevant neutral versions of the biased words.

4. The method of claim 3, wherein the biased words are gender-specific words and the neutral versions of the biased words are gender-neutral counterparts to the gender-specific words.

5. The method of claim 4, wherein the gender-specific words and gender-neutral counterparts to the genders specific words include proper nouns.

6. The method of claim 4 or 5, wherein the gender-specific words and genderneutral counterparts to the genders specific words include pronouns.

7. The method of claims 2, 3, 4, 5 or 6, wherein the step of information density filtering on the portion of the first or second electronic health records includes: scoring salient words as a multiple the frequency of a word in a given health record with respect to the word’s rarity across other given health records, to provide a wordlevel TF-IDF score for each word in the given health record; computing an averaged sum of word-level TF-IDF scores for each sentence in the given health record; and removing a predetermined percentage of the lowest scoring sentences.

8. The method of claim 1, 2, 3, 4, 5, 6, or 7, wherein the processing step further includes organizing data from the first and second collected electronic health records into a set of time limited bins.

9. A method for training and preparing a machine learning model for use in diagnosing or predicting a psychiatric or behavioral condition, or an event related to a psychiatric or behavioral condition, comprising: collecting electronic health records from previous patients who have been diagnosed with a psychiatric or behavioral condition ; processing the collected electronic health records into a training dataset; andtraining a machine learning model with the training dataset; wherein the processing step includes detecting the presence of bias in the training dataset by, at least in part, analyzing textual distributions in the electronic health records.

10. The method of claim 9, wherein the step of analyzing textual distributions in the electronic health records includes two or more of: evaluating skew ness of vocabulary frequency distribution in data from the electronic health records; evaluating burstiness or one or more words in data from the electronic health records; determining an average length of a patient note in w ords from the electronic health records; determining unique w ord vocabulary overlap in a first subset of the electronic health records versus a second, larger subset of the electronic health records; determining a topic count in the first subset of the electronic health records; and determining a percentage of the first subset of the electronic health records that does not pertain to identified topic.

11. The method of claim 10, wherein the step of evaluating skewness includes detecting the presence of rare words in the data from the electronic health records.

12. The method of claims 10 or 1 1, wherein the step of evaluating burstiness includes calculating an average probability that the one or more words will appear in the data from the electronic health records.

13. The method of claims 10, 11 or 12, wherein the first subset of the electronic health records is a first time limited bin of the data from the electronic health records and the second, larger subset of the electronic health records is a plurality of time limited bins of the data from the electronic health records.

14. The method of claims 10, 11, 12, 13 or 14 further comprising the step of rewriting at least a portion of the electronic health records to mitigate detected bias in the training dataset.

15. A method for training and preparing a machine learning model for use in predicting bias in electronic health records, comprising: collecting a plurality of previous electronic health records; for each of the plurality of previous electronic health records, calculating a balanced error rate (BER); and training a machine learning model with the electronic health records and each corresponding calculated balanced error rate (BER), which is a measure of model bias using classification parity.

16. The method of claim 15, wherein the balanced error rate (BER) is calculated according to the following equation,FP FN )BER=( FP+TN )+( FN+TP2 where FP is the number of false positives, FN is the number of false negatives, TN is the number of true negatives and TP is the number of true positives.

17. The method of claims 15 or 16, further comprising inputting a new electronic health record to the machine learning model to predict a balance error rate (BER) for a new electronic health record.

18. The method of claims 17, further comprising the step of re-writing at least a portion of the new electronic health record if the predicted balance error rate (BER) exceeds a predetermined value.

19. A method for diagnosing or predicting a patient’s psychiatric or behavioral condition, or for predicting an event related to a patient’s psychiatric or behavioral condition, comprising: providing a machine learning engine trained according to claims 1, 2, 3, 4, 5, 6, 7 or 8; collecting a patient’s one or more electronic health records; processing the patient’s one or more electronic health records to generate input data for the machine learning engine; feeding the input data to the machine learning engine, and receiving from the machine learning engine (a) a diagnosis or a prediction of the patient’s a psychiatric or behavioral condition, or (b) a prediction of an event related to the patient’s psychiatric or behavioral condition.

20. A method for treating a patient’s psychiatric or behavior condition, comprising: according to the method of claim 19, (a) diagnosing or a predicting a patient’s a psychiatric or behavioral condition, or (b) predicting an event related to the patient’s psychiatric or behavioral condition; andproviding treatment to the patient.

21. The method of claim 20, wherein providing the treatment to the individual comprises treating the individual with cognitive behavioral therapy or another similar evidence based behavioral therapy.

22. The method of claims 20 or 21, wherein providing the treatment to the individual comprises treating the individual with a medication selected from: (a) a selective serotonin reuptake inhibitor; (b) serotonin-norepinephrine reuptake inhibitor; (c) benzodiazepines; (d) beta blockers; (e) tricyclic antidepressants; (f) monoamine oxidase inhibitors; (g) atypical antidepressants; (h) lithium; (i) antiepileptics; (j) first, second, or third generations antipsychotics; (k) stimulants; (1) non-stimulant, such as atomoxetine, clonidine, guanfacine, and / or modafinil; and (m) non-benzodiazepine anxiolytics.

23. A method for training and preparing a machine learning model for use in diagnosing or predicting a patient’s health condition, or an event related to a patient’s health condition, comprising: collecting first electronic health records from previous patients who have been diagnosed with a certain health condition or have experienced a event related to the certain health condition and a second electronic health records from previous patients who have not been diagnosed with the certain health condition or have not experienced and event related to the certain health condition; processing the first and second collected electronic health records into a training dataset; and training a machine learning model with the training dataset; wherein the processing step includes re-writing at least a portion of the first or second electronic health records to mitigate bias in the training dataset.

24. The method of claim 23, wherein the re-writing step includes a step of information density filtering on the portion of the first or second electronic health records.

25. The method of claim 24, wherein the re-writing step includes a step of replacing biased words in the portion of the first or second electronic health records with relevant neutral versions of the biased words.

26. The method of claim 25, wherein the biased words are gender-specific words and the neutral versions of the biased words are gender-neutral counterparts to the genderspecific words.

27. The method of claim 26, wherein the gender-specific words and gender-neutral counterparts to the genders specific words include proper nouns.

28. The method of claim 26 or 27, wherein the gender-specific words and genderneutral counterparts to the genders specific words include pronouns.

29. The method of claims 24, 25, 26, 27 or 28, wherein the step of information density filtering on the portion of the first or second electronic health records includes: scoring salient words as a multiple the frequency of a word in a given health record with respect to the word’s its rarity across other given health records, to provide a wordlevel TF-IDF score for each word in the given health record; computing an averaged sum of word-level TF-IDF scores for each sentence in the given health record; and removing a predetermined percentage of the lowest scoring sentences.

30. The method of claim 23, 24, 25, 26, 27, 28, or 29, wherein the processing step further includes organizing data from the first and second collected electronic health records into a set of time limited bins.

31. A machine learning engine trained according to any of the methods of claims 1 - 30.

32. One or more non-transitory memory devices including computer instructions configured to direct one or more processors to perform any of the methods of claims 1-30.

Images