Health fair assessment system for clinical data products

Abstract

A confidence and health fairness assessment system (102, 214) is provided for an AI clinical data product (CDP (1102, 206, 175, 402)) that can be launched via a clinical workflow application (160, 212). When a CDP (904, 1102, 206, 175, 302, 402) is launched, a rating tool micro-application of the confidence and health fairness assessment system (102, 214) may be launched. When a condition is satisfied, the rating tool micro-application may request a user for clinical feedback regarding the balance, suitability, suitability, accuracy, or quality of the output of the CDP (904, 1102, 206, 175, 302, 402). The clinical feedback may include demographic data regarding various difference factors of the patient population analyzed by the CDP (904, 1102, 206, 175, 302, 402). After aggregating clinical feedback, the confidence and health fairness assessment system (102, 214) performs a health fairness assessment (408) of the CDP (904, 1102, 206, 175, 302, 402) and generates a confidence rating for the CDP (904, 1102, 206, 175, 302, 402), where the confidence rating is a multi-dimensional score indicative of a degree of confidence that the output of the CDP is not affected by algorithm deviations for various difference factors.

Description

Technical Field

[0001] Embodiments of the subject matter disclosed herein generally relate to AI-based clinical tools and, more particularly, to identifying bias in AI algorithms used in such tools. Background Art

[0002] Clinical decision support (CDS) tools that rely on artificial intelligence (AI) models and are deployed by medical device / product manufacturers are supported by clinical studies for regulatory approval. However, these clinical studies may not adequately cover patient populations in terms of geography, race, age, gender, various disease conditions, clinical study duration, etc. Lack of sufficient coverage may lead to bias in the AI algorithms used by the CDS tools, where the results generated by the CDS tools may have higher quality for some parts of the patient population and lower quality for other parts of the patient population. Detecting and monitoring bias in AI algorithms at scale is a challenging problem for which there are few technical solutions. In some methods, clinical experts can flag incorrect or incomplete AI insights, notifications, warnings, alerts, or recommendations. However, analyzing the health equity impact on a broad patient cohort and labeling (re)training data for the AI model can be both time-consuming and daunting, especially for already overburdened clinicians and nurses. Summary of the Invention

[0003] In one example, the present disclosure addresses the above problems using a health equity assessment system that includes a processor and a non-transitory memory storing instructions that, when executed, cause the processor to summarize the clinical context of an analysis performed on patient data by a clinical data product (CDP); perform a health equity assessment of the CDP based on the summarized clinical context using one or more differential assessment models trained to evaluate algorithmic bias of the CDP for a particular patient population; calculate a confidence score for the CDP based on the health equity assessment, the confidence score indicating the degree of confidence that patient analysis using the CDP is not subject to algorithmic bias; and store the confidence score in a database.

[0004] The above advantages, as well as other advantages and features, will be apparent from the following detailed description when considered in conjunction with the drawings. It should be understood that the above summary is provided to introduce in a simplified form a series of concepts that are further described in the detailed description. This is not meant to identify the key features or essential features of the claimed subject matter, the scope of which is uniquely defined by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to implementations that solve any disadvantages noted above or in any part of this disclosure. Brief Description of the Drawings

[0005] Aspects of the present disclosure may be better understood by reading the following detailed description and with reference to the accompanying drawings, in which:

[0006] Figure 1 A schematic block diagram of a health equity assessment system according to one or more embodiments of the present disclosure is shown;

[0007] Figure 2 A high-level schematic block diagram illustrating a workflow for generating a health equity assessment of a CDP according to one or more embodiments of the present disclosure is shown;

[0008] Figure 3 A schematic block diagram illustrating the data flow through various data ports of a CDP within the framework of a health equity assessment system according to one or more embodiments of the present disclosure is shown;

[0009] Figure 4 A schematic block diagram illustrating a CDP market according to one or more embodiments of the present disclosure is shown;

[0010] Figure 5 A flowchart showing an exemplary method for collecting ratings of CDPs selected by clinicians for use by a health equity assessment system according to one or more embodiments of the present disclosure;

[0011] Figure 6 A flowchart showing an exemplary method for generating a health equity assessment of a CDP using a health equity assessment system according to one or more embodiments of the present disclosure;

[0012] Figure 7 A flowchart showing an exemplary method for displaying health equity assessment data related to various CDPs according to one or more embodiments of the present disclosure;

[0013] Figure 8 A flowchart showing an exemplary method for determining drift in a CDP model according to one or more embodiments of the present disclosure;

[0014] Figure 9 An exemplary CDP for outputting a warning based on patient data according to one or more embodiments of the present disclosure is shown;

[0015] Figure 10 A set of exemplary control elements that can be displayed in a clinical workflow application to collect ratings of a CDP according to one or more embodiments of the present disclosure is shown;

[0016] Figure 11is a first image of an exemplary graphical user interface (GUI) for a digital catalog of a CDP that includes health equity assessment data according to one or more embodiments of the present disclosure; and

[0017] Figure 12 is a second image of an exemplary GUI for a digital catalog of a CDP that includes health equity assessment data according to one or more embodiments of the present disclosure.

[0018] The accompanying drawings illustrate specific aspects of the systems and methods. Together with the following description, the drawings show and explain the structures, methods, and principles described herein. In the drawings, for clarity, the dimensions of components may be enlarged or otherwise modified. Well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the components, systems, and methods. Detailed Description

[0019] Care providers within a healthcare system may use a clinical data product (CDP) to support clinical decision-making, identify patterns in patient data or patient population data, provide treatment recommendations, and / or other clinical tasks. A CDP is a digital asset created by a company or organization that aggregates and / or analyzes data to provide insights and knowledge to users. The CDP is one of the fundamental principles of what is referred to as "data as a product" from a data grid architecture. There are many formats in which data can be provided as a product, including APIs, data sets, data feeds, visualizations, reports, dashboards, notebooks, insights via email or messaging, AI-based tools (such as machine learning (ML) or deep learning (DL) models), rule-based algorithms, and the like. The goal of a CDP is to provide value to users by enabling them to make data-driven decisions or solve specific problems.

[0020] Health equity is a state in which everyone has a fair and reasonable opportunity to attain their highest level of health. While health equity initiatives seek to reduce health disparities and optimize the health of everyone, many organizations struggle to measure health equity progress, disparities, and fairness in how CDPs are used. Biased AI algorithms in healthcare can result in significant human costs and may include delayed or inaccurate diagnoses and treatments. Biased AI algorithms can further exacerbate existing healthcare disparities by disproportionately affecting certain patient groups. This can erode patients' trust in the healthcare system if patients feel that they are being treated unfairly or that their care is not being taken seriously. Healthcare organizations that use biased AI algorithms may also face legal and financial liabilities.

[0021] For example, a care provider may use an AI tool that includes a trained ML model to predict the effectiveness of a specific treatment for a patient. The ML model can be trained using training data from a training patient population defined by a set of characteristics. For example, the training patient population may mainly include white male patients. A patient may represent the training patient population (e.g., white male), and thus, the predicted effectiveness of the treatment output by the trained ML model may be accurate for the patient. However, the patient may not represent the training patient population (e.g., African American female). As a result of not representing the training patient population, the inferences of the ML model based on the patient data may be incorrect, and the predicted effectiveness of the treatment output by the trained ML model may be inaccurate for the patient. The result of such algorithmic bias is that an inappropriate treatment plan may be proposed for the patient, which may lead to adverse patient outcomes.

[0022] In practice, various examples of such algorithmic bias have been found. For example, it has been found that an AI model used to predict patient mortality in an intensive care unit (ICU) is less accurate for females than for males; it has been found that an AI model used to predict the risk of preeclampsia or eclampsia is less accurate for women of color; it has been found that an AI model used to identify patients at risk of developing sepsis is biased against African American patients; it has been found that an AI model used to predict the risk of patient readmission is less accurate for patients with a lower socioeconomic status (SES); it has been found that an AI model used to predict the risk of cardiovascular disease in patients is less accurate for patients with a mobility impairment; and others. It should also be understood that given that regulatory approval for CDP use is typically based on an evaluation of CDP trials conducted on a very limited group, it is generally expected that CDP has algorithmic bias and that a comprehensive assessment of algorithmic bias may not be possible until CDP is used in a broader patient population.

[0023] Several ways to measure health equity and bias in CDP use have been proposed, such as disparate impact analysis, sensitivity analysis, and monitoring algorithms after CDP is deployed. Disparate impact analysis involves comparing the results of AI algorithms for different populations of different races, genders, socioeconomic statuses, etc. to determine if there are statistically significant differences. Sensitivity analysis tests the robustness of the AI algorithm to data changes. For example, an AI algorithm used to predict the risk of readmission may be found to be sensitive to changes in a patient's socioeconomic status.

[0024] However, methods such as these may rely on collecting large amounts of patient outcome data linked to a specific AI tool or model (e.g., CDP), which can be cumbersome and time-consuming, if not infeasible. Patient data stored in hospital databases such as electronic medical record (EMR) systems may not include information about the tools clinicians used to make a diagnosis or treatment plan, which may be associated with patient outcomes. Alternatively, information about the tools or models used by clinicians may be included, but this information may be in a non-standardized format. For example, CDP may be referred to by a name within unstructured text (e.g., a report), where CDP may not be easily recognizable as an AI tool. Additionally, CDP may be referred to differently by different clinicians. For example, a first clinician may refer to CDP by an informal name (e.g., "patient mortality predictor"); a second clinician may refer to CDP by a company or organization name (e.g., "Siemens mortality predictor"); a third clinician may refer to CDP by an abbreviation; and so on. Similarly, a first clinician may refer to a patient as Black, a second clinician may refer to the patient as African American, etc.

[0025] Accordingly, determining whether CDP may produce different outcomes for different patient groups may require first retrieving and storing large amounts of data from different sources; determining that the data meets a diversity threshold; separating or classifying the data for various different differential factors (e.g., race, gender, age, social determinants of health (SDOH), socioeconomic status (SES), etc.) and determining the sufficiency of the differential factor data; tracking and aggregating mentions of CDP in the data; determining whether a patient's treatment was based on the output of CDP; determining whether that patient's health outcome was based on that treatment; and encoding and organizing the data (differential factor classes, outcome classes, etc.) into a standardized format for applying one or more statistical or ML models to evaluate algorithmic bias. This work relies on considerable memory and processing resources, which may have to be committed before knowing whether there is sufficient data to generate an accurate health equity assessment of CDP or when the data is sufficient.

[0026] In addition, CDPs can change continuously. For example, a company developing a readmission predictor based on an AI model can retrain the AI model periodically and release updated versions, or clinicians can add new rules to a rule-based intelligent warning CDP. Therefore, a relatively short time window may be available for performing a reliable health equity assessment on a CDP. For example, a health equity assessment of a CDP using three months of data may be accurate for the next six months, but a health equity assessment of a CDP taking twelve months may be inaccurate when it is generated. Therefore, it may be impossible to generate an accurate assessment of a CDP within an appropriate time frame using current tools and methods and currently available data.

[0027] Thus, the central problem in generating an accurate health equity assessment of a CDP for analyzing patient data lies in quickly and effectively collecting specific data applicable to the assessment process in a highly standardized format that currently does not exist. This data includes at least information on differential factors of the patients analyzed using the CDP (e.g., gender, race, SDOH, SES, etc.), as well as simplified / binary data on the performance of the CDP on the patients (e.g., whether the output of the CDP is accurate for the patient), which is linked to a specific version of the CDP at a specific point in time. By collecting and storing health equity data in a standardized format, rather than sifting through and cross-referencing data already stored in different databases of the healthcare system, the memory and processing resources of the healthcare information system used to analyze health equity data can be reduced, thereby improving the operating efficiency of the healthcare information system and increasing the amount of memory and processing resources available for other clinical tasks. Additionally, memory and processing resources may not be applied until the amount of collected health equity data is considered sufficient to generate an accurate assessment, thereby reducing the probability of wasted effort. The time spent processing health equity data can also be reduced, enabling the generation of appropriate assessments within a short enough time window to be useful for clinicians, researchers, and product owners, and shortening the development cycle for retraining and / or updating the model, thereby increasing the effectiveness of the CDP in practice and promoting increased trust in the medical and patient communities.

[0028] To address this issue, this article provides a confidence and health equity assessment system for CDPs that meet the general data product definition. In many implementations, CDPs can be obtained from a data product marketplace. Users such as care providers can select CDPs available at the data product marketplace via a clinical workflow application or a clinical information system (CIS). For example, a user can select a CDP from a digital catalog displayed within a clinical workflow application or CIS. The selected CDP can be launched by the user from within the clinical workflow application or CIS. When the CDP is launched, a rating tool micro - application of the confidence and health equity assessment system can be opened within the clinical workflow application and / or the CDP. When the CDP is used for diagnosing or treating a patient, the rating tool micro - application can request clinical feedback from the user regarding the suitability, appropriateness, accuracy, or quality of the CDP output via visual control elements displayed on the user's screen. This clinical feedback can include demographic data on various differential factors of the patient population analyzed by the CDP. The clinical feedback provided by the user is then aggregated by the confidence and health equity assessment system, which performs a health equity assessment and generates a confidence rating for the CDP, where the confidence rating is a multi - dimensional score indicating the degree of confidence that the CDP's output is not affected by algorithmic bias for various differential factors. In this way, biases in various CDPs can be identified and corrected, thereby improving the quality and equity of clinical data products and enabling care providers and patients to make more informed decisions. Overall, a greater degree of transparency can be provided regarding the unfairness of AI models, which can enable faster adoption of CDPs in standard clinical practice, resulting in better patient outcomes.

[0029] The confidence and health equity assessment system can rely on various models to generate the health equity assessment. The various models can include AI models such as machine learning (ML) or deep learning (DL) models, generative AI, or different kinds of models. For example, one or more of the various models can be implemented as a convolutional neural network (CNN) model including multiple hidden layers. These models can include classification or prediction models, probabilistic models (e.g., Bayesian models), statistical models, or different types of models. The various models can also include one or more natural language processing (NLP) models, such as private implementations of commercial, open - source, or in - house large language models (LLMs) (such as OpenAI's GPT), which can be trained to provide a natural language summary of the output of the confidence and health equity assessment system.

[0030] Now refer to the accompanying drawings, Figure 1An exemplary health equity assessment system 102 according to an embodiment is shown. As described herein, the health equity assessment system 102 can generate a confidence assessment of the effectiveness, fairness, and sensitivity of the results output by a CDP for diagnosing or treating patients in a healthcare system, with respect to geography, race, age, gender, disease status, and / or other factors susceptible to AI algorithm bias.

[0031] In some embodiments, at least a portion of the health equity assessment system 102 is disposed at a device (e.g., a workstation, an edge device, a server, etc.) that is communicatively coupled via wired and / or wireless connections to one or more healthcare systems and / or hospital networks and computer systems 136, and can receive or access medical data (including patient data) stored in one or more healthcare systems and / or hospital computer systems 136. The health equity assessment system 102 can also be operably / communicatively coupled to a user input device 132 and a display device 134. In some examples, the user input device 132 can be a shared input device of one or more healthcare and / or hospital computer systems 136, and the display device 134 can be a shared input device of one or more healthcare and / or hospital computer systems 136.

[0032] The health equity assessment system 102 can also be operably / communicatively coupled to one or more of the electronic medical records (EMRs) 140, from which the health equity assessment system 102 can retrieve patient records. The health equity assessment system 102 can be operably / communicatively coupled to one or more clinical workflow applications 160 used by care providers in a healthcare system, e.g., when determining a patient's diagnosis or treatment options. In particular, the health equity assessment system 102 can receive clinical feedback (e.g., ratings) regarding the effectiveness of one or more CDPs 175, which can be retrieved from the data product marketplace 170 and initiated within one or more clinical workflow applications 160. The clinical feedback can be transmitted to the health equity assessment system 102 via the CDP rating tool 162, a micro - application that can be installed in the GUI of the clinical workflow application 160 or the CDP 175. The CDP rating tool 162 is described in more detail below with reference to Figure 5 more detailed description of the CDP rating tool 162.

[0033] The data product marketplace 170 can be operably / communicatively coupled to the health equity assessment system 102 such that the data generated by the health equity assessment system 102 can be associated with the CDPs 175 of the data product marketplace 170. For example, a health equity assessment or rating can be associated with a CDP 175 in the CDP catalog 176 of the data product marketplace 170, which can include a list of CDPs 175 available for care providers as stand - alone products or within clinical workflow applications 160.

[0034] The health equity assessment system 102 includes a processor 104 configured to execute machine-readable instructions stored in a non-transitory memory 106. The processor 104 can be a single-core or multi-core processor, and the program executed thereon can be configured for parallel processing or distributed processing. In some embodiments, the processor 104 can optionally include separate components distributed across two or more devices, which can be located remotely and / or configured to coordinate processing. In some embodiments, one or more aspects of the processor 104 can be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0035] The non-transitory memory 106 can store an AI model module 108, a clinical user feedback database 118, and a confidence score database 120. The AI model module 108 can include one or more difference assessment models 110, a confidence score calculator 112, a model drift detector 114, and a clinical context summarizer 116. The AI model module 108 can include various AI models, such as ML and / or DL models. In particular, the AI model module can include one or more large language models (LLMs) that can be trained to generate summaries of patient cases in natural language, which can be evaluated against one or more difference assessment models 110 to determine if there is algorithmic bias in the CDP, as described in more detail below with reference to Figure 6 In various embodiments, the various AI models and / or difference assessment models can include neural network models such as convolutional neural networks (CNNs), generative adversarial networks (GANs), transformers, and traditional classifiers; Bayesian networks, decision trees, and / or other statistical, probabilistic, or hierarchical models. The AI model module 108 can include trained and / or untrained neural networks and can also include various data or metadata related to one or more AI models stored therein.

[0036] The one or more AI models can include a confidence score calculator 112 that can calculate case-specific, multi-dimensional confidence scores based on clinical user feedback received from a CDP rating tool 162 and stored in the clinical user feedback database 118. The confidence scores generated by the confidence score calculator 112 for various CDPs can be stored in the rating database 176. For example, the confidence scores can be retrieved by a CDP catalog 176 and displayed next to the listed CDPs 175 in the CDP catalog 176. When selecting a CDP for a given patient, the care provider can consider the confidence scores.

[0037] One or more AI models may include a model drift detector 114 that can monitor the health equity assessments and confidence ratings of various CDPs to determine trends in clinical user feedback. In one example, if clinical user feedback indicates that the overall confidence score of a CDP is decreasing, it can be inferred that the underlying AI model is becoming increasingly inaccurate due to algorithmic bias (e.g., model drift), and the software manufacturer of the CDP can be notified accordingly.

[0038] The clinical context summarizer 116 can retrieve social determinants of health disparity factors such as gender, race and ethnicity, socioeconomic status, geographic region, and disability status from the EMR 140 and / or one or more healthcare and / or hospital computer systems 136, and generate a patient case summary that is evaluated by the disparity assessment model 110. The patient case summary can be a structured text file, such as a JavaScript Object Notation (JSON) or similar file, which can be formatted for inputting patient data into one or more of the disparity assessment models 110. The patient case profile or second patient case summary can also be constructed as a natural language summary, which can be generated by the LLP. For example, when reviewing a CDP in the CDP catalog, the natural language summary can be displayed to the clinician.

[0039] The user input device 132 can include one or more of a touch screen, keyboard, mouse, touchpad, microphone, motion-sensing camera, or other devices configured to enable a user to interact with the health equity assessment system 102. In one example, the user input device 132 can enable a user to submit questions about a patient to the health equity assessment system 102 in natural language. For example, a user (e.g., a care provider) can use a keyboard to type questions into the health equity assessment system 102, or speak the questions into a microphone.

[0040] The display device 134 can include one or more display devices utilizing almost any type of technology. In some embodiments, the display device 134 can include a computer monitor. The display device 134 can be combined with the processor 104, non-transitory memory 106, and / or the user input device 132 in a shared package, or can be a peripheral display device, and can include a monitor, touch screen, projector, or other display devices known in the art that can enable a user to view responses to queries submitted to the health equity assessment system 102, and / or interact with various data stored in the non-transitory memory 106.

[0041] It should be understood that Figure 1 the health equity assessment system 102 shown is for illustration and not for limitation. Another suitable health equity assessment system may include more, fewer, or different components.

[0042] Figure 2 shows a data flow diagram 200 that illustrates how data can be processed by a health equity assessment system 214, which can be a non-limiting example of the health equity assessment system 102 of Figure 1 . A clinician 207 (e.g., a care provider of a healthcare system) may wish to analyze patient data of one or more patients from a patient group 202 of a healthcare system. The clinician 207 can open a clinical workflow application 212 on the clinician device 208 of the clinician 207. For example, the device can be a computer of the clinician 207, or a tablet or smartphone of the clinician 207, or a different type of computing device. The clinical workflow application 212 can be displayed on the display screen 210 of the device 208.

[0043] From within the clinical workflow application 212, the clinician 207 can initiate a CDP 206. For example, the CDP 206 can be a dashboard that displays patients of the patient group 202 organized based on patterns in the patient data detected by the CDP 206 or an AI tool or different data analysis products. The CDP 206 can be selected by the clinician 207 from a CDP marketplace 224, where the CDP marketplace 224 can be an online repository of various data analysis products (e.g., CDPs) available to the clinician 207. The CDP marketplace 224 can be hosted within a healthcare information system 220 of the healthcare system and can include products manufactured by various trusted companies having a business relationship with the healthcare system. In various embodiments, the clinician 207 can select the CDP 206 from a CDP catalog 226 of the CDP marketplace 224. The healthcare information system 220 includes a processor 222 that can process instructions for managing the display of CDPs within the CDP marketplace 224 and the CDP catalog 226. In some examples, the CDP catalog 226 can be part of or accessible within the clinical workflow application 212.

[0044] In various embodiments, a clinician 207 may use a clinical workflow application 212 to select or define a patient group 202. The clinician 207 may use the clinical workflow application 212 to select or define patient data 204, or the patient data 204 may be selected or defined by the CDP 206. The CDP 206 may analyze the patient data 204 and generate an output that may be displayed on a display screen 210. The output may be displayed within the GUI of the clinical workflow application 212, or within the GUI of the CDP 206, or within a different GUI. The output may include predictions, estimates, recommendations, or other results of analysis tools and AI or statistical models applied to the patient data 204. Based on the output, the clinician 207 may generate one or more treatment regimens 216 to be applied to the patients in the patient group 202 to achieve a desired clinical outcome 218 for the patients.

[0045] To collect data on potential algorithmic bias of the CDP 206, a health equity assessment system 214 may generate a rating tool 213 (e.g., Figure 1 the CDP rating tool 162) and display the rating tool on the display screen 210. For example, the rating tool 213 may be displayed within the GUI of the clinical workflow application 212 or within the GUI of the CDP 206 (if applicable). The clinician 207 may use the rating tool 213 to indicate the level of satisfaction with the output of the CDP 206 specifically regarding health equity, e.g., the extent to which the results produced by the CDP 206 are applicable to the patient group 202.

[0046] Using the rating tool 213, clinical user feedback 209 regarding the health equity of the CDP 206 relative to the patient group 202 may be sent to the health equity assessment system 214. The clinical user feedback 209 may be stored at the health equity assessment system 214 (e.g., in Figure 1 the clinical user feedback database 118) and processed by the health equity assessment system 214 to generate an overall health equity assessment of the CDP 206 (e.g., meaning a summary health equity assessment over time for various tasks). For example, the health equity assessment may be a multi-dimensional confidence score that the CDP 206 does not suffer from algorithmic bias. The overall health equity assessment of the CDP 206 may then be stored at the health equity assessment system 214 (e.g., in Figure 1 the confidence score database 120). The overall health equity assessment may be associated with the CDP 206 in the CDP directory 226 such that clinicians such as the clinician 207 may view the overall health equity assessments of the various CDPs listed in the CDP directory 226 before selecting an appropriate CDP for a given clinical task.

[0047] In some embodiments, when generating a list of CDPs to be displayed (e.g., in response to search terms used by clinician 207), the CDP catalog 226 can dynamically and in real-time retrieve, from the health equity assessment system 214, the overall health equity assessment of the various CDPs listed in the CDP catalog 226. In other embodiments, the overall health equity assessment of the various CDPs listed in the CDP catalog 226 can be retrieved and updated periodically from the health equity assessment system 214.

[0048] Figure 3 FIG. 300 is a schematic block diagram illustrating the data flow through the various data ports of a CDP 302 during its use within the framework of a health equity assessment system (such as Figure 1 health equity assessment system 102 and / or Figure 2 health equity assessment system 214). The CDP 302 can be a non-limiting example of the CDP 206. Although the CDP 302 is described herein as being initiated within a clinical workflow application such as the clinical workflow application 212, it should be understood that in some embodiments, the CDP 302 can be used as a stand-alone product.

[0049] To enable the health equity assessment system to perform a health equity assessment of the data output by the CDP 302, the CDP 302 can conform to a general data product definition. The general data product definition can establish a set of input data ports 351 into the CDP 302; a set of output data ports 352 from the CDP 302; and a set of standardized control data ports 353. In other words, the general data product definition establishes a formal structure that specifies the types of input data that can be received by the CDP 302, the output data generated by the CDP 302 for the clinician, and the metadata or background data generated from the CDP 302 during the processing of the input data.

[0050] The input data port 351 may include a streaming data port 304, an event data port 306, and a request port 308. The CDP 302 may use the streaming data port 304 to receive streaming data, such as time series data generated by a patient monitor (such as blood pressure, heart rate, ECG, or other vital signs) and / or environmental sensor data (such as temperature, oxygen, and humidity), which may be collected and streamed in real time. The CDP 302 may receive event data via the event data port 306, where the event data may include warning and alert data sent by physiological devices and clinical patient data and care observations collected and stored over time in a patient health record in an EMR (e.g., EMR 140). Requests sent to the CDP 302, such as requests from a clinician to perform a specific analysis or generate a specific report or result, may be received via the request port 308.

[0051] The output data port 352 may include a GUI port 310, an output event port 312, a dashboard port 314, and an output request port 316. Result data generated by the CDP 302 based on the input data received at the input data port 351 may be output via the GUI port 310 for display in a GUI, such as the GUI of the CDP 302 or the GUI of a clinical workflow application. Similarly, result data generated by the CDP 302 based on the input data received at the input data port 351 may be output via the dashboard port 314 for display in a dashboard, such as the dashboard of a clinical workflow application. For example, the dashboard data may be organized in a spatial layout of columns and / or rows and may include control elements for performing actions or accessing additional data (e.g., mining). Event data generated by the CDP 302 may be output via the output event port 312. For example, the event data may include data that can be input into different software components or tools for further analysis.

[0052] In some cases, the CDP 302 may generate requests for data from other sources, such as an EMR or other clinical database accessible on a hospital network (e.g., hospital computer system 136) or within a hospital information system (e.g., healthcare information system 220). For example, a first algorithm of the CDP 302 may perform a first analysis on patient data to generate a first result. Based on the first result, a request for additional information may be generated for processing by a second algorithm of the CDP 302, and the request may be output via the output request port 316. The requested data may be retrieved and received by the CDP 302 via the input event port 306 and processed by the second algorithm. The result of the second algorithm may be output to the GUI via the GUI port 310 or to the dashboard via the dashboard port 314.

[0053] The standardized control data port 353 may include a background port 320, an interpretability port 322, a usage port 324, a warning port 326, and a description port 328. The background port 320 may include patient / clinical background data passed to the CDP via input data (e.g., patient identifiers such as medical record number, SSN, admission date, location, department, etc.), which may enable the explicit lookup of additional patient demographics and SDOH information from the EMR.

[0054] Data from the healthcare AI model through the interpretability port 322 may take various forms, depending on the specific model type and the desired level of understandability. For example, it may include feature attribution data such as saliency maps, LIME (Local Interpretable Model-agnostic Explanations), SHAP (Shapley Additive exPlanation), or other feature attribution data. This data may be generated during AI model inference.

[0055] Data through the usage port 324 may provide information about the clinical workflow applications using the AI model. This data may be static data embedded within the CDP 302 when the CDP 302 is deployed within a clinical workflow.

[0056] The warning port 326 allows the collection of background information about clinical warnings, alerts, notifications, etc. that may have been triggered by the AI model. This background information may be generated during AI model inference.

[0057] The description port 328 may allow the collection of a human-readable description (e.g., static text) of the purpose of the CDP 302, including one or more embedded AI models of the CDP 302.

[0058] It should be understood that the examples described above are for illustrative purposes, and in other embodiments, more or fewer ports, or different types of ports, may be defined for the CDP 302 without departing from the scope of the present disclosure. For example, in some embodiments, the standardized control data port 353 may include an AI model output performance metric port, a data drift distribution (e.g., input features) port for implementing AI model drift assessment as described in more detail below, and / or other types of ports.

[0059] Figure 4 An exemplary data product market 400 is shown, which may be associated with Figure 1is the same as or similar to the data product market 170. The data product market 400 includes a plurality of CDPs 402 (e.g., CDP 206), which can be selected by clinicians for analyzing patient data. In various embodiments, the CDPs 402 can be listed in a CDP catalog 404 (e.g., CDP catalog 226), where clinicians 416 can view or search for CDPs that can be applied to patient data for desired clinical tasks. In addition to clinicians 416, users of the CDP catalog 404 and the CDPs 402 included therein can include clinical researchers 418, CDP owners 420, data scientists 422, and the like.

[0060] As described above, the CDP catalog 404 can be created and hosted by a hospital information system such as the healthcare information system 220. The CDP catalog 404 can be displayed within the GUI of the hospital information system, or the CDP catalog 404 can be retrieved from the hospital information system and displayed within a clinical workflow application (e.g., clinical workflow application 212) used by clinicians 416. In other embodiments, the CDP catalog 404 can be generated by the clinical workflow application. In still other embodiments, the CDP catalog 404 can be created by a health equity assessment system and displayed on a display device (e.g., display device 134) of the health equity assessment system, or retrieved from the health equity assessment system and displayed in the GUI of the clinical workflow application.

[0061] For each CDP 402 listed in the CDP catalog 404, data about the CDP can be displayed or made accessible (e.g., via a link) to users of the CDP catalog 404. A first portion of the data can be displayed in a top view, which means that some data for each CDP 402 can be displayed next to the CDP 402 on the list to help users select a suitable CDP 402 for a desired task. The first portion of the data can include, for example, the intended use of the CDP 402, the results or benefits that can be expected from analyzing patient data using the CDP 402, the intended patient population of the CDP 402, and the like. When a CDP 402 is selected from the list, a second portion of the data can be displayed. The second portion of the data can include, for example, one or more reviews 406 of the CDP 402; a health equity assessment 408 of the CDP 402; a drift warning 410 of the CDP 402; and / or a retraining backlog 412 of the CDP 402.

[0062] The different elements of the second part of the CDP data can be associated with different types of users of the CDP 402. As a first exemplary use case, a clinician 416 can scan a list of available CDPs 402 in the CDP catalog 404 and use the CDP data of the first part to identify candidate CDPs for a desired clinical analysis. The clinician 416 can select a CDP 402 in the CDP list of the CDP catalog 404, which can display a page or view with additional information of the second part. The additional information can include a review of the CDP 402. The clinician 416 can read the review to determine the potential effectiveness of the CDP 402 for the desired clinical analysis. The clinician 416 can also view the health equity assessment of the CDP 402 to evaluate the likelihood that the CDP 402 may have been created or trained based on data from a training patient population that may not match the patient cohort (e.g., patient cohort 202) of the desired clinical analysis. If the health equity assessment causes the clinician 416 to believe that the training patient population may not match the patient cohort, the clinician 416 can select a different CDP 402 for the desired clinical analysis.

[0063] In some embodiments, the health equity assessment can include a multi-dimensional rating or score of the CDP 402, where each dimension of the multi-dimensional rating or score can correspond to a factor of difference. For example, the multi-dimensional rating can include a first rating for gender differences; a second rating for racial differences; a third rating for differences related to disability; a fourth rating for SDOH; a fifth rating for language barriers; and so on. In this way, the algorithmic bias of the CDP 402 can be evaluated and indicated separately for each factor of difference.

[0064] In some embodiments, the health equity assessment can be stored in the CDP catalog 404 or in the CDP 402. In other embodiments, the clinician 416 can select a health equity assessment control element in the CDP catalog 404 or in a page or view with additional information, and the health equity assessment can be retrieved from a database of the health equity assessment system (e.g., the confidence score database 120) and displayed in the page or view with additional information. For example, the health equity assessment can be retrieved in real time.

[0065] As a second exemplary use case, a clinical researcher 418 can conduct a study on the health equity of CDPs available on the CDP market 400. The clinical researcher 418 can review the various health equity assessments 408 of the various CDPs 402, compare the various health equity assessments 408, integrate the various health equity assessments 408 into other studies, and / or view other data associated with the various CDPs 402 to confirm or further clarify the health equity assessments 408.

[0066] As a third exemplary use case, the CDP owner 420 can review the CDP 402 owned by the CDP owner 420 to determine whether a drift warning 410 might be associated with the CDP 402. The drift warning 410 can be generated by a model drift detector of a health equity assessment system (e.g., Figure 1 model drift detector 114). In some embodiments, the model drift detector can monitor for drift in the CDP model and automatically generate a drift warning 410 in response to detecting a decrease over time in the accuracy, performance, or effectiveness of the CDP 402 when the CDP 402 is applied to different patient populations. In response to the drift warning, the CDP owner 420 can decide to remove the CDP 402 from the CDP catalog 404 and / or notify one or more data scientists 422 responsible for the CDP 402 to retrain one or more models of the CDP 402. The generation of drift warnings is described in more detail below with reference to Figure 8 ..

[0067] Additionally, the CDP owner 420 can review the ratings of the CDP product and, based on the potential biases found, decide how and where to package the CDP product into various digital applications or solutions with respect to usage applicability. For example, a CDP can be recommended for a first patient population where no algorithmic bias is detected and not recommended for a second patient population where algorithmic bias is detected. For example, a CDP can be recommended for patients within an age range or not recommended for patients taking certain medications, etc.

[0068] As a fourth exemplary use case, the data scientist 422 can consult the retraining backlog 412 for the CDP 402. In various embodiments, the retraining backlog 412 can be a collection of clinical user feedback stored with reference to the CDP 402. For example, in some embodiments, when providing a rating for the CDP 402, a clinician can also enter written feedback that can be stored at the health equity assessment system. The written feedback from various clinicians can accumulate over time into the retraining backlog 412. The data scientist 422 can use the retraining backlog 412 of the accumulated feedback to inform further training of the CDP 402. For example, the accumulated feedback can indicate that the AI model of the CDP 402 is performing poorly on patients from a certain demographic group, and the data scientist 422 can retrain the AI model on a training data set compiled from patients of the demographic group to increase the overall accuracy of the AI model and / or reduce the algorithmic bias of the AI model.

[0069] Now referring to Figure 5 , there is shown a CDP rating tool micro - application for use (e.g., Figure 1The CDP rating tool 162) A method 500 for capturing clinical feedback regarding CDP, the CDP rating tool micro - application can be installed in a clinical workflow application such as Figure 2 within the clinical workflow application 212. The method 500 can be executed by a processor of a computing device of a care provider, and the clinical workflow application runs on this computing device. Each part of the method 500 can be executed based on instructions stored in the CDP rating tool micro - application and executed by the processor. The clinical feedback can then be processed by a health equity assessment system (such as Figure 1 the health equity assessment system 102 and / or Figure 2 the health equity assessment system 214) to evaluate whether algorithmic bias may affect the output of the CDP. The processing of clinical feedback by the health equity assessment system is described below with reference to Figure 6 Description of the processing of clinical feedback by the health equity assessment system.

[0070] The method 500 begins at 502, where the method 500 includes receiving a clinician's selection of a CDP. As described above, a clinician can select a CDP from a data product catalog based on a clinical task desired to be performed on the clinician's patient.

[0071] At 504, the method 500 includes retrieving the selected CDP from a repository of CDPs linked to the CDP catalog and deploying the selected CDP within the clinical workflow application. When the selected CDP is deployed, the CDP can receive input data including patient data at one or more input ports of the CDP (e.g., input port 351), as specified by the clinician via the CDP or the GUI of the clinical workflow application. The CDP can perform a predetermined analysis on the input data and output the result via one or more output ports of the CDP (e.g., output port 352).

[0072] At 506, the method 500 includes determining whether the CDP generates an output event. An output event can be generated when the result of the analysis performed by the CDP on patient data is available. The result can include the output of an AI model (such as an ML model) or a statistical model, or the result of a rule - based system (such as a decision tree). For example, the result can be a warning, or a possible diagnosis, or a recommendation for a proposed treatment, or different types of results.

[0073] Brief reference Figure 9, The CDP event generation diagram 900 shows an exemplary CDP 904 that can be deployed within a clinical workflow application as described in method 500. The exemplary CDP 904 is an intelligent warning data product that can generate a warning 908 in response to a set of patient data 902 of a patient meeting one or more conditions. In the depicted embodiment, the CDP 904 is an intelligent warning related to a patient's respiratory depression, where the warning 908 can be generated in response to a set of conditions established within the rule engine 906 of the CDP 904. The warning 908 can be displayed within the GUI of the clinical workflow application or sent to a care provider who is configured to receive intelligent warnings in a different manner.

[0074] The CDP 904 can be created by a care provider, and the set of conditions established in the rule engine 906 can be established by the care provider. Alternatively, the CDP 904 can be predefined with the set of conditions and selected via a CDP catalog, as described with reference to method 500.

[0075] In the depicted embodiment, the set of conditions includes various parameters and corresponding target values or thresholds. For example, the set of conditions of the rule engine 906 can determine that if the patient's end-tidal CO2 measured over at least three minutes is below 15 mmHg or above 60 mmHg, or the patient's apnea lasts longer than 30 seconds, or the patient's respiratory rate measured over at least three minutes is below 5 breaths per minute, or the patient's oximeter reading is below 85% over at least three minutes, then the warning 908 can be generated. It should be understood that the CDP 904 is an example of a CDP, and other examples can include fewer or more conditions and / or different types of conditions without departing from the scope of the present disclosure. Additionally, in other embodiments, the CDP 904 can be a different type of model that generates different types of outputs.

[0076] Returning to method 500, if it is determined at 506 that the CDP has not generated an output event, then method 500 proceeds to 508. At 508, method 500 includes waiting until an output event is generated, and method 500 returns to 506. Some time may elapse before an output event is generated, or in the case of a warning, an output event may not be generated.

[0077] Alternatively, if it is determined at 506 that an output event has been generated by the CDP, method 500 proceeds to 510. At 510, method 500 includes deploying a CDP rating tool within the GUI of the CDP or within the GUI of a clinical workflow application. In other words, the CDP rating tool can be automatically deployed in response to an event generated by the CDP. In one embodiment, when the CDP rating tool is deployed, a set of control elements (e.g., selectable icons) can be displayed that allow a clinician to provide clinical feedback to the health equity assessment system. After the clinician has reviewed the output event, a single selection action, such as a click of a mouse or similar input device, can be used to provide the clinical feedback.

[0078] Briefly referring to Figure 10 , a set of exemplary control elements 1000 is shown that can be displayed in the GUI of the CDP or a clinical workflow application that uses the CDP. For example, the set of control elements 1000 can be overlaid on a portion of the GUI, such as next to an indication of the results output by the CDP. For example, the set of control elements 1000 includes a first control element 1002, a second control element 1004, a third control element 1006, and a fourth control element 1008. The first control element 1002 includes a "thumbs up" icon, where selecting the first control element 1002 can indicate that the output of the CDP is correct or accurate, or that the clinician has found no evidence that the output of the CDP may be biased towards a different patient population than the patient or patient group being rated by the clinician (e.g., patient group 202). The third control element 1006 includes a "thumbs down" icon, where selecting the third control element 1006 can indicate that the output of the CDP is incorrect or inaccurate, or that the clinician believes the output of the CDP may be biased towards a different patient population than the patient or patient group. The second control element 1004 includes a "thumbs sideways" icon, where selecting the second control element 1004 can indicate that the clinician does not have enough information to evaluate the potential bias of the output of the CDP. The fourth control element includes a "comment" icon, where selecting the fourth control element 1008 can open an editor pane that can allow the clinician to enter comments or written feedback to be collected by the health equity assessment system. For example, the written feedback can be used to generate the retraining backlog 412 described above with reference to Figure 4 In this way, the clinician can quickly and effectively rate the output of the CDP within the clinician's normal workflow without being distracted or burdened by opening and closing additional screens, entering information, or navigating various control options.

[0079] It should be noted that in other embodiments, the set of control elements 1000 of the CDP rating tool can appear differently and can include different icons, images, or symbols and can include a greater or lesser number of options.

[0080] Returning to method 500, at 512, method 500 includes receiving, for example in the manner described above, the health equity rating of CDP via the CDP rating tool. When a clinician selects one of the control elements 1002, 1004, or 1006, all of the control elements 1002, 1004, or 1006 can disappear from the display. In some examples, the CDP rating tool can additionally be displayed via a menu of the clinical workflow application or the CDP GUI, such that the clinician can change their feedback in case of an error. The rating of CDP can be accompanied by written feedback entered via the fourth control element 1008, or written feedback may not be provided.

[0081] At 514, method 500 includes sending the clinical feedback (e.g., the rating generated by the CDP rating tool) to the health equity assessment system, and method 500 ends. In various embodiments, the clinical feedback can be sent to the health equity assessment system via a hospital network (e.g., hospital computer system 136), or via the Internet, or via a different private or public network.

[0082] Now referring Figure 6 , a method 600 for generating a health equity assessment of CDP using a health equity assessment system such as Figure 1 the health equity assessment system 102 is shown. Method 600 can be a machine-implemented method executed by a processor (e.g., processor 104) of the health equity assessment system based on instructions stored in a non-transitory memory (e.g., non-transitory memory 106) of the health equity assessment system. The health equity assessment can be generated based on ratings provided by clinicians, as described above with reference to Figure 5 method 500.

[0083] Method 600 begins at 602, where method 600 includes receiving a rating of CDP used by a clinician to analyze patient data, where the rating can be sent via the CDP rating tool as described above. The rating can include an indication of the clinician's satisfaction with the output of CDP. Specifically, the rating can indicate whether the clinician believes the output is accurate for the patient; whether the clinician believes the output is inaccurate for the patient, which may be due to algorithmic bias in the CDP model; or whether the clinician lacks sufficient information to form a judgment about potential algorithmic bias in the CDP model. The rating can also include written feedback provided by the clinician. The rating (and written feedback, if included) can be stored in a database (e.g., Figure 1 the clinical user feedback database 118) of the health equity assessment system. The rating and written feedback can be aggregated with other ratings and written feedback provided by other clinicians.

[0084] At 604, method 600 includes generating a patient case summary regarding the use of a CDP for which a patient or group of patients is rated. The patient case summary can rely on a clinical context summarizer model or module of the health equity assessment system (e.g., clinical context summarizer 116), which can retrieve social determinants of one or more patients' health factors (e.g., health disparity factors), such as the racial or ethnic background of the patient or group of patients; the gender of the patient or group of patients; the age of the patient or group of patients; and / or other factors that may affect the accuracy of the output of the CDP. In various embodiments, an LLM can be used to generate the patient case summary as a natural language (e.g., unstructured text) summary. For example, the LLM can be prompted to search the patient records of the EMR (e.g., EMR 140) and retrieve clinical context data including patient conditions, diagnoses, treatments, and health disparity factors, and generate a summary report for future review by clinicians. The natural language summary can be stored in the database of the health equity assessment system, and the natural language summary can be retrieved from the database and included in the health equity assessment information provided regarding the CDP, such as in the CDP catalog (e.g., CDP catalog 226). Examples of natural language summaries are described below with reference to Figure 11 Examples of natural language summaries are described below with reference to

[0085] At 606, method 600 includes determining whether a sufficient amount of rating data has been collected to perform a health equity assessment of the CDP. In some examples, the health equity assessment can be performed each time a rating is received, and the result of the health equity assessment can be associated with the CDP based on one rating or a small amount of rating data. In other examples, the health equity assessment may not be performed each time a rating is received, and the health equity assessment can be performed after a predetermined threshold amount of rating data is received from a clinician or from multiple clinicians. The predetermined threshold amount of rating data can be established in various ways and according to various criteria, and can vary depending on, for example, the type of CDP, or the risk or other factors associated with acting on the output of the CDP. In some cases, the health equity assessment can be performed when the collected rating data includes a threshold number of negative ratings indicating possible algorithmic bias of the CDP.

[0086] If it is determined at 606 that the rating data is insufficient to perform a health equity assessment of the CDP, method 600 returns to 602, where method 600 waits to receive additional ratings. Alternatively, if it is determined at 606 that the rating data is sufficient to perform a health equity assessment of the CDP, method 600 proceeds to 608.

[0087] At 608, method 600 includes performing a health equity assessment of the CDP based on aggregated rating data collected from one or more clinicians and / or users of the rating tool. Performing the health equity assessment of the CDP includes, at 610, aggregating and summarizing the clinical context of each patient's use of the CDP for which the rating data has been collected. The clinical context may include social determinants of one or more of the above patient health factors (e.g., health disparity factors). The clinical context may be summarized by a clinical context summarizer of the health equity assessment system. In one embodiment, the clinical context summarizer may include an AI model, such as a rule-based system or an ML model, which may summarize the clinical context into a structured text document or file, such as a JSON file. The structured text document may include tagged data in a standardized format, which may be used as input data for the disparity assessment model. In some examples, the clinical context summarizer may rely on an LLM to generate the structured text document, and / or may rely on a patient case summary generated by the LLM as described above to generate the structured text document.

[0088] At 612, performing the health equity assessment includes using one or more disparity assessment models to evaluate the summarized clinical context data. The disparity assessment model may obtain data from the structured text document and analyze or process the summarized clinical context data to determine the extent to which the output of the CDP, as rated by the clinician, may be affected by algorithmic bias with respect to one or more of the disparity factors. The disparity assessment model may evaluate the output of the CDP according to various metrics and / or techniques. For example, the disparity assessment model may evaluate the accuracy, precision, or sensitivity of the CDP. The disparity assessment model may use metrics such as calculating the F1 score; calculating the mean absolute error (MAE) or root mean square error (RMSE); calculating the R-squared value; calculating the area under the receiver operating characteristic (ROC) curve; or different metrics.

[0089] For example, a first disparity model may include a statistical model that uses various statistical techniques known in the art to analyze the correlation between ratings and different subgroups of the patient population analyzed using the CDP. A second disparity model may include a convolutional neural network (CNN) that is trained to predict the degree of algorithmic bias in the summarized clinical context data. Generally, there may be various types of health disparity models, each focusing on different aspects of the complex problem of unequal health outcomes across groups. Examples may include conceptual models, statistical models, and / or other types of models.

[0090] Conceptual models may include, for example:

[0091] ● Social Determinants of Health (SDOH) Model: This model views social and economic factors such as income, education, and housing as fundamental determinants of health, creating inequalities in access to healthcare, quality of life, and ultimately health outcomes.

[0092] ● Intersectionality Model: This model highlights the complex interactions of multiple identities and social positions (e.g., race, gender, socioeconomic status) in shaping health experiences and disparities.

[0093] ● Equity Goal Intervention Model: These interventions are specifically designed to address the needs and challenges of groups experiencing health disparities, promoting targeted resource allocation and culturally appropriate strategies.

[0094] Examples of statistical models can include, for example:

[0095] ● Causal Inference Model: These models attempt to establish causal relationships between specific interventions or strategies and changes in health disparities, thus guiding effective interventions.

[0096] ● Machine Learning Model: These models can be used to identify patterns and associations in complex health data, potentially revealing previously unknown factors contributing to disparities.

[0097] It should be understood that the examples described herein are for illustrative purposes, and various different types of disparity assessment models known in the art can be used to evaluate the health equity of CDP without departing from the scope of the present disclosure.

[0098] In some embodiments, the disparity assessment model can include or be used in conjunction with a model drift detector (e.g., model drift detector 114) to detect changes in the performance of CDP over time. Various metrics can be used to measure model drift.

[0099] Brief reference Figure 8 , shows a method 800 for evaluating model drift in CDP. Before deploying CDP and making it available in the market, the baseline performance of CDP can be evaluated against an initial training dataset so that the future performance of CDP can be measured against the baseline performance. The baseline performance data can include, for example, key metrics such as model accuracy, precision, recall, F1 score, area under the receiver operating characteristic curve (AUC-ROC), and / or other metrics and methods of the metrics described below. The key metrics used in the baseline performance and the statistical distribution of the model input features can be collected and stored in the memory of the health equity assessment system, such as Figure 1 the non-transitory memory 106.

[0100] After deploying the CDP into a clinical environment for use, the model drift detector (e.g., model drift detector 114) can automatically track, periodically or continuously, the same key metrics for new unseen data as those used in the baseline performance assessment. Similarly, the statistical distribution of the model input features in the real-world data can be continuously tracked. One or more steps of method 800 can be used to periodically evaluate the drift of the data. It should be understood that in different embodiments, one or more steps of method 800 can be executed in an order different from that described below, and / or additional steps can be added. Method 800 can be executed by a processor of the health equity assessment system (such as Figure 1 the processor 104).

[0101] Method 800 begins at 802, where method 800 includes receiving a CDP to be evaluated for model drift. The CDP can be a CDP rated by a clinician, as described above with respect to methods 500 and 600.

[0102] At 803, method 800 includes retrieving the baseline performance data of the CDP from the memory of the health equity assessment system.

[0103] At 804, method 800 includes measuring the change in the distribution of the input data used by the CDP over time using various methods. For example, these methods can include one or more distance metrics, such as the following:

[0104] ● Kolmogorov-Smirnov (KS) test: Compares the cumulative distribution functions of two datasets, where a larger KS statistic indicates a larger drift.

[0105] ● Earth Mover's Distance (EMD): Measures the minimum cost of transforming one distribution into another, where a higher EMD indicates a more significant drift.

[0106] ● Jensen-Shannon Divergence (JSD): Measures the information divergence between two probability distributions, where a higher JSD indicates a larger difference.

[0107] Various methods can include one or more statistical tests, such as the following:

[0108] ● Chi-squared Test: Compares the observed and expected frequencies of categorical variables in two datasets, where a significant p-value indicates drift.

[0109] ● Wilcoxon signed-rank test: Compares paired samples from two populations to observe whether their medians are different, where a large statistic indicates drift.

[0110] ● Population Stability Index (PSI): Measures the change in the distribution of individual characteristics between two data sets, where a higher PSI value indicates greater drift.

[0111] Various methods can include one or more model-based methods, such as the following:

[0112] ● Residual analysis: Analyzes the difference between the predictions of the model and the actual results. Residuals that increase over time can indicate drift.

[0113] ● Change point detection algorithm: Identifies statistically significant changes in model performance over time.

[0114] ● Drift detector based on anomaly detection: A separate model can be trained to identify abnormal data points that may indicate drift.

[0115] At 806, method 800 includes measuring the change over time of the relative importance of different features of the CDP. Data drift occurs when the first distribution of the input data of the AI model in production is significantly different from the second distribution of the data used for training. This can lead to inaccurate predictions and model degradation over time. Methods for measuring changes in input features include:

[0116] ● Statistical tests, such as the Kolmogorov-Smirnov test (KS test), Wasserstein distance, and Jensen-Shannon divergence (JSD), for testing by comparing the input feature distributions between training data and production data.

[0117] ● Feature importance analysis - The features that have the most impact on model predictions are established during training to closely monitor drift during production.

[0118] ● Monitoring data range and distribution, tracking statistical measurements (mean, standard deviation, quantiles) over time and visualizing the distribution for comparison.

[0119] ● Concept drift detection algorithms - Machine learning algorithms and toolkits specifically designed to detect changes in data distribution, such as Adaptive Windowing (ADWIN), Drift Detection Method (DDM), Hierarchical Bayesian Estimation of the Drift Detection Method (HDDM), etc.

[0120] At 808, method 800 includes measuring the change in the prediction or output of the CDP over time. Various methods can be used to measure the change in the output, prediction, and performance of an AI model for production data over time, such as:

[0121] ● Tracking performance metrics - Mean Squared Error (MSE), Mean Absolute Error (MAE), R-squared for regression tasks, and AUC-ROC for binary classification.

[0122] ● Monitoring the prediction distribution - Tracking the distribution of model predictions over time to detect changes in prediction patterns that may indicate drift. Common metrics include the Population Stability Index (PSI), Kolmogorov-Smirnov (KS) test, and histogram comparison.

[0123] ● Analyzing the error distribution - The distribution of prediction errors can be analyzed to identify changes in systematic bias or error patterns.

[0124] ● Using concept drift detection techniques and tracking of input feature changes: Algorithms specifically designed to detect changes in the data distribution or relationship between features and the target variable. Examples:

[0125] ADWIN, DDM, HDDM.

[0126] ● Monitoring model confidence based on user feedback - Tracking model confidence scores to detect changes in uncertainty, where a decrease in the confidence score may indicate drift in model uncertainty.

[0127] At 810, method 800 includes determining whether the measurements made in steps 804 - 808 indicate that the underlying model of the CDP has drifted beyond a threshold allowable drift amount. In various embodiments, drift can be expressed as a numeric score and a threshold can be set accordingly. The drift score can be used as a quantitative metric for changes in the underlying data or behavior of the model. For example, the drift score can be distribution-based or performance-based.

[0128] Distribution-based scoring measures the difference between the distribution of input features or model output in training and production data and can include the following:

[0129] ● Kolmogorov-Smirnov (KS) statistic: Quantifies the maximum difference between the cumulative distribution functions of two datasets. The values range from 0 to 1, where higher values indicate greater differences. A KS score above 0.3 typically indicates a significant difference between distributions pointing to a potentially impactful drift.

[0130] ● Population Stability Index (PSI): This calculates the expected relative frequency of categories or values in one distribution compared to those in another distribution. PSI < 0.1: No significant drift, PSI < 0.2: Moderate drift, PSI > 0.2 is significant drift.

[0131] ● Earth Mover's Distance (EMD): This measures the minimum cost of transforming one distribution into another, providing a more granular comparison than KS. EMD scores can be monitored over time and compared to a baseline score. A relative EMD value close to 0 indicates highly similar distributions, and an EMD value exceeding 1 potentially indicates impactful drift.

[0132] Performance-based scoring tracks changes in performance metrics of a model based on new data encountered in production and can include the following:

[0133] ● Changes in accuracy, precision, recall, F1 score, AUC-ROC: Monitoring the absolute or relative changes in these metrics over time can indicate potential drift.

[0134] ● Mean Squared Error (MSE), Root Mean Squared Error (RMSE): These error metrics can reveal performance biases in regression models.

[0135] ● Quantile loss: Evaluating changes in quantiles of the error distribution of predictions helps identify changes in prediction behavior.

[0136] If at 810 it is determined that the base model of the CDP does not show signs of drift, method 800 proceeds to 814. At 814, method 800 includes continuing to evaluate the health equity of the CDP, and method 800 ends. Alternatively, if at 810 it is determined that the base model of the CDP has drifted beyond the threshold allowed drift amount, method 800 proceeds to 811.

[0137] At 811, method 800 includes displaying model drift on a display device. For example, a drift score, a textual description of the drift, and / or one or more visualizations of the drift can be displayed. The display device can be the display device of the health equity assessment system (e.g., display device 134), or the model drift can be displayed within the CDP list in the CDP catalog (e.g., CDP catalog 404) on the display device of a personal computing device of the CDP owner, clinician, researcher, data scientist, or different individual. The visualization can be generated using one or more visualization techniques, such as the following as a non-limiting list:

[0138] ● Histogram: Visually compares the shape and spread of feature distributions. Overlapping or shifted histograms imply drift.

[0139] ● Box plot: Compares the median, quartiles, and outliers of the feature distributions in two datasets. Box plots with significant differences indicate drift.

[0140] ● Time series plot: Plots key metrics such as accuracy or error over time to visually identify trends and sudden changes that may indicate drift.

[0141] At 812, method 800 includes storing the evaluated degree of model drift in the database of the health equity assessment system and / or notifying the owner of the CDP and / or other stakeholders in the health equity assessment system, and method 800 ends.

[0142] Returning to method 600, at 614, performing a health equity assessment includes calculating a multi-dimensional confidence score for the CDP based on the outputs of one or more difference assessment models. Each dimension of the multi-dimensional confidence score can correspond to a health disparity factor.

[0143] As an example, in one embodiment, input data can be extracted from a structured text document and input into the input layer of a CNN, and the CNN can output confidence scores for one or more difference factors, where each confidence score indicates the degree of confidence that the output of the CDP does not suffer from algorithmic bias with respect to the corresponding difference factor (e.g., in the range of 1 to 10). For example, the CNN can output a first confidence score of 9 for the difference factor "gender", indicating a high degree of confidence that the output of the CDP is not biased towards a given gender. The CNN can output a second confidence score of 6 for the difference factor "race", indicating a lower degree of confidence that the output of the CDP is not biased towards patients of a given race (e.g., there is algorithmic bias). The CNN can output a third confidence score of 10 for the difference factor "age", indicating a high degree of confidence that the output of the CDP is not biased towards patients in a certain age range; and so on. In this way, the multi-dimensional confidence score can include various confidence scores output by the CNN.

[0144] In other embodiments, different dimensions (e.g., confidence scores) of the multi-dimensional confidence score can be generated by different difference assessment models. That is, a first model can be used to generate a confidence score for algorithmic bias regarding race; a second model can be used to generate a confidence score for algorithmic bias regarding gender; a third model can be used to generate a confidence score for algorithmic bias regarding SES; and so on. The first, second, and third models can be different models or different types of models, or can include the same model or the same type of model. For example, the first model can be a first statistical model; the second model can be a CNN; the third model can be a second statistical model; etc.

[0145] Further, in some embodiments, various models can be applied to evaluate algorithmic bias with respect to a single differential factor, and the output (e.g., confidence score) of the best performing model can be selected. Alternatively, a confidence score for the differential factor can be generated based on the consistency between the outputs of various models.

[0146] At 616, method 600 includes storing the multi-dimensional confidence score of the CDP in a database of the health equity assessment system (e.g., Figure 1 confidence score database 120). For example, within a clinical workflow application, via a hospital network or the Internet, users of the CDP can access this database. Users can include clinicians, researchers, product owners, data scientists, or other types of users, as described above with reference to Figure 4 . For example, the multi-dimensional confidence score and / or other health equity assessment data can be retrieved from the database and displayed in the CDP catalog, as described above with reference to Figure 7 .

[0147] Now referring to Figure 7 , an exemplary method 700 for displaying health equity information generated by a health equity assessment system such as health equity assessment system 102, such as Figure 1 , is shown. The health equity information can be displayed together with the CDP in the CDP catalog and can include a multi-dimensional confidence score that can be generated according to the above-described method 600. In one embodiment, method 700 is executed by a processor of a hospital information system, such as processor 222 of healthcare information system 220, such as Figure 2 . In other embodiments, method 700 can be executed by a processor of the health equity assessment system (e.g., processor 104) or a processor of a different system.

[0148] Method 700 begins at 702, where method 700 includes receiving a request to view the CDP catalog. For example, a clinician may wish to select a suitable CDP for analyzing data of a patient or group of patients, or a researcher may wish to review the CDPs in the CDP catalog to obtain evidence of algorithmic bias in the CDP.

[0149] At 704, method 700 includes displaying the CDP catalog on a display screen. In various examples, the CDP catalog can be displayed within a clinical workflow application or a GUI of a hospital information system used by clinicians, researchers, and / or other users.

[0150] At 706, method 700 includes retrieving health equity assessment data associated with a CDP included in a CDP catalog from a health equity assessment system. In other words, for each CDP listed in the CDP catalog, health equity assessment data collected and generated by the health equity assessment system can be retrieved. The health equity assessment data for a CDP can include a multi-dimensional confidence score that indicates, for one or more differential factors (e.g., race, gender, age, SES, etc.), the degree of confidence of the health equity assessment system that the CDP is not algorithmically biased towards a subset of patient populations with respect to the differential factor. The health equity assessment data for a CDP can also include other data, such as patient cases or patient cohort summaries generated for patient data analyzed using the CDP.

[0151] Other health equity assessment data can also include protected characteristic labels (race and ethnicity, gender, age, disability, nationality, marital status, pregnancy or childbirth, veteran status, sexual orientation, and genetic information), as well as data from external data sources such as demographic data (e.g., census data, health surveys, socioeconomic indicators); disease prevalence and outcome data (e.g., disease incidence and severity in different groups); and / or healthcare disparity studies (e.g., existing studies on healthcare access and outcome biases).

[0152] At 708, method 700 includes displaying the retrieved health equity assessment data corresponding to a CDP in the CDP catalog together with the CDP such that the health equity assessment data can be reviewed by a user of the CDP catalog. In some examples, a first portion of the health equity assessment data can be displayed at the "top level" of the CDP catalog, such as next to (e.g., inline with) the name, identifier, or icon representing the CDP, and a second portion of the health equity assessment data can be referenced via a link or control element that can display a separate page or data panel including the information of the second portion. For example, the first portion of the health equity assessment data can include an icon, image, or text summarizing the health equity information available for the CDP, and when the icon, image, or text is selected, more comprehensive or substantial health equity information can be viewed. For example, the icon, image, or text can include a summary of the multi-dimensional confidence score associated with the CDP. In this way, a user of the CDP catalog can see a preview of the health equity assessment data when reviewing a CDP and can access more detailed data on demand.

[0153] In particular, it should be noted that when the health equity assessment system is in an unactivated state, more detailed data can be advantageously displayed. In other words, the generation of health equity assessment data can occur at a first time using the resources of the health equity assessment system. When displaying the CDP catalog, the health equity assessment data can be retrieved from the health equity assessment system at a second time using the resources mainly of the relevant healthcare information system. When the user of the CDP catalog expects to see additional data, the detailed health equity data (e.g., the second part) can be displayed at a third time. For example, the user can view the multi-dimensional confidence score displayed next to the relevant CDP, and the user can select the multi-dimensional confidence score to view the specific patient data related to the multi-dimensional confidence score, such as the patient case summary indicating the specific health outcome of the patient, the output of the CDP affecting the specific health outcome, and the health disparity factors that can be applied to the patient. Therefore, the display of the specific data related to the multi-dimensional confidence score can be independent of the health equity assessment system that is operable or accessible to the user. More specifically, the specific patient data related to the multi-dimensional confidence score can be displayed without having to request and retrieve data from the health equity assessment system. Therefore, the more detailed second part of the health equity assessment data can be displayed in the GUI in a fast and responsive manner, which can encourage the user to view the health equity assessment information in detail and consider the impact of potential algorithmic biases when selecting a CDP to perform an analysis.

[0154] Figure 11 An exemplary simplified CDP catalog 1100 is shown, which can be a Figure 2 non-limiting example of the CDP catalog 226 and / or Figure 4 the CDP catalog 404. In one embodiment, the CDP catalog 1100 is accessed via a healthcare information system network and is displayed within the GUI of an online resource or a clinical software application (such as a clinical workflow application used by a clinician), as described above with reference to Figure 7 method 700. A clinician can use the CDP catalog 1100 to search for CDPs to apply to patient data, e.g., to assist the clinician in diagnosing a patient's condition, determining a patient's treatment options, identifying patterns in patient data, generating a warning if a change in patient data is detected, or performing different clinical tasks. In some examples, the clinician can apply the CDP to the patient data of various patients (e.g., patient group 202). Additionally, the CDP catalog 1100 can be used by other types of users, such as Figure 4 clinical researchers, CDP owners, and data scientists as described with reference to

[0155] The CDP catalog 1100 includes a plurality of CDPs, which may, for example, have been licensed from a data product market (e.g., CDP market 224) integrated with a relevant clinical application. The plurality of CDPs includes a first CDP 1102 titled "Silent Hypoxemia"; a second CDP 1104 titled "Ventilator Weaning Prediction Model"; and a third CDP 1106 titled "Electrical Shock". Each of the CDPs 1102, 1104, and 1106 can perform different types of analysis on patient data. For example, the first CDP 1102 can be an intelligent warning that can be triggered in a silent hypoxemia event, similar to the respiratory depression CDP described above with reference to Figure 9 The second CDP 1104 can be a prediction model that returns a confidence score percentage indicating the degree to which a patient is ready to be weaned from the ventilator. The third CDP 1106 can be an intelligent warning triggered in an electrical shock event, and so on. The plurality of CDPs can be displayed in rows, where each row corresponds to a CDP included in the CDP catalog 1100. Each corresponding row of the CDP can include various information about the CDP and / or control elements, which can be selected to display additional information or can be displayed in different ways.

[0156] In particular, for one or more CDPs included in the CDP catalog 1100, the CDP catalog 1100 can include control elements that provide an assessment of the health equity of one or more CDPs performed by a health equity assessment system (such as Figure 1 The health equity assessment system 102). For example, the first CDP 1102 includes a first health equity assessment control element 1103, which can provide an indication of the confidence of the health equity assessment system that the algorithmic bias of the first CDP 1102 is below a threshold level. The threshold level can be a level below which the output of the first CDP 1102 may not be considered accurate for some patients, patient types, or patient groups. In other words, the first CDP 1102 can be developed or trained based on patient data from a first narrow patient population, where the first narrow patient population may not be representative of a broader patient population. Similarly, the second CDP 1104 includes a second health equity assessment control element 1105, which can provide an indication of the confidence of the health equity assessment system that the algorithmic bias of the second CDP 1104 may be at or above the threshold level, and the third CDP 1106 includes a third health equity assessment control element 1107, which can provide an indication of the confidence of the health equity assessment system that the algorithmic bias of the third CDP 1106 may be above the threshold level.

[0157] Specifically, the first health equity assessment control element 1103 includes a "thumbs up" icon 1130, which can be a visual indication of the health equity assessment system's confidence that the algorithmic bias of the first CDP 1102 is below a threshold level. The first health equity assessment control element 1103 may also include a first confidence score (e.g., 95%) 1132 generated by the health equity assessment system. For example, the first confidence score can be generated by following Figure 6 one or more steps of method 600. In the depicted embodiment, the first confidence score 1132 is a single value, where the single value can be a summary confidence score calculated based on a multi-dimensional confidence score output by the health equity assessment system.

[0158] For example, the health equity assessment system can generate a multi-dimensional confidence score that includes a first confidence dimension score based on an assessment of the algorithmic bias of the patient race analyzed using the first CDP 1102; a second confidence dimension score based on an assessment of the algorithmic bias of the patient gender analyzed using the first CDP 1102; and a third confidence dimension score based on an assessment of the algorithmic bias of the patient SES analyzed using the first CDP 1102. In one embodiment, the first confidence score 1132 can be the average of the first confidence dimension score, the second confidence dimension score, and the third confidence dimension score. In another embodiment, the first confidence score 1132 can be calculated based on the first confidence dimension score, the second confidence dimension score, and the third confidence dimension score in a different manner. For example, the first confidence score 1132 can be the lowest confidence score among the first confidence dimension score, the second confidence dimension score, and the third confidence dimension score. In yet other embodiments, the multiple confidence dimension scores of the multi-dimensional confidence score can be displayed within the first health equity assessment control element 1103.

[0159] Thus, a user of the CDP catalog 1100 can see at a glance a preview of the health equity assessment data for the first CDP 1102. The thumbs up icon 1130 can indicate that the overall algorithmic bias of the CDP 1102 may be low (e.g., below a threshold), and the confidence score 1132 can provide a numerical confirmation of a very high confidence that the CDP 1102 does not suffer from algorithmic bias.

[0160] Similar icons and confidence scores are shown for the second CDP 1104 and the third CDP 1106. However, the second CDP 1104 indicates a confidence score 1135 of 60%, which may be near a threshold level at which the output of the first CDP 1102 may not be considered accurate for some patients, patient types, or patient groups. Thus, the thumb lateral icon 1134 is shown. The third CDP 1106 indicates an even lower confidence score 1137 of 25%, which may be below the threshold level, whereby the thumb down icon 1136 is shown.

[0161] However, the ability of the confidence score or health equity assessment to accurately reflect the probability of algorithmic bias in the CDP can depend on the amount of patient data considered in the health equity assessment performed by the health equity assessment system. For example, a first health equity assessment based on patient data from hundreds of patients may be more accurate than a second health equity assessment based on patient data from one or a small number of patients. Alternatively, the CDP may have a high confidence score based on data from hundreds of patients, but these hundreds of patients may be drawn from a narrow patient population. For example, the high confidence score may be based on a large amount of patient data from Caucasian patients, but a very small amount of patient data from African American patients, where the analysis generated by the CDP for African American patients may not be as accurate as the analysis generated by the CDP for Caucasian patients. Thus, determining whether the algorithmic bias of the CDP is likely to be significant enough to render the CDP unusable for a given patient, or determining whether the parameters of the CDP can be changed to reduce the algorithmic bias, may include reviewing the specific patient data used by the health equity assessment system to generate the assessment and / or confidence score.

[0162] To allow users of the CDP directory 1100 to review specific patient data, a case count element 1110 may be included for one or more CDPs, which indicates the number of patients or patient cases considered by the health equity assessment system in the generation of the health equity assessment and / or confidence score. For example, in the depicted embodiment, the second CDP 1104 includes a second health equity assessment control element 1105, which includes a second confidence score (e.g., 60%) 1135 generated by the health equity assessment system, where the case count element 1110 indicates that the 60% confidence score shown for the second CDP 1104 is based on four patient cases. The user may select the case count element 1110 or the second health equity assessment control element 1105 to view the four patient cases.

[0163] When the user selects the case count element 1110, a patient case summary panel 1112 can be displayed (e.g., as a pop-up window, or overlaid on the CDP directory 1100, or in an expanded portion of a row of the CDP directory 1100, etc.). The patient case summary panel 1112 can include various patient case summaries, which can be displayed in a scrollable display, or in a series of pages, or in a series of nested pages that can be drilled into. In the depicted example, the patient case summary panel 1112 can include a first patient case summary 1120 of four patient cases and a second patient case summary 1122 of four patient cases, with the remaining two patient case summaries on Figure 12 subsequent pages shown in

[0164] In various embodiments, the first patient case summary 1120 and the second patient case summary 1122 can be generated by a clinical context summarizer of the health equity assessment system (such as Figure 1 clinical context summarizer 116). The clinical context summarizer can use an LLM to generate the first patient case summary 1120 and the second patient case summary 1122 (and other patient case summaries), as described above.

[0165] The user can first review the four patient case summaries to determine the basis for potential algorithmic bias indicated in the second confidence score 1135. The user can see that the first patient case summary 1120 pertains to a 58-year-old Caucasian male patient, in whom no algorithmic bias was detected. The user can see that the second patient case summary 1122 pertains to a 42-year-old Caucasian female patient, in whom no algorithmic bias was detected. As a result of no algorithmic bias being detected, a thumbs-up icon 1121 can be displayed adjacent to the first patient case summary 1120 and the second patient case summary. Additionally or alternatively, an algorithm summary statement 1123 can be displayed for one or more of the patient case summaries.

[0166] The user can view additional patient cases by selecting a "more" link 1124 or a similar control element, which can cause one or more new patient cases to be displayed in subsequent pages of the patient case summary panel 1112. In other examples, the user can scroll down the patient case summary panel 1112 by manipulating a scroll bar as known in the art, or additional case summaries can be displayed in a different manner.

[0167] Figure 12Shows a first scroll view 1200 of a simplified CDP directory 1100, where a third patient case summary 1202 of four patient cases is displayed on a patient case summary panel 1112. The user can review the third patient case summary 1202 to determine the basis for the potential algorithmic bias indicated in the second confidence score 1135. The user can see that a possible algorithmic bias has been detected, which can be indicated by the relevant algorithm summary statement 1123 and / or the thumbs-down icon 1206. The user can see that the third patient case summary 1202 pertains to a 35-year-old Hispanic male, where the ventilator weaning model predicted that the weaning trial should be performed with 92% confidence, and yet, despite this, the extubation was unsuccessful. Thus, the clinician concludes that the algorithmic bias may be the cause of the incorrect prediction, where the population used to train the ventilator weaning model may not have included a significant number of non-white subjects during training.

[0168] The user can similarly review the fourth patient case summary 1204 to determine the basis for the potential algorithmic bias indicated in the second confidence score 1135. The user can see that a possible algorithmic bias has been detected, which can be indicated by the relevant algorithm summary statement 1123 and / or the thumbs-down icon 1206. The user can see that the fourth patient case summary 1204 pertains to a 63-year-old African American female, where the ventilator weaning model predicted that the weaning trial should be performed with 89% confidence, and yet, despite this, the spontaneous breathing trials repeatedly failed, leading to long-term ventilator dependence and hospitalization. Thus, the clinician concludes that the algorithmic bias may be the cause of the incorrect prediction, where the population used to train the ventilator weaning model may not have included a significant number of non-white subjects during training.

[0169] In this way, the user can determine the reason for the relatively low confidence score of 60% in the second confidence score 1135: out of four cases, the ventilator weaning prediction model made accurate predictions for two Caucasian patients and inaccurate predictions for two non-Caucasian patients. Thus, the user can take one of several actions. The user can be a clinician and can decide not to use the second CDP 1104 for the user's patient due to potential flaws in the model (e.g., algorithmic bias). Alternatively, the clinician can decide not to use the second CDP 1104 for patients of color and use the second CDP 1104 for Caucasian patients. In a third scenario, the clinician can decide to use the second CDP 1104 for non-Caucasian patients, but the clinician can interpret the results of the second CDP 1104 cautiously to generate more data from which a more accurate assessment of the second CDP 1104 can be made, as there may not be enough information to infer the algorithmic bias and confidence due to the small number of available cases.

[0170] In some embodiments, additional health equity assessment data can be displayed or provided by selecting the case count element 1110 or the second health equity assessment control element 1105. For example, a user can select the case count element 1110 to view a text summary of the analyzed patient cases to generate a confidence score 1135, and the user can select the second health equity assessment control element 1105 to view additional health equity assessment data associated with the confidence score 1135. For example, the patient data used to generate the health equity assessment of the second CDP 1104 may include too many patient cases to be effectively displayed in the patient case summary panel 1112. In such a case, a text summary of the patient data can be displayed in the patient case summary panel 1112 (e.g., instead of individual patient cases), and additional information on how the confidence score 1135 was generated can be displayed in an alternative or additional display panel. The additional information (e.g., the second part described above with reference to method 700) can include, for example, statistical and / or demographic data showing how the total patient population analyzed by the health equity assessment system can be broken down into subgroups representing different differential factors. The additional information can include a multi-dimensional confidence score that shows different confidence dimensions corresponding to different differential factors. The additional information can include visual elements that can summarize the patient data, such as graphs, charts, tables, etc. Additionally, the additional information and / or visual elements can also be selected by the user such that the user can browse the health equity assessment data (including patient data) in a "drill-down" manner, where data with increasing granularity is displayed based on the user selection.

[0171] In addition, it should be understood that when generating or displaying the CDP catalog 1100, health equity assessment scores, ratings, patient case summaries, and additional relevant data can be retrieved from the health equity assessment system. When the CDP catalog 1100 is displayed, a preview or summary of the health equity assessment data can be shown at the top level, meaning next to the CDPs shown in the CDP catalog 1100. The user can then view more detailed health equity assessment data by selecting the preview, and the more detailed health equity assessment data can be shown to the user without communicating with the health equity assessment system. For example, the health equity assessment system can be in an inactive state and inaccessible to the user after the CDP catalog 1100 is generated. In this way, the operating efficiency of the computing device displaying the CDP catalog 1100 can be increased. That is, in an alternative scenario where the CDP catalog 1100 is generated with previewed, summarized health equity assessment data rather than more detailed health equity assessment data, to view the more detailed health equity assessment data, the user would select the previewed, summarized health equity assessment data (e.g., confidence score), and the more detailed health equity assessment data would be retrieved from the health equity assessment system on demand. As a result of retrieving the more detailed health equity assessment data on demand, when the user is evaluating the algorithmic bias of a CDP, the consumption of the processing power of the computing device will increase, which may increase the amount of time it takes for the user to evaluate the algorithmic bias and impede the user from requesting additional data.

[0172] In other words, the method disclosed herein for generating and displaying health equity assessment information at a level of detail desired by a user improves the ability of both a health equity assessment system and a healthcare information system that displays a CDP catalog by reducing the amount of processing that would otherwise be performed on demand when compiling various elements of health equity assessment data from a hospital database or retrieving health equity assessment data from a health equity assessment system. Health equity assessment data is generated by the health equity assessment system using one or more disparity assessment models in a first step. A first portion of the health equity assessment data, which may be a preview summarizing the health equity assessment data, is associated with the CDP in a second step when the CDP catalog is displayed or populated with CDPs. When a CDP is selected, a more detailed second portion of the health equity assessment data corresponding to the preview can be displayed to the user in a third step, which may be at a later time during which the health equity assessment system may be inaccessible or even not running. Thus, if a user wishes to view the more detailed second portion of the health equity assessment data corresponding to the preview, the detailed health equity assessment data can be quickly generated with minimal processing via controls presented in the GUI. In this way, the dynamic display of health equity assessment data improves the way in which both the health equity assessment system and the healthcare information system store and retrieve data in memory to reduce resource consumption. Specific ways of displaying health equity assessment data to a user based on a limited set of data are described such that the user is not burdened by time-consuming iterative calculations or browsing through pages of health equity assessment data aggregated for different patients or products. Because the user is not forced to scroll down or browse through layers of data to view the health equity assessment data, a fast and efficient process for reviewing potential algorithmic bias in CDPs is achieved. Thus, CDPs with a lower probability of algorithmic bias can be selected and within a shorter time frame compared to what is allowed by prior art systems and GUIs. Accordingly, the disclosed invention improves the efficiency of the health equity assessment system and healthcare information systems that rely on the health equity assessment system.

[0173] The technical effect of collecting specific disparity factor data applicable to assessing CDPs for algorithmic bias and converting the data into a highly standardized format is that, compared to current methods that rely on screening and cross-referencing data already stored in different databases of a healthcare system, the memory and processing resources of the healthcare information system used to analyze data for algorithmic bias can be reduced.

[0174] The present disclosure also provides support for a health equity assessment system that includes: a processor and a non-transitory memory storing instructions that, when executed, cause the processor to: summarize the clinical context of the analysis performed on patient data by a clinical data product (CDP); based on the summarized clinical context, perform a health equity assessment of the CDP using one or more differential assessment models trained to evaluate algorithmic bias of the CDP for a specific patient population; calculate a confidence score for the CDP based on the health equity assessment, the confidence score indicating the degree of confidence that patient analysis using the CDP is not subject to algorithmic bias; and store the confidence score in a database. In a first embodiment of the system, the summarized clinical context of the analysis performed on the patient data by the CDP includes demographic data on a plurality of differential factors for the patient population in which the algorithmic bias can be detected, the differential factors including at least race, gender, age, disability, social determinants of health (SDOH), and socioeconomic status (SES). In a second embodiment of the system, optionally including the first embodiment, the confidence score is a multi-dimensional confidence score that includes a plurality of confidence scores for the respective plurality of differential factors. In a third embodiment of the system, optionally including one or both of the first and second embodiments, the clinical context of the analysis performed on patient data by the CDP is summarized by a large language model (LLM) that generates a natural language text summary of the patient data. In a fourth embodiment of the system, optionally including one or more or each of the first to third embodiments, the clinical context of the analysis performed on patient data by the CDP is summarized as a structured text file including tagged data in a standardized format, the tagged data being used as input data for the differential assessment model. In a fifth embodiment of the system, optionally including one or more or each of the first to fourth embodiments, additional instructions are stored in the non-transitory memory that, when executed, cause the processor to perform the health equity assessment of the CDP in response to receiving clinical feedback from the clinician on the output of the CDP during the analysis of the patient data performed by the clinician of a healthcare system using the CDP, the received clinical feedback including demographic data on the plurality of differential factors. In a sixth embodiment of the system, optionally including one or more or each of the first to fifth embodiments, the clinical feedback includes a health equity rating of the CDP, the health equity rating being generated by the clinician via a CDP rating tool micro-application of the health equity assessment system, the CDP rating tool micro-application being opened within a software application used by the clinician to perform the analysis of the patient data using the CDP.In a seventh embodiment of the system, optionally including one or more or each of the first through sixth embodiments, the CDP rating tool micro - application displays: a first control element that, when selected, sends a positive rating of the output of the CDP to the health equity assessment system; a second control element that, when selected, sends a negative rating of the output of the CDP to the health equity assessment system; a third control element that, when selected, sends an indication to the health equity assessment system that there is not enough information available to rate the output of the CDP; and a fourth control element that allows a user of the CDP rating tool micro - application to send text data regarding the output of the CDP to the health equity assessment system. In an eighth embodiment of the system, optionally including one or more or each of the first through seventh embodiments, the system further includes: a model drift detector configured to monitor trends in health equity assessments and confidence ratings of the CDP to determine whether a differential assessment model among the one or more differential assessment models is becoming less accurate over time due to algorithmic bias. In a ninth embodiment of the system, optionally including one or more or each of the first through eighth embodiments, the model drift detector is further configured to measure the performance of the CDP based on multiple key metrics relative to a baseline performance of the CDP for at least one of the following: changes over time in the distribution of input data used by the CDP; changes over time in the relative importance of different features of the CDP; and changes over time in the predictions or outputs of the CDP. In a tenth embodiment of the system, optionally including one or more or each of the first through ninth embodiments, these key metrics include at least one of the following: accuracy, precision, or recall of a differential assessment model; output of a drift detection algorithm; Kolmogorov - Smirnov (KS) test; Earth Mover's Distance (EMD); Jensen - Shannon Divergence (JSD); chi - square test; Wilcoxon signed - rank test; Population Stability Index (PSI); and Area Under the Receiver Operating Characteristic Curve (AUC - ROC). In an eleventh embodiment of the system, optionally including one or more or each of the first through tenth embodiments, the CDP is selected by a clinician from a CDP catalog that displays a list of CDPs available to the clinician and, for each CDP in the CDP list, a preview of the health equity assessment data available for the CDP.

[0175] The present disclosure also provides support for a machine-implemented method of a health equity assessment system for a healthcare system, the method comprising: receiving a rating of a clinical data product (CDP) used by a clinician to analyze patient data; summarizing the clinical context of the analysis performed by the CDP on the patient data; performing a health equity assessment of the CDP, the health equity assessment using one or more disparity assessment models to evaluate the summarized clinical context to assess algorithmic bias of the CDP; calculating a confidence score of the CDP based on the assessed algorithmic bias; and storing the confidence score in a database. In a first embodiment of the method, summarizing the clinical context of the analysis performed by the CDP on the patient data further comprises summarizing the clinical context into a text summary, the text summary including demographic data on a plurality of disparity factors of a patient population in which the algorithmic bias can be detected, the disparity factors including at least race, gender, age, disability, social determinants of health (SDOH), and socioeconomic status (SES). In a second embodiment of the method, optionally including the first embodiment, the confidence score is a multi-dimensional confidence score, the multi-dimensional confidence score including a plurality of confidence scores for respective plurality of disparity factors. In a third embodiment of the method, optionally including one or both of the first and second embodiments, summarizing the clinical context into the text summary further comprises using a large language model (LLM) to generate a natural language text summary of the patient data. In a fourth embodiment of the method, optionally including one or more or each of the first to third embodiments, the natural language text summary is included in a list of CDPs in a CDP catalog displayed to the clinician. In a fifth embodiment of the method, optionally including one or more or each of the first to fourth embodiments, the list of CDPs in the CDP catalog is configured to display a preview of the natural language text summary, the preview including at least the number of patient cases included in the natural language text summary, and the natural language text summary is displayed in a display panel of a graphical user interface (GUI) of the CDP catalog, and when the health equity assessment system is in an unactivated state, the display panel can be directly reached by selecting the preview.

[0176] The present disclosure also provides support for a Clinical Data Product (CDP) rating tool micro - application, which includes a rating tool launched within a software application used by clinicians to perform an analysis on patient data using the CDP. The rating tool is configured to: receive clinical feedback from the clinician regarding algorithmic bias of the CDP and send the clinical feedback to a health equity assessment system configured to evaluate the algorithmic bias of the CDP. In a first embodiment of the system, the rating tool includes: a first control element which, when selected, sends a positive rating of the output of the CDP to the health equity assessment system; a second control element which, when selected, sends a negative rating of the output of the CDP to the health equity assessment system; a third control element which, when selected, sends an indication to the health equity assessment system that there is not enough information available to rate the output of the CDP; and a fourth control element which allows a user of the CDP rating tool micro - application to send text data regarding the output of the CDP to the health equity assessment system.

[0177] When introducing elements of the various embodiments of the present disclosure, the articles “a”, “an” and “the” are intended to mean that there is one or more of such elements. The terms “first”, “second”, etc. do not denote any order, quantity or importance, but are used to distinguish one element from another. The terms “comprising”, “including” and “having” are intended to be inclusive and mean that additional elements may exist in addition to the listed elements. As used herein, terms such as “connected to”, “coupled to” etc., an object (e.g., a material, an element, a structure, a component, etc.) may be connected to or coupled to another object, regardless of whether the one object is directly connected or coupled to the other object, or whether there are one or more intervening objects between the one object and the other object. Further, it should be understood that references to “an embodiment” or “embodiments” of the present disclosure are not to be construed as excluding the existence of additional embodiments that also incorporate the recited features.

[0178] In addition to any previously indicated modifications, those skilled in the art can design many other variations and alternative arrangements without departing from the spirit and scope of this specification, and the appended claims are intended to cover such modifications and arrangements. Thus, although the information has been specifically and detailedly described above in connection with the currently considered most practical and preferred aspects, it will be apparent to those of ordinary skill in the art that many modifications can be made without departing from the principles and concepts set forth herein, including but not limited to form, function, mode of operation and use. Also, as used herein, in all respects, embodiments and implementations are meant to be illustrative and should not be construed as limiting in any way.

Claims

1. A health equity assessment system (102, 214), the health equity assessment system comprising: a processor (104) and a non-transitory memory (106) storing instructions, the instructions when executed causing the processor (104) to: summarize the clinical context of the analysis performed on patient data (902, 204) by a clinical data product (CDP (904, 1102, 206, 175, 302, 402)); perform a health equity assessment (408) of the CDP (904, 1102, 206, 175, 302, 402) using one or more difference assessment models (110) trained to evaluate algorithmic bias of the CDP (904, 1102, 206, 175, 302, 402) for a specific patient population based on the summarized clinical context; calculate a confidence score (1132, 1135) of the CDP (904, 1102, 206, 175, 302, 402) based on the health equity assessment (408), the confidence score (1132, 1135) indicating the degree of confidence that patient analysis using the CDP (904, 1102, 206, 175, 302, 402) is not subject to algorithmic bias; and store the confidence score (1132, 1135) in a database.

2. The health equity assessment system (102, 214) according to claim 1, wherein the summarized clinical context of the analysis performed on the patient data (902, 204) by the CDP (904, 1102, 206, 175, 302, 402) includes demographic data on a plurality of difference factors of a patient population capable of detecting the algorithmic bias, the difference factors including at least race, gender, age, disability, social determinants of health (SDOH), and socioeconomic status (SES).

3. The health equity assessment system (102, 214) according to claim 2, wherein the confidence score (1132, 1135) is a multi-dimensional confidence score (1132, 1135), the multi-dimensional confidence score including a plurality of confidence scores for the respective plurality of difference factors.

4. The health equity assessment system (102, 214) according to claim 1, wherein the clinical context of the analysis performed on patient data (902, 204) by the CDP (904, 1102, 206, 175, 302, 402) is summarized by a large language model (LLM) that generates a natural language text summary of the patient data (902, 204).

5. The health equity assessment system (102, 214) according to claim 1, wherein the clinical context of the analysis performed on patient data (902, 204) by the CDP (904, 1102, 206, 175, 302, 402) is summarized as a structured text file including tagged data in a standardized format, the tagged data being used as input data for a difference assessment model.

6. The health equity assessment system (102, 214) according to claim 2, wherein additional instructions are stored in the non-transitory memory (106), and when executed, the instructions cause the processor (104) to perform the health equity assessment (408) of the CDP (904, 1102, 206, 175, 302, 402) in response to clinical feedback received from a clinician (416, 207) of a healthcare system regarding the output of the CDP (904, 1102, 206, 175, 302, 402) during the analysis performed on the patient data (902, 204) using the CDP (904, 1102, 206, 175, 302, 402), and the received clinical feedback includes demographic data regarding the plurality of differential factors.

7. The health equity assessment system (102, 214) according to claim 6, wherein the clinical feedback includes a health equity rating of the CDP (904, 1102, 206, 175, 302, 402), the health equity rating being generated by the clinician (416, 207) via a CDP rating tool micro - application of the health equity assessment system (102, 214), and the CDP rating tool micro - application is opened within a software application used by the clinician (416, 207) to perform the analysis on the patient data (902, 204) using the CDP (904, 1102, 206, 175, 302, 402).

8. The health equity assessment system (102, 214) according to claim 7, wherein the CDP rating tool micro - application displays: a first control element (1002) that, when selected, sends a positive rating of the output of the CDP (904, 1102, 206, 175, 302, 402) to the health equity assessment system (102, 214); a second control element (1004) that, when selected, sends a negative rating of the output of the CDP (904, 1102, 206, 175, 302, 402) to the health equity assessment system (102, 214); a third control element (1006) that, when selected, sends an indication to the health equity assessment system (102, 214) that there is not enough information available to rate the output of the CDP (904, 1102, 206, 175, 302, 402); and a fourth control element (1008) that allows a user of the CDP rating tool micro - application to send text data regarding the output of the CDP (904, 1102, 206, 175, 302, 402) to the health equity assessment system (102, 214).

9. The health equity assessment system (102, 214) according to claim 6, further comprising a model drift detector (114) configured to monitor trends in the health equity assessment (408) and the confidence ratings of the CDP (904, 1102, 206, 175, 302, 402) to determine whether the differential assessment model in the one or more differential assessment models (110) is gradually becoming less accurate due to algorithmic bias.

10. The health equity assessment system (102, 214) according to claim 9, wherein the model drift detector (114) is further configured to measure the performance of the CDP (904, 1102, 206, 175, 302, 402) based on multiple key metrics relative to a baseline performance of the CDP (904, 1102, 206, 175, 302, 402) with respect to at least one of the following: Changes over time in the distribution of input data used by the CDP (904, 1102, 206, 175, 302, 402); Changes over time in the relative importance of different features of the CDP (904, 1102, 206, 175, 302, 402); and Changes over time in the predictions or outputs of the CDP (904, 1102, 206, 175, 302, 402).

11. The health equity assessment system (102, 214) according to claim 10, wherein the key metrics include at least one of the following: Accuracy, precision, or recall of the differential assessment model; Output of the drift detection algorithm; Kolmogorov - Smirnov (KS) test; Earth Mover's Distance (EMD); Jensen - Shannon Divergence (JSD); Chi - square test; Wilcoxon signed - rank test; Population Stability Index (PSI); and Area Under the Receiver Operating Characteristic Curve (AUC - ROC).

12. The health equity assessment system (102, 214) according to claim 1, wherein the CDP (904, 1102, 206, 175, 302, 402) is selected by a clinician (416, 207) from a CDP catalog (226, 176, 404, 1100) that displays a list of CDPs (1102, 206, 175, 402) available to the clinician (416, 207), and for each CDP (904, 1102, 206, 175, 302, 402) in the CDP list, a preview of the health equity assessment data available for the CDP (904, 1102, 206, 175, 302, 402).

13. A machine - implemented method (600) for a health equity assessment system of a healthcare system, the method comprising: Receiving a rating (602) of a Clinical Data Product (CDP) used by a clinician to analyze patient data; Summarize the clinical context (604) of the analysis performed by the CDP on the patient data; Perform a health equity assessment (608) of the CDP, the health equity assessment using one or more disparity assessment models to evaluate the summarized clinical context to assess algorithmic bias of the CDP; Calculate a confidence score (614) of the CDP based on the evaluated algorithmic bias; and Store the confidence score in a database (616).

14. The machine-implemented method (600) according to claim 13, wherein the clinical context that summarizes the analysis performed by the CDP on the patient data further includes summarizing the clinical context into a text summary, the text summary including demographic data on a plurality of disparity factors of a patient population in which the algorithmic bias can be detected, the disparity factors including at least race, gender, age, disability, social determinants of health (SDOH), and socioeconomic status (SES).

15. The machine-implemented method according to claim 14, wherein the confidence score is a multi-dimensional confidence score, the multi-dimensional confidence score including a plurality of confidence scores for corresponding respective plurality of disparity factors.

Images