Clinical endpoint determination system and method

Abstract

The present disclosure relates to systems and methods used in performing clinical endpoint adjudication that provide efficient, automated clinical event classification and medical review triage, reduce the time to identify clinical events, provide a unified and consistent process for classifying clinical events, and provide pre-identification of events in near real-time.

Description

Technical Field

[0001] The present disclosure relates to systems and methods used in performing clinical endpoint determinations.

Background Art

[0002] Outcome trials are a legal requirement for Cardiovascular, Renal, and Metabolic (CVRM) projects to demonstrate safety and efficacy / benefit. Outcome trials require a large sample size (thousands of patients) and are expensive to conduct. Clinical endpoint event determination is the process by which an independent blind expert committee reviews clinical events that occur during a trial. These are rated-determined-compared against a set of predefined criteria to classify the events. This is because otherwise, the local principal investigators or physicians may have their own opinions about the type of event the patient had, which may lead to a high degree of variation. By using a blind expert committee, this variation can be reduced. Endpoint determination can be used to rate both efficacy outcomes and safety outcomes and has been used by all major pharmaceutical companies across the industry for 25 - 30 years. Endpoint determination provides an independent review of events and avoids regional differences and bias from individual principal investigators.

[0003] However, event determination represents 5% of the average CVRM outcome trial cost. Event determination is a manual and iterative process. Event determination requires highly skilled clinicians who are occupied by events that are easy to evaluate. Event determination generates duplicate determination data across multiple systems. Therefore, event determination is time-consuming and costly for the sponsor and may delay the drug development life cycle.

[0004] Clinical endpoint event determination is expensive and resource-intensive (approximately $8.5 million per large-scale CVRM outcome study). It can result in delays of approximately 4-5 months, including event capture and manual determination processes. Clinical endpoint event determination relies on secondary or tertiary reporting and is largely a manual, iterative process. Capture of all events is essential because missing events can impact trial outcomes. This problem is exacerbated when large-scale studies are conducted across multiple centers and countries. [Overview of the Initiative] [Means for solving the problem]

[0005] Aspects of the present invention are described in the independent claims, and optional features are described in the dependent claims. Aspects of the present invention can be provided in association with each other, and features of one aspect may be applicable to another aspect.

[0006] A first aspect of this disclosure provides a computer-aided method for performing clinical trial endpoint determination. The method includes receiving data from multiple healthcare-related data sources in a computing system or device. Optionally, the method includes analyzing each data source to determine whether the data held by the data source includes structured data and / or unstructured data. If the data includes unstructured data, the method includes applying a natural language processing model to the unstructured data to obtain embeddings related to features in the unstructured data. If the data includes structured data, the method includes extracting features from that data. The method further includes applying a machine learning classification model to the embeddings from the unstructured data and the features extracted from the structured data to classify whether a healthcare event has occurred based on the embeddings and the features extracted from the structured data.

[0007] Optionally, the method further includes assigning a probability score as an attribute to a classification, the probability score providing an indication of the likelihood that an event occurred, and providing the user with a notification to review the classification if the probability score is below a selected threshold.

[0008] Applying natural language processing models may involve applying multiple natural language processing models, including a first dedicated model trained on text available from a data source, along with a second general-purpose model trained on, for example, Wikipedia®.

[0009] If the data includes unstructured data, the method may include applying a named entity recognition model to the unstructured data to obtain formal event characteristics from the unstructured data, and then applying a machine learning classification model to the formal event characteristics obtained via the named entity recognition model.

[0010] The method may further include assigning a confidence score to data as an attribute based on (i) the data source and (ii) at least one of the confidence levels identified based on the optical character recognition process applied to the unstructured data, and having a machine learning classification model use the confidence scores as weights. For example, data obtained from known or trusted sources may be given a higher weight than data obtained from unknown or untrustworthy sources. In some examples, only data with weights exceeding a selected threshold may be used—for example, to avoid using erroneous or untrustworthy data that may distort the method. Thus, the method may include excluding data with confidence scores below a selected threshold.

[0011] In some examples, extracting features from data and applying natural language processing models to unstructured data to obtain feature-related embeddings in the unstructured data involves obtaining a predefined set of features for use in determining clinical endpoints, mapping the extracted features and / or embeddings to a predefined set of features, and discarding features and / or embeddings that are not related to the predefined set of features.

[0012] Machine learning classification models can provide a ranking of the importance of features involved in classifying whether or not a healthcare event has occurred. This can be useful for regulatory and / or diagnostic purposes to show how the model is performing and which features are being used to make the decision. Providing a ranking of feature importance may include identifying the SHAP value for each feature. Additionally or alternatively, providing a ranking of feature importance may include applying a local surrogate model to the machine learning classification model to identify the relative contribution of each feature to the classification.

[0013] The method may be executed if the amount of available data exceeds a selected threshold. In this way, classification can only be performed if there is enough data available to perform an endpoint determination. Additionally or alternatively, the method may be executed in response to an event occurrence indication provided by the user.

[0014] In another embodiment, a method is provided for training a machine learning classification model to perform clinical trial endpoint determination. The method includes receiving data from multiple healthcare-related data sources, the data including determination documents from previous clinical trials and determination decisions associated with those determination documents; parsing each data source to determine whether the data held by the data source includes structured data and / or unstructured data; if the data includes unstructured data, applying a natural language processing model to the unstructured data to obtain embeddings related to features in the unstructured data; if the data includes structured data, extracting features from that data; further including providing instructions for determination decisions based on the data from the determination documents; updating a machine learning classification model based on the determination decisions and the data from the determination documents; and storing the updated machine learning classification model in a relational database.

[0015] In another embodiment, a method is provided for monitoring clinical trial endpoint determinations. The method includes receiving multiple notifications of determination decisions from a clinical trial endpoint determination system, the determination decisions including a probability score that provides indication of the likelihood that an event occurred; ranking the notifications based on at least one of the probability score and (ii) the severity of the event; obtaining documentation of the data used to perform the determinations; and reviewing the accuracy of the determination decisions by providing the user with a list of the determination decisions and the corresponding documentation of the data, the order of the list being based on the ranking. Such a method may help ensure that the clinical trial endpoint determination process is kept efficient, and may help ensure, for example, that input from healthcare professionals is obtained in a timely manner if such input is required.

[0016] The method may further include obtaining a ranking of the importance of features related to the classification of whether or not a healthcare event has occurred, and providing the user with the ranking of feature importance along with a list of determinations and corresponding documents of the data. These may help identify whether any required input is overdue and / or whether input from, for example, a healthcare professional is urgently needed.

[0017] In another embodiment, a computer-aided method is provided for harmonizing and collating data from multiple healthcare-related sources for clinical trial endpoint determination. The method includes analyzing each data source to determine whether the data held by the data source includes structured data and / or unstructured data. If the data includes unstructured data, the method includes performing optical character recognition on one or more regions of the data that are not yet in a machine-readable format. The method further includes assigning confidence scores to the data as attributes based on (i) the data sources and (ii) the confidence identified based on the optical character recognition process; performing feature analysis on the data to extract features from the data; mapping the extracted features to a predefined set of features; and publishing the extracted and mapped features in JSON format for use by a machine learning model in performing clinical trial endpoint determination, wherein the confidence scores are attributes of the features.

[0018] In some examples, the method involves exposing extracted and mapped features in JSON format if the confidence score exceeds a selected confidence threshold.

[0019] The method involves obtaining a set of features necessary for determining the clinical trial endpoint, and the required set of features is obtained based on the endpoint. This involves comparing features obtained from multiple data sources with a set of features necessary for determining clinical trial endpoints to determine whether any given feature is missing or incomplete. If any feature is determined to be missing or incomplete, the user shall be provided with a notification that the feature is missing, and the notification may further include providing, or providing, an indication of the missing or incomplete feature.

[0020] It will be understood that different features will be required for determination depending on the clinical endpoint being considered (e.g., death vs. myocardial infarction). In some examples, the method may further include determining a set of features required for determining a clinical trial endpoint. The method may further include performing named entity recognition on the data and selecting formal event characteristics associated with a predefined set of features before performing feature analysis on the data.

[0021] In some examples, the method includes obtaining a set of features that a data source should (or is expected to) provide, determining whether any features are missing from that data source (for example, by comparing them to a set of expected features), and, if features are missing from that data source, providing the user with a notification that the features are missing.

[0022] In some examples, performing feature analysis on data to extract features from the data further includes checking for and removing any duplicate, inconsistent, or inappropriate features.

[0023] In another embodiment, a monitoring system is provided to determine whether a healthcare event has occurred in a participant in a clinical trial. The system comprises a communication interface configured to receive data signals related to multiple participants from multiple sources, each data signal containing information indicating the participant and related parameters, and a processor. For each participant, the processor is configured to process each received data signal and to apply a first weight to each data signal based on the source of the data signal. It will be understood that this weighting may be based on business rules—for example, this type of signal may be defined as a significant “trigger” signal, and this type of signal may be defined as a “situation” signal that provides useful information that may help determine whether an event has occurred, but cannot be used on its own to make such a determination. The processor is configured to determine the probability of a healthcare event occurring based on at least one of the following: (i) a data signal indicating that a patient and related parameter exceeds a selected threshold for that participant (e.g., the patient is within a selected range of the hospital beyond a selected time); and (ii) the first weight exceeds a selected trigger threshold (e.g., the signal is a “trigger” signal and not a “situation” signal, and / or a sufficient number of “situation” signals have been acquired to enable the determination).

[0024] The processor is configured to determine that a healthcare event has occurred based on whether a specified probability exceeds a selected threshold, and if the processor determines that a healthcare event has occurred, the processor is configured to provide a notification to the user of the monitoring system, and the monitoring system may be configured to rank the notifications based on the specified probability of the healthcare event occurring.

[0025] The processor may be configured to determine the type of healthcare event based on the source of the data signal and the parameters associated with the participant.

[0026] In some examples, the monitoring system is configured to rank notifications based on a determined type of healthcare event - for example, more severe events (e.g., as held in a lookup table, for example) can be ranked higher. Additionally or alternatively, the monitoring system can be configured to rank notifications based on the known health of the participant.

[0027] The processor can be configured to identify the probability of a healthcare event occurring based on a plurality of data signals indicating at least one of (i) a parameter associated with the patient exceeding a selected threshold and (ii) the weight of that data signal exceeding a selected threshold. For example, an algorithm or multiplication function can be used to combine the data signals in a particular manner.

[0028] In some examples, the processor identifies the probability of a healthcare event occurring based on a received data signal indicating that a parameter has exceeded a selected threshold, in which case the processor reviews information indicating the previous value of that parameter associated with the participant at a selected time interval preceding the identification.

[0029] A set of internal rules can define these selected time intervals - for example, whether previous events / data are searched for and over what period of time previous events / data are searched for. For example, the process can be configured to review, for example, blood pressure over a one-week period and heart rate over a previous 12-hour period. These windows can also vary depending on the device used (e.g., its reliability). In some examples, the processor can be configured to review past data only if a parameter exceeds a selected threshold.

[0030] In some examples, the processor can be configured to obtain multiple repeated measurements to provide a degree of verifiability. The processor can be configured to distinguish measurement errors (e.g., missing data / bad data quality) from potential safety events (e.g., a single-point abnormality in respiratory rate or a trend abnormality in weight gain).

[0031] In some cases, if the probability of an event occurring exceeds a selected threshold, the processor is configured to determine whether more information is needed from the patient, and if so, a notification is provided to the healthcare provider / system administrator to contact the patient. For example, the processor may be configured to compare the information obtained with a lookup table of known information and / or previous decisions made to determine whether it is sufficient.

[0032] The processor may also be configured to determine the reliability of a data signal and apply a second weight based on the reliability of the data signal (e.g., not uploaded, not executed correctly, poor connection, low battery), and the processor may be configured to determine the probability of a healthcare event occurring based on the data signal indicating that a patient-related parameter exceeds a selected threshold, as well as the first and second weights.

[0033] The processor may be configured to determine the probability of a participant experiencing a healthcare event, based on the probability of any event occurring previously identified for that participant.

[0034] In another embodiment, a method is provided for determining whether a healthcare event has occurred in a participant in a clinical trial. The method includes receiving data signals related to multiple participants from multiple sources, each data signal containing information indicating a parameter associated with the participant; and for each participant, processing each received data signal and applying a first weight to each data signal based on the source of the data signal. The method further includes determining the probability of a healthcare event occurring based on at least one of (i) a data signal indicating that a patient-related parameter exceeds a threshold selected for that participant, and (ii) a first weight exceeding a selected trigger threshold.

[0035] The method may further include determining that a healthcare event has occurred based on whether a identified probability exceeds a selected threshold, and providing a notification to the user if a healthcare event is determined to have occurred, with the notifications being ranked based on the identified probability of the healthcare event occurring.

[0036] The method may further include determining the type of healthcare event based on the source of the data signal and the participant and associated parameters. The method may further include ranking notifications based on the determined type of healthcare event. The method may further include ranking notifications based on the participant's known health.

[0037] The method may further include identifying the probability of a healthcare event occurring based on multiple data signals, where at least one data signal indicates that (i) a patient-related parameter exceeds a selected threshold, and (ii) the weight of that data signal exceeds a selected threshold.

[0038] The method may further include identifying the probability of a healthcare event occurring based on received data signals indicating that a parameter exceeds a selected threshold, and the identification is based on information indicating the previous value of that parameter in relation to the participant in a selected time interval preceding the identification.

[0039] In some cases, if the probability of an event occurring exceeds a selected threshold, the processor is configured to determine whether more information is needed from the patient, and if so, a notification is sent to the healthcare provider / system administrator to contact the patient.

[0040] The method may further include determining the reliability of a data signal and applying a second weight based on the reliability of the data signal, and the method includes determining the probability of a healthcare event occurring based on the data signal indicating that a patient-related parameter exceeds a selected threshold, as well as the first and second weights.

[0041] The method may further include determining the probability of a participant experiencing a healthcare event, based on the probability of any event occurring prior to the participant's identification.

[0042] In another embodiment, a monitoring system is provided for determining whether a healthcare event has occurred in a participant in a clinical trial. The system comprises a communication interface configured to receive data signals related to multiple participants from multiple sources, each data signal containing information indicating a location associated with the participant, and a processor. For each participant, the processor is configured to process each received data signal, and the processor is configured to determine the probability of a healthcare event occurring for that participant based on (i) the participant's proximity to a known healthcare center and (ii) the duration of the participant's proximity to the known healthcare center. If the processor determines that the probability of a healthcare event occurring exceeds a selected threshold, the processor is configured to send a notification to the participant requesting confirmation from the participant that a healthcare event has occurred.

[0043] In another embodiment, a method is provided for determining whether a healthcare event has occurred in a participant in a clinical trial. The method includes receiving data signals from multiple sources relating to multiple participants, each data signal containing information indicating a location associated with the participant; and for each participant, processing each received data signal to determine the probability of the participant experiencing a healthcare event based on (i) the participant's proximity to a known healthcare center and (ii) the duration of the participant's proximity to the known healthcare center. If it is determined that the probability of a healthcare event has occurred exceeds a selected threshold, the method includes sending a notification to the participant requesting confirmation from the participant that a healthcare event has occurred.

[0044] In another embodiment, a computer-readable non-temporary storage medium is provided which includes a computer program configured to cause a processor to perform any of the methods described above.

[0045] Embodiments of the present disclosure are described here merely as examples with reference to the accompanying drawings. [Brief explanation of the drawing]

[0046] [Figure 1] This shows an example of a schematic process flowchart for a conventional method of determining clinical endpoints. [Figure 2] An example of a schematic process flowchart for an example of a method for determining clinical endpoints according to an embodiment of this disclosure is shown. [Figure 3] An example of a functional overview of an event sniffer or monitoring system is shown. [Figure 4] This example shows a series of screenshots of an application running on a mobile device to provide notifications to users and to obtain input from users who can determine, for example, that they have received a healthcare event. [Figure 5] This shows an example of a schematic process flowchart for the functionality of the event sniffer module. [Figure 6] Figure 3 shows an example process flowchart of steps that the processor of the event sniffer or monitoring system may take to determine whether a potential anomaly has occurred. [Figure 7] This is an example of a process flowchart for determining whether or not an anomaly has occurred in the acquisition of healthcare data. [Figure 8] The rules used in the process example in Figure 7 to determine whether or not a healthcare event has occurred are shown below. [Figure 9] An example of a process flowchart for determining whether an abnormality or healthcare event has occurred is shown. [Figure 10] Here is another example of a process flowchart for determining whether an abnormality or healthcare event has occurred. [Figure 11] An example of a schematic diagram of a module that may be equipped with a data harmony and collection system is shown. [Figure 12]This is an example of a process flowchart showing how to import data from an unstructured data source. [Figure 13] This shows the number of fields used in each of the THEMIS, DAPA-HF, and DELIVER clinical trials, as well as the percentage of overlap in fields among the three trials. [Figure 14] This shows an example of a process flowchart for users to manually review documents that they have determined to have areas of low reliability. [Figure 15] This is an example of a high-level process flowchart of the steps involved in deploying an automated event detection module. [Figure 16] This shows an example process flowchart of a computer-based method for performing clinical endpoint determination. [Figure 17] This section provides a high-level description of the components (e.g., modules that can be implemented in software and / or hardware) and the data flow between them that can perform the example method in Figure 16 described above and train a machine learning model to perform the example method in Figure 16, along with a brief description of the function of each component. [Figure 18] Figure 2 shows an example of how the automatic event determination module can be used in combination with the data harmonization and collection module. [Figure 19A-9C] The results of three algorithms (CV death, non-CV death, and indeterminate) trained on 817 patients from the DECLARE clinical trial are shown. [Figures 20A-20B] To provide interpretability, we will show how the models or algorithms used in the examples of this disclosure can be analyzed. [Figure 21] An example graph showing the relative SHAP values ​​of various features used in one example mode is shown. [Figure 22] This shows the performance of a machine learning model across several metrics (e.g., AUC - Area Under Curve, accuracy, balanced accuracy, F1, etc.) and model versions (first three columns). [Figure 23]This section shows the performance of a machine learning model across several metrics (e.g., AUC - area under the curve, accuracy, balanced accuracy, F1, etc.) applied to data from three clinical trials. [Figure 24] A schematic block diagram of a computer system suitable for implementing one or more embodiments of this disclosure is shown. [Modes for carrying out the invention]

[0047] Figure 1 shows an example of a conventional method for determining clinical endpoints. As can be seen, clinical endpoint determination can be initiated when a health event requiring medical intervention (e.g., myocardial infarction) occurs. This triggers an alert to the endpoint office (EPO), which then conducts data collection (e.g., related to the patient and the circumstances leading to the event, medical interventions, any trials performed, etc.) and a medical review. Once this is done, a determination is made. A team of experts or a determination committee is appointed and performs the determination assessment. This may or may not consent to the medical review, and if necessary, disagreement resolution is performed before the determination outcome is determined.

[0048] As mentioned earlier, this is a time-consuming and costly process. Delays can occur in reporting events to the endpoint office, and further delays can occur in collecting data. The decision-making process itself can also take a considerable amount of time before it can be implemented, making it a very time-consuming process for the committee. Furthermore, in the case of very large-scale studies involving numerous centers and multiple countries, it is not always possible to have the same decision-making team.

[0049] This invention has identified a novel solution to address these problems. The novel solution aims to reduce the number of events sent to medical review and provide automated classification for the majority of outcome study endpoints. This was achieved by implementing an automated event determination process.

[0050] Automatic event detection can provide the following: • Efficient and automated clinical event classification and medical review triage, • Reduced time for identifying clinical events. • A unified and consistent process for classifying clinical events across TAs, and • Pre-event identification and near real-time support for IoT endpoint IDs.

[0051] There are three main embodiments or modules developed to enable automatic event determination. These three modules are shown in Figure 2. (i) Implementation of the "event sniffer" module 201, which can detect when an event occurs. (ii) Tools to harmonize acquired data and ensure that the quality of acquired data meets the requirements for event determination - “Data Harmonization and Collection” Module 203 and (iii) Machine learning methods that perform the clinical endpoint determination itself - "Automated Event Determination" Module 205 That is the case.

[0052] As can be seen in Figure 2, these three embodiments or modules can function together, thereby allowing the outputs from the event sniffer 201 and the data harmonization and collection module 203 to be used as inputs to the automated event determination module 205. For example, if the event sniffer module 201 determines that the probability of a healthcare event occurring is relatively high (e.g., exceeding a selected likelihood threshold level), the automated event determination module 205 can use the results from the event sniffer 201 and the harmonized data output from the data harmonization and collection module 203 to identify a determination outcome. It will be understood that these three embodiments or modules can be implemented individually or in combination in hardware and / or software. For example, the modules may be implemented in a computer system 2600, as described later with reference to Figure 24, and may be stored in memory 2610 or storage 2616 and implemented by a processor 2614. However, the modules may also be implemented in each computer system. It will also be understood that the modules listed above may be implemented on a remote server, for example, operating as a "cloud" and accessible via a telecommunications network such as the Internet. It should be understood that all modules may be executed on the same remote server, or on different remote servers.

[0053] These three aspects will be discussed in more detail below.

[0054] Event Sniffer Historically, researchers have struggled to obtain timely and relevant information about endpoint events (e.g., hospitalization) in cardiovascular outcome trials (CVOTs). Furthermore, some events may never be reported and therefore never known to researchers. Such data gaps and delays negatively impact data quality, the timeliness of endpoint event data collection, and the patient experience in the trial. To increase the number of unplanned hospitalizations detected and to speed up the detection of such events, "event sniffers" have been developed to monitor and analyze signals about patient status from diverse data sources in near real-time.

[0055] An "event sniffer" is a monitoring system used to determine whether or not a healthcare event has occurred in a participant in a clinical trial. An event sniffer can, for example, allow patients to proactively report that an event has occurred through the use of an application on their mobile device, as shown in Figure 4. An event sniffer can combine signals from diverse sources and, depending on the source of the data signal, can determine the probability that a healthcare event is / has occurred based on the patient parameters provided by the signal and weights applied based on the signal source. For example, the signal may be received from a patient-connected device that reports patient parameters (heart rate, blood pressure, respiratory rate, etc.), and the weights may be applied based on the source of the signal—for example, if a heart rate monitor from a known brand or manufacturer is known to be more reliable, it may be weighted more favorably than a heart rate monitor known to be less reliable.

[0056] The system can also be configured to acquire information indicating patient parameters from other sources, as well as from other sources—this is illustrated in detail in Figure 3 and will be described in more detail below with reference to, for example, Figures 5-10. For example, the system can be configured to identify when a patient visited a healthcare facility (such as a hospital) based on the amount of time spent at that location by acquiring location data and performing a search by cross-referencing it with a database of known healthcare facility locations. For example, if a user has been at a known location of a healthcare facility for more than a selected threshold amount of time, the system may determine that the user is at a healthcare facility. If the system determines that the user is at a healthcare facility, it may be configured to provide the user with a notification or alert (for example, on an app running on the user's mobile device) asking for confirmation of whether the user is at a healthcare facility and whether or not a healthcare event has occurred or occurred, as shown in Figure 4.

[0057] The advantage of an event sniffer system is that it can operate to identify clinical events in near real-time, independently of the clinical setting and medical review processes.

[0058] The Event Sniffer Module 201 receives signals from multiple sources, including a proprietary, internally generated predictive risk algorithm based on patient data available from AstraZeneca®'s Unify app, electronic medical records from healthcare institutions and / or national registries, and electronic data capture (EDC). These signals are shown in Figure 5 and will be further subdivided and described in more detail later, depending on the role each source and signal plays within the "Event Sniffer" function. Figure 5 shows a schematic process flowchart of the Event Sniffer Module's functionality, where signals from Unify are an example of real-world or "real-time" data (e.g., acquired in real-time from patients), and signals from the AIDA Sniffer are an example of pre-known data that can be acquired from medical records and databases.

[0059] Signal from Unify: A. Medical Geofence (Trigger Event Signal): An algorithm within Unify that uses the location services of a smartphone operating system to monitor when a patient enters a hospital and stays for a specified amount of time (e.g., 18 hours) in accordance with a CV-related endpoint event; the hospital geofence database is a proprietary dataset; when a geofence rule is triggered, the Unify app prompts the patient to confirm they have a medical event, and all metadata is sent to the Unify backend. This source can represent a “trigger event” signal, meaning that an event sniffer system can be triggered to determine whether a healthcare event has occurred when a signal is received from this source. It will be understood that different weightings can be applied from the “trigger event” signal to the “situation” signal. B. Patient-Reported Events (Trigger Event Signals): A feature of the Unify app that allows patients to self-report if they are potentially experiencing an endpoint event; the app also reminds patients to self-report an event once every two weeks if they have not done so; the app sends all self-reported events and associated metadata to the Unify backend. This source may represent a trigger event signal. C. Measurements from connected devices (trigger event signals): A feature of the Unify app that allows patients to self-manage measurements using connected medical devices (e.g., blood pressure cuffs). The app enables a Bluetooth connection between the device and the smartphone / tablet. The app then sends the recorded measurements, including metadata, to the Unify backend. An algorithm assesses each measurement for the measurement data and associated potential endpoint events. The algorithm can be embedded in the app on the "edge" (i.e., the app on the smart device), the Unify backend, or both. Measurement data and metadata from all connected devices, including signals for potential endpoint events, are sent by the app to the Unify backend. This source may represent trigger event signals. Devices available to patients in a trial via Unify include, for example, a. Omron® blood pressure cuff, b.Marsden (registered trademark) weighing scales, c.MightySat® Rx Pulse Oximeter There is. D. App User Statistics (Status Signals): The Unify app also collects patient user statistics (e.g., time spent on tasks, time between logins, etc.), which can be used as status signals. This means that while trends in user statistics may not in themselves trigger the event sniffer system to determine that a healthcare event has occurred, they provide additional status signals about the patient's condition at the time of the trigger (e.g., from a geofenced event).

[0060] Additional signals: E. Real-world evidence (RWE) based predictive models (situational signals): Machine learning algorithms designed to assist healthcare providers (HCPs) in prioritizing notifications about potential patient events; the algorithms are trained using RWD claim data and / or historical AZ data from similar trials. These algorithms may provide situational signals; they may provide additional situational signals about the patient's condition at trigger time (e.g., from geofenced events). F. Electronic Health Records (Trigger Event Signals): This data source consists of structured and unstructured electronic health record data received directly or indirectly (i.e., via a service such as TriNetX) from healthcare institutions that have provided electronic data capture (EDC) for a patient's past hospital visits. This data provides detailed information about the patient's hospital visits (e.g., discharge reports) that signal potential endpoint events. These signals can consist of raw signals (e.g., direct data from EHRs) and / or the results of AI algorithms trained on the above EHR data. This source can represent trigger event signals. G. Population Registries (Status Signals): This source consists of national registry data provided by several national and / or local governments for populations, including deaths. This source provides status signals; it may provide additional status signals about patient status at trigger time (e.g., from geofence events).

[0061] The Event Sniffer module is a fully automated workflow that directs signals, for example, by combining them according to trigger events into a single aggregated signal (also known as an Event Sniffer document), and then ranks those signals as potential endpoints or healthcare events. See below for details on each of these functions. • Data command and merging: When any trigger event occurs (as described above), the system is configured to scan for simultaneous (and / or recent) trigger and context signals from all other sources, combine them together into a single aggregated signal called an event sniffer document. Specific data elements scanned by the event sniffer system include: the presence of other event triggers and / or context signals; the age of all identified event triggers and / or context signals. • Event Ranking: Furthermore, the system is configured to provide an overall signal ranking, which is then provided to Unify for presentation to healthcare providers (HCPs) in the field. This ranking is performed using a combination of business rules and machine learning to evaluate and weight each signal available in the event sniffage.

[0062] Two examples of how the event sniffer module 201 can function are described below, for illustrative purposes only. Example 1 - Low-rank event: A connected device rule triggers (daily hypertension), but no other triggering events are present in the patient's documentation (no hospital geofence information; no patient-reported events). Furthermore, the RWD / RCT predictive risk algorithm indicates that the patient currently has a low / moderate risk of myocardial infarction. This information is combined in the documentation (an event sniffer digital document aggregating metadata about all events and the patient from different sources) to generate an event ranking score of 0.3, which means the event is unlikely to be an associated endpoint event. Note: The score of 0.3 is currently representative and does not reflect the output of the working algorithm. Example 2 - High-Ranking Event: A patient self-reports an event via the Unify app. This event is combined with two other trigger signals: a hospital geofence alert (triggered several hours prior) and a connected device business rule (hypertension from each of the past two days). Furthermore, an RWD / RCT predictive risk algorithm indicates that the patient is at high risk for myocardial infarction at present. This information is combined in a document (event sniffer digital document) to generate an event ranking score of 0.9, which indicates that the event is likely to be an associated endpoint event. Note: The score of 0.9 is currently representative and does not reflect the output of the working algorithm. Note: The previous triggers for connected devices and geofences imply that there was a score that existed before 0.9. The intended function is to "update" the patient's event sniffer document with relevant information for each new event and update the risk score for ranking.

[0063] Figure 3 shows an example of an event sniffer or monitoring system 2000 that may have the functions described above. The event sniffer system 2000 may be a monitoring system for determining whether or not a healthcare event has occurred in a participant in a clinical trial. System 2001 includes a communication interface 2005 coupled to a processor 2003. It will be understood that the event sniffer system 2000 may be provided on a portable electronic device (e.g., a smartphone, tablet, or laptop) or may be provided remotely and accessible, for example, via the cloud. In some examples, system 2001 may have the same or identical functions and / or components as system 2600, which will be discussed later with reference to Figure 24. In some examples, system 2001 may further include an optional location module, such as a GPS module or other means of locating system 2001.

[0064] The communication interface 2005 is configured to receive data signals related to multiple participants from multiple sources, each data signal containing information indicating the participant and associated parameters. The processor 2003 is configured to process each received data signal for each patient and to apply a first weight to each data signal based on the source of the data signal. The weighting is understood to be applicable to indicate whether a signal is a “trigger event” signal or a “situational” signal, with “trigger event” signals receiving a greater weight than “situational” signals.

[0065] The processor 2003 is further configured to identify the probability of a healthcare event occurring based on (i) a data signal indicating that a patient and related parameter exceeds a threshold selected for that participant, and (ii) at least one of a first weight that exceeds a selected trigger threshold.

[0066] For example, the thresholds selected could be the distance to a medical facility (such as a hospital) less than 100m, a heart rate exceeding 160 bpm, or a respiratory rate exceeding 30. The thresholds selected can also be based on other parameters of the patient—for example, the thresholds selected may vary with age, such as a lower heart rate threshold for older individuals than for younger ones.

[0067] In some examples, the processor 2003 may be configured to identify the probability of a healthcare event occurring based on both (i) a data signal indicating that a patient-related parameter exceeds a selected threshold for that participant and (ii) a first weight exceeding a selected trigger threshold. In this way, for example, the processor 2003 may make the identification only when a “trigger event” signal is received, for example, when the patient is within a selected distance from the healthcare facility. In this way, the processor 2003 may be configured to identify the probability of a healthcare event occurring based on a plurality of data signals, each having at least one data signal indicating at least one of (i) that a patient-related parameter exceeds a selected threshold and (ii) that the weight of that data signal exceeds a selected threshold. For example, the processor 2003 may be configured to combine the plurality of signals into an event sniffer document, as described above.

[0068] The processor 2003 may be configured to determine that a healthcare event has occurred based on whether the identified probability exceeds a selected threshold (e.g., the probability is greater than 50%, greater than 70%, greater than 90%), and if the processor 2003 determines that a healthcare event has occurred, the processor 2003 is configured to provide a notification to the user of the monitoring system (e.g., as shown in Figure 5), and the monitoring system 2000 is configured to rank the notifications based on the identified probability of the healthcare event occurring, for example, based on the patient's response to the notification.

[0069] The processor 2003 may be configured to identify the type of healthcare event based on the source of the data signal and the parameters associated with the participant. In some examples, the processor 2003 may be configured to rank notifications based on the identified type of healthcare event—for example, events judged to be more severe may be ranked higher. In some examples, the processor 2003 may be configured to rank notifications based on the participant's known health.

[0070] In some examples, when processor 2003 identifies the probability of a healthcare event occurring based on an incoming data signal indicating that a parameter has exceeded a selected threshold, processor 2003 reviews information showing the participant and the relevant parameter's previous value during a selected time interval prior to the identification. The selected time interval is variable, for example, depending on the relevant parameter. For example, processor 2003 may review blood pressure over the past week, but not heart rate over the previous 12 hours. Processor 2003 may only review the previous value of a parameter if that parameter has exceeded a selected threshold, but in other examples, it may review the previous value of one or more parameters if different parameters have exceeded the selected threshold. Doing so can not only help in identifying the type of healthcare event but also provide some degree of verifiability. As will be discussed in more detail later with reference to Figures 9 and 20, it can help distinguish between measurement errors (e.g., missing data / poor data quality) and potential safety events (e.g., a single-point anomaly in increased respiratory rate or an anomaly in weight gain trend).

[0071] If the probability of an event occurring is determined to exceed a selected threshold, the processor 2003 may be configured to determine whether more information is needed from the patient, and if more information is needed, a notification is provided to the healthcare provider / system administrator to contact the patient, for example, via an app as shown in Figure 4. For example, the processor 2003 may request information from the patient indicating a minimum set of distinct parameters, and the processor 2003 may be configured to determine whether any of that information is missing and / or not acquired sufficiently recently, for example, within a specific selected time window.

[0072] The processor 2003 can also be configured to determine the reliability of a data signal and apply a second weight based on the reliability of the data signal. The processor 2003 may be configured to determine the probability of a healthcare event occurring based on the data signal indicating that a patient-related parameter exceeds a selected threshold, as well as the first and second weights. The reliability of a data signal can be determined, for example, from metadata acquired with the signal, indicating whether the device from which the signal was acquired had a low battery level or a poor connection, or whether the data had not been recently uploaded. The determination of signal reliability will be described in more detail later with reference to Figures 6 to 10.

[0073] The processor 2003 may, in addition or alternatively, be configured to determine the probability of a participant experiencing a healthcare event based on any previously identified probability of such an event occurring for that participant.

[0074] It will also be understood that the monitoring system 2000 shown in Figure 3 may be capable of operating to determine whether a healthcare event has occurred based solely on location data and the patient's time spent at a particular location. For example, the communication interface 2005 may be configured to receive data signals related to multiple participants from multiple sources, each data signal containing information indicating the participant and the location associated with them. For each participant, the processor 2003 may be configured to process each received data signal and determine the probability of a healthcare event occurring for that participant based on (i) the participant's proximity to a known healthcare center and (ii) the duration of the participant's proximity to the known healthcare center. If the processor 2003 determines that the probability of a healthcare event occurring exceeds a selected threshold, the processor 2003 is configured to send a notification to the participant requesting confirmation from the participant that a healthcare event has occurred (as shown, for example, in Figure 4).

[0075] As described above, the processor 2003 can be configured to determine the reliability of data signals. For example, the processor 2003 may seek evidence that there is minimal variation in measurements taken from the same device by the same subject at the same time point as the endpoint of interest. One example of how this can be done is as follows: • Multiple consecutive blood pressure measurements are taken using a connected device + app (Bluetooth), and the data is sent to backend storage. • Perform at least 15 measurements within the window for 30 minutes, and repeat the experiment as needed. • If the interclass correlation coefficient of the measured values ​​is >0.9, It will succeed.

[0076] Figures 6 to 10 show methods for dealing with potential anomalies, particularly for determining whether an event is an anomaly or whether it indicates that a healthcare event has occurred. The anomaly may be an anomaly of the device that acquired the data (device anomaly) or an anomaly of the data itself (data anomaly). Figure 6 shows the steps that processor 2003 may take to determine whether a potential anomaly has occurred. First, upon receiving data via the data signal, processor 2003 determines whether there are potential measurement errors and / or potential safety events. In determining whether there are potential measurement errors, processor 2003 determines whether there is missing data and / or low-quality data. If there is missing data, processor 2003 determines whether the data is missing due to adherence (e.g., the patient did not wear the measurement device for a sufficient amount of time or the patient did not perform the scheduled measurement) or due to poor connectivity (e.g., the measurement device was not connected to the measurement application and / or the application was not connected to the backend server). If low-quality data is present, processor 2003 attempts to determine whether the device malfunctioned and / or was used improperly by the patient. For example, a potential safety event may be identified if there is a single-point abnormality and / or trend abnormality that does not conform to other data indicating other parameters of the patient. However, a potential safety event may indicate a healthcare event, and therefore in some cases, a potential safety event may trigger a determination of whether or not a healthcare event has occurred, using other data indicating other parameters of the patient, for example, from event sniffer documents.

[0077] Figures 7–10 illustrate how the monitoring system 2000 can operate to determine whether a healthcare event has occurred based on location data and the patient's time spent at a specific location, and how the system 200 can also attempt to address anomalies and inconsistencies in the data. Figure 7 shows a process flowchart of how the determination is made, and Figure 8 shows the rules used to determine whether a healthcare event has occurred. For example, as seen in Figures 7 and 8, if a patient has been in the hospital for a continuous period exceeding a selected minimum threshold time, the patient may be given a notification on the app asking them to confirm whether they have or have had an event. Figures 7 and 8 also show what happens if the patient's phone is turned off / loses signal and / or the patient leaves the hospital. In both cases, if the patient returns and / or the signal is regained within 24 hours, and the patient has been in the hospital for a non-continuous period exceeding the threshold time, the patient is again asked to confirm whether they have or have had an event. If the patient confirms, the system determines that an event occurred and assigns a high probability (e.g., over 90%) to the event. If the patient replies with a negative instruction indicating that they did not have the event, the system determines that the event did not occur and assigns a low probability (e.g., less than 10%) to the event. If the patient does not reply, the system determines that the patient may have potentially had the event and assigns an appropriate probability (e.g., 50%). Naturally, these probabilities may be adjusted by the system based on other parameters obtained from the patient as well as each data signal and weight that can be obtained from the event sniffer document.

[0078] Examples of parameters that can be obtained from a patient via data signals include: ·age, BMI, ·height, Systolic blood pressure, • Creatinine clearance (mL / min) • Glomerular filtration rate CG (mL / min / 1.7) • Glomerular filtration rate (MDRD) (mL / min / 1.7) Alkaline phosphatase (IU / L), • Apolipoprotein A1 (g / L) • Apolipoprotein B (g / L) • Glucose (mmol / L) Hemoglobin (g / L), Lymphocytes (10^9 / L) Lymphocytes / white blood cells (%) ·Neutrophils (10^9 / L), ·Platelets (10^9 / L), ·Uric acid (umol / L), ·White blood cells (10^9 / L), • Whether or not the patient is taking glycerin trinitrate and / or the dosage thereof, • Whether or not the patient is taking furosemide and / or the dosage thereof, • Whether or not the patient is taking heparin and / or the dosage thereof.

[0079] Figure 9 shows a process flow of an example of method 2400 for determining whether an anomaly or healthcare event has occurred. In step 2401, data (in this example, continuous respiratory rate) is received from the patient. In the illustrated example, the data may be received from a wearable device worn by the patient and transmitted wirelessly to a handheld device (e.g., via Bluetooth®), which may then transfer the data to a remote monitoring system operating in the cloud. In step 2403, the received data is processed to determine whether there is a device anomaly. This may include reviewing any metadata provided with the data, such as indicating that the device that transmitted the data had low battery, a poor connection, or did not obtain accurate readings. If the data is determined to contain a device anomaly, the process ends there and the data is not used. Conversely, if the data is determined not to contain a device anomaly, the data is then analyzed in step 2405 to determine whether there is a data anomaly indicating a healthcare event or an anomaly in data collection. This analysis is performed by comparing recent values ​​or a set of values ​​(e.g., acquired over a selected time interval to identify an overall trend) with previous and / or expected values ​​of the patient's data. For example, the method may involve determining the expected value of the patient's resting respiratory rate, given the patient's age and other basic health status, and comparing the received value with the expected value. The method may do this by retrieving the patient's medical history from a data store (2409). If the comparison shows a deviation greater than a selected deviation level threshold, this may indicate a potential healthcare event and / or data anomaly. In this case, the data shows a recent surge in respiratory rate exceeding 30 breaths / minute. This may indicate a potential healthcare event.

[0080] If a potential healthcare event and / or data anomaly is detected, a set of rules / actions is applied in step 2407. These rules may include providing the user with instructions or notifications asking whether a healthcare event has occurred and ensuring that data is being acquired correctly in the correct format (e.g., using the respiratory rate monitor correctly), as shown in Figure 4 for example. If the user responds by indicating that the device (in this case, the respiratory rate monitor) is being used correctly, the system may determine that there is a high probability that a healthcare event has occurred. In this example, the data shows a recent surge in respiratory rate exceeding 30 breaths / minute, so the user receives a notification asking them to verify that they are using the device (respiratory rate monitor) correctly. This notification may include instructions on how to use the device, in this example the respiratory rate monitor, correctly.

[0081] Figure 10 shows another example of a process flowchart for method 2500 for determining whether an anomaly or healthcare event has occurred. In step 2501, data (in this example, discrete weight data) is received from the patient at discrete time intervals. In the illustrated example, the data may be received by the patient entering their weight into an app running on a handheld device, and then the data may be transferred to a remote monitoring system running in the cloud or via a “smart” scale set that automatically uploads the data to the remote monitoring system. In step 2503, the received data is processed to determine whether there is a device anomaly. This may include, for example, reviewing any metadata provided with the data that indicates, for example, that the device that sent the data had a low battery level, a poor connection, or did not obtain accurate readings. If the data is determined to contain a device anomaly, the process ends there and the data is not used. Conversely, if the data is determined not to contain a device anomaly, the data is then analyzed in step 2505 to determine whether there is a data anomaly indicating a healthcare event or an anomaly in data collection. This analysis is performed by comparing recent values ​​or a set of values ​​(e.g., acquired over a selected time interval to identify an overall trend) with previous and / or expected values ​​of the patient's data. For example, the method may include determining an expected value for the patient's resting respiratory rate, given the patient's age and other basic health conditions, and comparing the received value with the expected value. The method may do this by retrieving the patient's medical history from a data store (2509). If the comparison shows a deviation greater than a selected deviation level threshold, this may indicate a potential healthcare event and / or data anomaly. In this case, the data shows a rapid increase in weight over recent time (e.g., over a series of consecutive days). This may indicate a potential healthcare event.

[0082] If a potential healthcare event and / or data anomaly is detected, a set of rules / actions is applied in step 2507. These rules may include providing the user with instructions or notifications asking whether a healthcare event has occurred and ensuring that data is being obtained correctly in the correct format (e.g., using the scale correctly). If the user responds by indicating that the device is being used correctly, the system may determine that there is a high probability that a healthcare event has occurred. In this example, the data shows a sudden increase in weight over a recent period, so the user is notified to verify that the device is being used correctly and / or to repeat the measurement. If the measurement is repeated and the same or similar results (e.g., within a selected threshold) are obtained, the system may determine that a healthcare event has occurred.

[0083] Data Harmonization As described above, a reliable and robust dataset or determination document is required to perform effective clinical endpoint determination. The problems associated with this stem from the nature of such clinical studies, the fact that they may be conducted across large geographical areas, and the dramatically varied methods by which data are acquired and recorded. Therefore, we have developed a data harmonization and collection system 203 that redesigns the clinical event data flow to support diverse data types and the extraction of significant events across data types and streaming data.

[0084] More specifically, the data harmonization and collection system 203 aims to apply machine learning to prepare and provide machine learning (ML)-ready datasets.

[0085] To achieve the above objectives, the data harmonization and acquisition system 203 may include several modules schematically shown in Figure 11: • ClinIQ Bot 1309: Software-driven intelligent automation for the assembly, curation, and quality control of clinical documents. • EDC2 Document 1302 and Document Minor 1303: A configurable clinical data extraction tool from source documents. EDC2 Document 1302 is for extracting structured data from the underlying Electronic Data Capture (EDC) system, and Document Minor 1303 is for extracting structured data from the underlying Electronic Data Capture (EDC) system. It is for extracting data from source documents. • Notification Bot 1307: A suite of microservices that provides focused functionality for research organization, notifications, digital document generation for ongoing research, document assembly, and quality control.

[0086] It will be understood that the data harmonization and collection system 203 and the modules listed above can be implemented individually or collectively in hardware and / or software. For example, a module may be implemented in computer system 2600, as described later with reference to Figure 24, for example, stored in memory 2610 or storage 2616 and implemented by processor 2614. However, the module may be implemented in each computer system. It will also be understood that the modules listed above may operate, for example, as a "cloud" and be implemented on remote servers accessible via a telecommunications network such as the Internet. It will also be understood that all modules may be implemented on the same remote server or on different remote servers.

[0087] Client event determination (CEA) is a critical component of clinical trials, using determination documents (ADs) as the primary input for making determination decisions. The current process for assembling ADs is complex, time-intensive, and involves many sub-processes. Some of these sub-processes include data extraction, document collection, quality control, and on-site follow-up. The data collection process for assembling ADs can take more than 30 days. The combination of manual processes and the time they take presents an ideal opportunity to innovate and automate the document creation workflow, which can lead to many benefits in terms of time, quality, and process simplification.

[0088] The AD, as shown in Ft, consists of data from both structured and unstructured sources. Structured data includes information such as patient profiles, medical history, and medications from the underlying EDC system. Because each study is unique, the standards and requirements for capturing this data may differ from study to study. EDC2 documentation module 1302 can create digital documents based on EDC data. Unstructured data may originate from documents such as discharge summaries, death certificates, and autopsy reports. The format of these documents may differ from country to country, and sometimes from hospital to hospital participating in the study. Documentation minor module 1303 also creates digital documents based on source documents.

[0089] The output of the document miner module 1303 can be verified using a set of quality rules and managed by the QC service. The digital document service combines these outputs to create a complete virtual judgment document 1313. Any notifications or communications necessary to resolve any missing data or document issues are managed by the notification bot module 1307.

[0090] The ClinIQ bot module 1309 is an end-to-end automated workflow that assembles a virtual decision document 1313 by integrating the EDC2 document module 1302, the document minor module 1303, and the notification bot module 1307. Figure 11 shows a high-level diagram of the end-to-end automated workflow ClinIQ bot 1309, which, once deployed, has the potential to accelerate the decision process in clinical trials and set a new industry standard for improving clinical trials by applying intelligent automation.

[0091] The EDC2 documentation module 1302 focuses on automating the process of extracting and processing structured data from completed clinical trials in an EDC system. It assembles all relevant structured EDC data into digital documents, providing a configurable and reusable pipeline to drive event determination or event detection of given events in a clinical trial. The digital documents are essentially machine learning-enabled clinical trial data in JSON format, intended to better perform event classification or event detection algorithms.

[0092] In summary, Document Minor Module 1303 focuses on data extraction from unstructured source documents, such as PDFs. Document Minor Module 1303 is a reusable tool for extracting data from free text, tables, and images present in these documents, and can be converted into ML-enabled data in JSON format using current-level ML & Optical Character Recognition (OCR) techniques, as detailed in the paper “Confidence Prediction for Lexicon-Free OCR” 28 May 2018 by Noam Mor and Lior of the School of Computer Science, Tel Aviv University, which is incorporated herein by reference. For example, the calculation of the Tesseract confidence metric is based on comparing it to a character prototype and calculating a distance metric from this representation.

[0093] In addition to data extraction, the document miner 1303 can also assess the quality of each word, table, image, and page and add their quality confidence levels to a JSON file. These JSON files are combined with the digital document created by the EDC2 document module 1302 to assemble a complete virtual judgment document 1313, which becomes the primary input to the automated event judgment system 205 ("event classifier").

[0094] Therefore, more specifically, Document Minor Module 1303 is operable to perform a computerized implementation method for harmonizing and collating data from multiple healthcare-related sources toward clinical trial endpoint determination. The method includes analyzing each data source to determine whether the data held by the data source includes structured and / or unstructured data. As shown in Figure 12, if the data includes unstructured data, optical character recognition (e.g., Tesseract) is performed on one or more regions of the data that are not yet in a machine-readable format. A confidence score is assigned to the data as an attribute based on at least one of (i) the data source and (ii) the confidence determined based on the optical character recognition process. This can be done by performing the Noam Mor and Lior Wolf methods as described above. The confidence score may be assigned at a global level, or the data source may be divided into regions and a confidence score may be assigned to each region.

[0095] Next, feature analysis is performed on the data to extract features from it, and the extracted features are mapped to a predefined set of features. Mapping is important because it ensures that the extracted data closely matches the same information used by the decision-making committee. For example, mapping can be based on what information is available in the decision documents of past studies. This is illustrated in Figure 13, which shows that different clinical studies / trials may have a different number of fields in which data can be recorded, and these fields are not common to all trials. For example, as shown in Figure 13, the DAPA-HF trial has 17 fields, the THEMIS trial has 27 fields, and the DELIVER trial has 28 fields. The overlap of these fields among the three trials is only 27%. Therefore, it is important that features from each trial are mapped to a common set of features so that decisions can be made on a common dataset.

[0096] Once this is done, the extracted and mapped features can be exposed in JSON format for use by machine learning models in performing clinical trial endpoint determination (as will be described in more detail later), with confidence scores being attributes of the features. In some cases, exposure may only be performed if the confidence score exceeds a selected threshold, for example, if a relatively high level of confidence is used, and thus only for the extracted and mapped features.

[0097] Performing feature analysis on data to extract features from the data may further include checking for and removing any duplicate, inconsistent, or irrelevant features. For example, irrelevant features may be those that are not related to the events determined by the model.

[0098] In some cases, if the confidence score is low (e.g., below a selected threshold), the document miner module 1303 may prompt the user to manually review the source of the data, as shown in Figure 14. This can be done, for example, by the document miner module 1303 determining that there are several areas with consecutively low confidence scores, and the document miner module 1303 may prompt the user to manually review those areas. This can be done, for example, by sending the user a notification with instructions or images of the areas that need to be manually reviewed.

[0099] It will be understood that the features required for determining clinical trial endpoints can vary and may be based on different endpoints related to the clinical trial (e.g., hospitalization, myocardial infarction, death, etc.). Therefore, in some examples, document minor module 1301 may be configured to retrieve a set of features required for determining clinical trial endpoints, the required set of features being based on the endpoint, and document minor module 1301 compares the features retrieved from multiple data sources with the set of features required for determining clinical trial endpoints to determine whether any feature is missing or incomplete. If any feature is determined to be missing or incomplete, document minor module 1303 may be configured to provide the user with a notification that the feature is missing, and the notification provides instructions on which feature is missing or incomplete.

[0100] In some examples, the document minor module 1303 may perform named entity recognition on the data before performing feature analysis on the data (see the “Automatic Event Determination” module 205 for more details, and select formal event characteristics associated with a predefined set of features).

[0101] The document miner module 1303 may be further configured to retrieve a set of features to be provided by a data source, determine whether any features are missing from that data source, and, if features are missing from that data source, provide the user with a notification that the features are missing. This is advantageous because it allows for the preparation of a complete and accurate judgment document and improves the performance of the judgment process.

[0102] Classification and determination The clinical endpoint event determination process exists due to variability in the judgment of clinical investigators regarding specific classes of clinical events, such as major adverse cardiovascular events (MACE). In global clinical trials, clinical investigators are not always trained in the relevant clinical field (e.g., cardiology), and differences in the interpretation of events such as cardiovascular death (CVD) and heart failure hospitalization (HF) can lead to quality issues in event reporting. The conventional solution to this is clinical event determination, in which a committee of trained clinicians, consisting of multiple clinicians, evaluates the clinical events that occurred in the trial until agreement is reached on the type of event. These events are clearly defined in the procedures established during the study design, and clear criteria for what constitutes an event are compiled. Clinical assessors determine the event type by reviewing a collection of structured data, such as assessment documents and clinical measurements, and unstructured data, such as medical source documents, such as discharge reports.

[0103] As a solution to this time-consuming and resource-intensive process, the inventors developed an automated event determination module 205. The automated event determination module 205 is an end-to-end process that uses a machine learning algorithm trained on the decisions of the clinical assessor as ground truth, which can evaluate the determination documents and determine clinical endpoint events such as CVD and HF. This method can efficiently provide quality assurance (QA) checks on the decisions of the on-site principal investigator regarding specific events.

[0104] The automated event determination module 205 retrieves data from a designated clinical trial based on the clinical event being determined, applies a machine learning algorithm or a set of algorithms, and outputs a report and recommendations regarding the classification of the clinical event.

[0105] Figure 15 provides a high-level overview of the steps involved in deploying the automated event determination module 205. In step 301, the system receives input data. The data received or used by the module can be broadly categorized as structured data (sourced from clinical databases) and unstructured data (clinical free text without structure or schema, encompassing a wide range of clinical documents specified per endpoint event). Data input can be obtained from the data harmonization and collection module 203, as described above and as will be discussed in more detail later with reference to Figure 18.

[0106] In step 303, relevant data is selected, and in step 305, features are extracted from that data. To do this, in the illustrated example, clinical free text is transformed using several known machine learning methods, including (but not limited to) bert, biobert, and fasttext, and named entity recognition (NER) with performance on a given set of features evaluated downstream of the classification task.

[0107] In step 307, the model is executed, and in step 309, the outcome of the likelihood of the event occurring is obtained.

[0108] Figure 16 shows a flowchart of an example of a computer-aided procedure 500 for performing clinical endpoint determination. The computer-aided procedure can be performed on a computer system 2600, as will be described later with reference to Figure 26, and can be stored, for example, in memory 2610 or storage 2616 and executed by a processor 2614.

[0109] In step 501, the method includes receiving data from multiple healthcare-related data sources. The data sources may include structured and unstructured data sources. Optionally, the method includes a step of parsing each data source to determine whether the data held by that data source includes structured and / or unstructured data. However, it will be understood that the data itself may have identifiers / metadata indicating the source, and therefore this parsing step may not be necessary.

[0110] In step 503, if the data includes structured data, the method includes extracting features from that data. In step 505, if the data includes unstructured data, the method includes applying a natural language processing model to the unstructured data to obtain embeddings related to features in the unstructured data. The natural language processing models may include, for example, bert, biobert, and / or fasttext. In some examples, a combination of natural language processing models may be used. The natural language processing models may be pre-trained on, for example, clinical datasets. For example, applying a natural language processing model may include applying multiple natural language processing models, including a first dedicated model trained on available text from a data source, along with a second general-purpose model trained on Wikipedia. In some examples, applying a natural language processing model may include applying BIObert (a model pre-trained on biomedical text) which converts input biomedical free text into a 768-dimensional vector, and a modified version of fasttext which combines a dedicated model trained on available free text with a general-purpose model trained on Wikipedia, both of which convert 300-dimensional vectors.

[0111] If the data includes unstructured data, in some examples the method further includes applying a named entity recognition model to the unstructured data to obtain formal event characteristics from the unstructured data, and then applying a machine learning classification model to the formal event characteristics obtained via the named entity recognition model.

[0112] In step 507, the method includes applying a machine learning classification model to embeddings from unstructured data and features extracted from structured data to classify whether a healthcare event has occurred based on the embeddings and features extracted from structured data. The machine learning classification model may include, for example, a random forest, a linear SVM, or XGBoost.

[0113] Optionally, the method includes step 509, which assigns a probability score to the classification as an attribute, the probability score providing an indication of the likelihood of an event occurring. For example, the method may further include providing the user with a notification to review the classification if the probability score is below a selected threshold. In this way, the review committee only needs to review a subset or selected set of events, and therefore the speed at which the review can be completed is greatly improved.

[0114] For example, a confidence score can be assigned to data as an attribute based on at least one of (i) the data source and (ii) the reliability determined based on an optical character recognition process applied to unstructured data, and the confidence score can be used as a weight by a machine learning classification model. The method may include excluding data with confidence scores below a selected threshold.

[0115] Extracting features from data and applying natural language processing models to unstructured data to obtain embeddings associated with those features may include obtaining a predefined set of features for use in determining clinical endpoints, mapping the extracted features and / or embeddings to the predefined set of features, and discarding features and / or embeddings that are not associated with the predefined set of features.

[0116] As will be described in more detail later with reference to Figures 20A, 20B, and 21, a machine learning classification model may provide a ranking of the importance of features involved in classifying whether or not a healthcare event has occurred. For example, providing a ranking of feature importance may include identifying the SHAP value for each feature and / or applying a local surrogate model to the machine learning classification model to identify the relative contribution of each feature to the classification.

[0117] In some examples, method 500 is executed only if the amount of available data exceeds a selected threshold. In some examples, method 500 is executed in response to an indication provided by the user that an event has occurred (for example, by the event sniffer 201 as described above).

[0118] As shown in Figures 2 and 5, it will be understood that human assessment is required for determining certain cases, such as cases assigned a low probability score. Therefore, it is the responsibility of healthcare providers or managers to ensure that these cases are monitored and determined correctly and in a timely manner. Accordingly, a method for monitoring clinical trial endpoint determinations is disclosed herein. The method comprises receiving multiple notifications of determination decisions from a clinical trial endpoint determination system, the determination decisions comprising probability scores that provide indications of the likelihood of an event occurring. The method then comprises ranking the notifications based on (i) the probability score and (ii) the severity of the event, and obtaining a document of the data used to perform the determination (i.e., a virtual determination document 1313). The method then comprises providing the user with a list of determination decisions and the corresponding virtual determination documents 1313 of the data to review the accuracy of the determination decisions (as shown in Figure 11 and described above), the order of the list being based on rank.

[0119] The method may further include obtaining a ranking of the importance of features related to classifying whether or not a healthcare event has occurred, and providing the user with the ranking of the importance of the features, along with a list of determinations and corresponding documents for the data.

[0120] Figure 17 shows at a high level the components (e.g., modules that can be implemented in software and / or hardware) and the data flow between them that can perform the method described in Figure 16 and train a machine learning model to perform the method in Figure 16, along with a brief description of the function of each component.

[0121] More specifically, in 601, data (such as PDFs) is input. In 603, structured and unstructured data (free text documents) are extracted and, in this example, output as pickle files, as the system can operate using the Python language. In 605, preliminary extraction is performed. 607 includes feature engineering, where cleaned structured data is extracted from preliminary data according to specifications. In some examples, this is combined with NLP data (from embeddings in 609, described later) and NER (also described later in 611). In some examples, 697 also includes performing test / training splits on the data for use in training machine learning models.

[0122] In step 609, embeddings are obtained. This involves applying an NLP model to free text to generate document embeddings. This may include applying a pre-trained Bert model, a fasttext model (one for each event type), and / or a Wikipedia®-trained fasttext model. This may output one feature pickle file per document, using one entry per event. The model may be refistered to ML flow 615b, and the features may be registered in the feature database 615a. Both ML flow 615b and the feature database 615a may be part of a remote data store (RDS) 615.

[0123] In 611, a pre-trained Bern model can be applied to a free text document from a patient-specific pickle file. A pickle file with entries for each event can be output. The model can be registered in ML flow 615b, and the features can be registered in feature database 615a.

[0124] Each of 607, 609, and 611 may provide features to be stored in feature store 613. Feature store 613 is used in 617 to perform event classification. This may perform cross-validation parameter search and evaluation using holdout tests set up in the model. The results and model may be output to a pickle file, and the results and model may be registered in ML flow 615b.

[0125] In step 621, visualization is performed to visualize the classifier's performance (which will be described in more detail later with reference to Figures 20A, 20B, and 21). Model statistics and results can be pulled from ML flow 615b and model artifacts.

[0126] It will be understood that the models used for decision-making may need to be trained. Accordingly, with reference to Figure 17 above, a method for training a machine learning classification model to perform clinical trial endpoint decisions is disclosed herein. It will be understood that the training method can be performed by implementing the method and system shown in Figure 17. The method comprises receiving data from multiple healthcare-related data sources, the data comprising decision documents from previous clinical trials and decision decisions associated with those decision documents. Each data source is parsed to determine whether the data held by the data source includes structured data and / or unstructured data. If the data includes unstructured data, the method comprises applying a natural language processing model to the unstructured data to obtain embeddings related to features in the unstructured data. If the data includes structured data, the method comprises extracting features from that data. The method then comprises providing instructions for decision-making based on the data from the decision documents and updating a machine learning classification model based on the decision decisions and the data from the decision documents. The updated machine learning classification model is then stored in a relational database.

[0127] After the models have been trained and their performance has been retrospectively validated under the conditions of the registered and completed trials, the best-performing feature sets and machine learning models can be selected for deployment in standalone form or as an ensemble.

[0128] As described above, the "Automated Event Detection" module 205 can be used in combination with the "Data Harmonization and Collection" module 203 and optionally with the "Event Sniffer" module 201. Figure 18 shows an example of how the "Automated Event Detection" module 205 can be used in combination with the "Data Harmonization and Collection" module 203. Figure 18 shows conceptual steps that can be applied to a process for automatically detecting adverse events in a clinical trial.

[0129] As shown in Figure 18, in 401, the data is input to the data harmonization and collection module 203. The data is split into unstructured source documents 403 or EDC (structured) data 405. In the case of unstructured source documents 403, extraction 407 is performed to obtain extracted data (e.g., by obtaining embedding and / or NER using NLP as described above). In the case of structured data, the data is extracted (409) to obtain extracted structured data 413. In 415, the structured and unstructured data are harmonized (e.g., to ensure consistency of fields related to the clinical endpoints being considered) to obtain harmonized data 417. At this stage, quality control checks 419 are performed using the QC bot 1305, for example, as described above with reference to Figure 111. The data is then passed to the "Automated Event Determination" module 205, which may perform model-specific QC checks 421 (e.g., using the QC bot 1311). In step 423, features are constructed to obtain a set of features 425, which are then used for classification in step 427. The result of classification 427 is the decision output 429. In some cases (for example, when the probability of an event occurring is relatively low), manual decision 431 may be performed.

[0130] It will be understood that different machine learning algorithms may be trained and deployed depending on the event designed to be identified or determined (e.g., CV death). For example, Figures 19A–9C show the results of three algorithms (CV death, non-CV death, and indeterminate) trained on 817 patients from the DECLARE clinical trial. Regardless of whether a linear SVC, RBF SVC, random forest, or XGBoost model was applied to the data, the ROC curves are all relatively similar and close to perfect classifiers. Figure 19A shows that the AUC on the trial set (i.e., holdout set) is above 80% for each model. Figure 19C shows the distribution of model prediction scores (X-axis) by class (color) and shows that the models generally produce higher scores (closer to 1) for positive cases and lower scores (closer to 0) for negative cases. This separation tends to indicate a valid classifier that can distinguish between binary classes.

[0131] Machine learning models are extremely complex across the global feature space, but much simpler locally. A particular set of tests can be perturbed to explain and learn a local linear approximation of the model's behavior as an explanation. This can reveal how much perturbation to the input affects the model's predictions and how each feature contributes to the model's predictions. This can be called a locally interpretable model-independent explanation, or LIME.

[0132] Figures 20A and 20B illustrate how a model or algorithm can be analyzed to provide interpretability. For example, they show how a model can be analyzed to provide indications of top features that have mutual information, even with structured and unstructured data. In Figure 20A, a feature importance analysis of structured features in this model shows that "CT scan unknown" and "neurological impairment unknown on examination" have the highest importance, meaning they are more useful than other structured features in predicting the target variable.

[0133] Figure 20B demonstrates model interpretability by showing how an NLP algorithm (in this case, BioBERT) interprets and groups similar terms together based on the context in the text by converting text into numerical vectors. For example, in the lower right of the graph, "coronary heart disease" and "myocardial infarction," both terms from the general domain of cardiovascular diseases are identified as similar by the BioBERT model and are therefore closer to each other compared to other uses of the chart. A visual inspection of this chart confirms that the BioBERT model is functioning as expected.

[0134] Shapley's additive explanation, or SHAP, is a game theory method for explaining the output of any machine learning model. For understanding, it links optimal credit allocation to local explanations. • The importance of each feature to the overall model prediction. • Effects that can be reasonably expected to be present in a feature value compared to the average. The SHAP value indicates the influence of each feature on the model output. This can be represented, for example, by the chart shown in Figure 21.

[0135] Figure 22 shows the performance of a machine learning model across several metrics (e.g., AUC - Area Under Curve, accuracy, balanced accuracy, F1, etc.) and model versions (first three columns). For example, the first row shows metrics (model performance) for a death event random forest algorithm trained on structured EDC data, where the indetermination class is mapped to non-CV deaths.

[0136] Example 1 Accurate identification of clinical outcome events is crucial for obtaining reliable results in cardiovascular outcome trials (CVOTs). Current event determination processes are expensive and cumbersome. As part of a larger project to identify outcomes with greater reliability, we evaluated the use of machine learning to automate event determination using data from the SOCRATES trial (NCT01994720), a large randomized trial comparing ticagrelor and aspirin in reducing the risk of major cardiovascular events following acute ischemic stroke or transient ischemic attack (TIA).

[0137] We studied machine learning algorithms to determine whether they could replicate the outcomes of the expert assessment process for clinical events such as ischemic stroke and TIA. We trained the classification model on historical CVOT data to see if it could perform comparably to human assessment.

[0138] Using data from the SOCRATES trial, we tested multiple machine learning algorithms using grid search and cross-validation. The models tested included support vector machines, random forests, and XGBoost. Performance was assessed on a validation subset of decision data not used for training or testing during model development. Metrics used to evaluate model performance were receiver operating characteristics (ROC), Matthews correlation coefficient, precision, and recall. Both mutual information and recursive feature reduction were used to examine the contribution of data features and attributes used by the algorithms that contributed to classification when trained to classify events.

[0139] The classification models were trained on historical CVOT data using judge consensus decisions as ground truth. Best performance was observed in models trained to classify ischemic stroke (ROC 0.95) and TIA (ROC 0.97). Top-rank features contributing to the classification of ischemic stroke or TIA correspond to variables used to define events in the trial procedure, such as the investigator's decision at the scene or the duration of symptoms. Model performance was comparable across different machine learning algorithms tested, with XGBoost showing the best ROC in the validation set regarding correctly classifying both stroke and TIA.

[0140] Therefore, the results indicate that machine learning can improve or replace clinician judgment in clinical trials, potentially gaining efficiency, accelerating clinical development, and maintaining reliability. The model according to the present invention demonstrates good performance in the binary classification of ischemic stroke and TIA within a single CVOT, with high consistency and accuracy between automated and clinician judgments.

[0141] As further evidence of the model's efficacy, Figure 23 shows the performance of the machine learning model when assessing cardiovascular death as an outcome across several metrics (e.g., AUC - area under the curve, accuracy, balanced accuracy, F1, etc.) performed on data from three clinical trials (DECLARE, THEMIS, and DAPA-HF).

[0142] As described above, the three modules shown in Figure 2 (event sniffer module 201, data harmonization and collection module 203, and automated event determination module 205) may utilize machine learning models or ensembles of models. These models may include known models such as Bert, Biobert, and FastText, and / or adapted versions of these known models.

[0143] Machine learning models may include neural networks. Neural networks may include at least one of the following: deep residual networks, highway networks, high-density connectivity networks, and capsule networks.

[0144] In any such network, the network may contain multiple different neurons organized into different layers. Each neuron is configured to receive input data, process this input data, and provide output data. Each neuron may be configured to perform a specific action on its input, which may include, for example, mathematically processing the input data. The input data of each neuron may contain outputs from multiple other preceding neurons. As part of the neuron's action on the input data, each stream of input data (e.g., one input data stream for each preceding neuron that provides an output to that neuron) is assigned a weight. Thus, the processing of the input data by a neuron involves applying weights to different input data streams so that different items of the input data contribute more or less to the neuron's overall output. For example, adjusting the values ​​of a neuron's input as a result of changing the input weights may result in a change in the value of that neuron's output. The output data from each neuron may be sent to multiple subsequent neurons.

[0145] Neurons are organized into layers. Each layer contains multiple neurons that operate on data provided by the outputs of neurons in the previous layer. Within each layer, there can be many different neurons, each applying different weights to the input data and performing different actions on the input data. The input data for all neurons in a layer may be the same, and the output from a neuron is passed to a neuron in a subsequent layer.

[0146] Strict routing between neurons within different layers is a key difference between capsule networks and deep residual networks (including variants such as highway networks and high-density connectivity networks).

[0147] In residual networks, layers can be organized into blocks such that the network contains multiple blocks, each block containing at least one layer. In residual networks, output data from a layer of a neuron can follow two or more different paths. In conventional neural networks (e.g., convolutional neural networks), output data from one layer is passed to the next layer, and this continues until the end of the network; therefore, each layer receives input from the previous layer and provides output to the following layer. However, in residual networks, different routing can occur between layers. For example, the output from one layer can be passed to multiple different subsequent layers, and the input to one layer can be received from multiple different preceding layers.

[0148] In a residual network, layers of neurons can be organized into different blocks, each block containing at least one layer of neurons. A block can consist of layers stacked together such that the output of the previous layer (or layers) feeds into the input of the next layer block. The structure of a residual network can be such that the output from one block (or layer) is passed to both the immediately following block (or layer) and at least one other subsequent block (or layer). Shortcuts can be introduced into the neural network to pass data from one layer (or block) to another, bypassing other layers (or blocks) in between. This can allow for more efficient training of the network, for example, when dealing with very deep networks, because it allows for addressing degradation-related problems when training the network (more details below). The structure of a residual neural network can allow for branching so that the same input provided to one layer or layer block is provided to at least one other layer or layer block (for example, so that the other layer can operate on both the input and output data from that one layer or layer block). This configuration allows for deeper intrusion into the network when training it using a backpropagation algorithm. For example, during training, it may be possible to take a layer or block of layers as input, the input of the previous layer / block, and the output of the previous layer / block, and when updating the network's weights, shortcuts can be used to provide deeper intrusion.

[0149] In a capsule network, layers can be nested within other layers, providing "capsules." Different capsules can be configured to be more adept at performing different tasks than others. A capsule network can provide dynamic routing between capsules so that, for a given task, that task is assigned to the capsule most capable of handling that task. For example, a capsule network can avoid routing outputs from any neuron in a layer to any neuron in the next layer. Lower-level capsules are configured to send their inputs to higher-level (successor) capsules that are judged to be the capsule most likely to handle their inputs. Capsules can predict the activity of capsules in higher layers. For example, a capsule may output a vector, the direction of which represents the nature of the object in question. In response, each successor capsule may, as an output, provide the probability that the object the capsule was trained to identify is present in the input data. Information (e.g., probabilities) can be fed back to the capsules, which can then dynamically determine routing weights and forward the input data to the successor capsule that is most likely to be the appropriate capsule for processing that data.

[0150] Any type of neural network may contain multiple different layers with different functions. A neural network may contain at least one convolutional layer configured to convolve input data over its height and width. A neural network may also have multiple filtering layers, each containing multiple neurons configured to focus on different parts of the input data and apply filters. Other layers may be included to process the input data, such as pooling layers (to introduce nonlinearity), including max pooling and global average pooling, normalized linear unit layers (ReLU), and loss layers, some of which may include regularization functions. The final block of layers may receive input from the last output layer (or more layers, if branches exist). The final block may contain at least one fully connected layer.

[0151] The final output layer may include a classifier such as a softmax, sigmoid, or tanh classifier. Different classifiers may be suitable for different types of outputs; for example, a sigmoid classifier may be suitable if the output is a binary classifier. The output of the neural network may provide a probability indication.

[0152] The input can be supplied to a set of 3D layers within the neural network. Several features of this network can change as the network is trained. Each neuron can have multiple weights, each weight applied to each input stream of output data from neurons in the previous layer. These weights are variables and can be changed to provide changes to the neural network's output. These weights can be changed in response to training to provide more accurate data. The changed weights in response to being trained are said to be "learned". Furthermore, the size and connectivity of the layers can depend on the network's typical input data, but these can also be variables and can be changed and learned during training, including augmenting connectivity.

[0153] For example, to train a network to learn the values ​​of its weights, these weights are assigned initial values. These initial values ​​can be essentially random, but to improve network training, a value-appropriate initialization such as Xavier / Glorot initialization can be applied. Such initialization can prevent situations where the initial random weights are too large or too small, and the neural network can never be adequately trained to eliminate these initial biases. This type of initialization may involve assigning weights using a distribution with a zero mean but a constant variance.

[0154] Once weights are assigned, training object data can be supplied to or input to the neural network. This may involve operating the neural network with the results of previous clinical trial decisions, as previously referenced when illustrating Figure 6. During this process, algorithms such as minibatch gradient descent, RMSprop, Adam, Adadelta, and Nesterov can be used. This allows for the identification of the amount by which each different point (neuron) or path (between neurons in subsequent layers) in the network contributes to the inaccurate score, and thus allows for the determination of any weight adjustments that need to be made. The weights can then be adjusted according to the calculated error, for example, to minimize or remove the contribution from the neuron that contributes most to the inaccurate identification.

[0155] After training iterations of the network, the weights can be updated (750), and this process can be repeated many times. To suppress the likelihood of the network overfitting, training variables such as learning speed and momentum can be modified and / or controlled to selected values. Furthermore, regularization techniques such as L2 or dropout can be used, which reduce the likelihood that other different layers will overfit and become too specific to the training data without being generally applicable to other similar data. Similarly, batch normalization can be used to aid training and improve accuracy. Generally, the weights are adjusted so that the network produces the expected results when run again on the same training data. However, the extent to which this is true will depend on training variables such as learning speed.

[0156] It should be understood that increasing the depth of a neural network may lead to problems during training, such as the vanishing gradient problem, and may also result in slower networks. However, this disclosure may enable the provision of networks with increased depth and accuracy without sacrificing the ability to train the network appropriately.

[0157] The network depth used can be chosen to provide a balance between accuracy and the time it takes to deliver an output. Increasing the network depth can provide higher accuracy, but it can also increase the time it takes to deliver an output. The use of branched structures (as opposed to convolutional neural networks) can increase the network depth and provide increased accuracy, thus allowing for sufficient training of the network.

[0158] Figure 24 is a block diagram of a computer system 2600 suitable for implementing one or more embodiments of the present disclosure. In various embodiments, the computer system 2600 may include user devices such as mobile cellular phones, personal computers (PCs), laptops, wearable computing devices, and tablets configured for wireless communication. However, it will also be understood that in some examples, embodiments of the present disclosure may be implemented on a remote server in the cloud, and similar functionality as shown in Figure 24 may be provided by the remote server.

[0159] The computer system 2600 includes a bus 2612 or other communication mechanism for communicating information data, signals, and information between various components of the computer system 2600. The components include an input / output (I / O) component 2604 that processes user (i.e., sender, receiver, service provider) actions such as selecting keys from a keypad / keyboard, selecting one or more buttons or links, and transmits corresponding signals to the bus 2612. The I / O component 2604 may also include output components such as a display 2602 and a cursor control mechanism 2608 (keyboard, keypad, mouse, etc.). The display 2602 may be configured to display a login page for logging into a user account or a checkout page for purchasing goods from a vendor. An optional audio input / output component 2606 is also included, which may allow the user to use voice to input information by converting audio signals. The audio I / O component 2606 may allow the user to hear audio. The transceiver or network interface 2620 transmits and receives signals between the computer system 2600 and other devices such as another user device, vendor server, or service provider server via the network 2622. In one embodiment, the transmission is wireless, but other transmission media and methods may also be suitable. The processor 2614 may be a microcontroller, a digital signal processor (DSP), or other processing component, which processes these various signals for, for example, display on the computer system 2600 or transmit to other devices via the communication link 2624. The processor 2614 may also control the transmission of information such as cookies or IP addresses to other devices.

[0160] The components of computer system 2600 also include a system memory component 2610 (e.g., RAM), a static storage component 2616 (e.g., ROM), and / or a disk drive 2618 (e.g., a solid-state drive, hard drive). Computer system 600 performs specific operations by the processor 614 and other components by executing one or more instruction sequences contained in the system memory component 2610.

[0161] Logic may be encoded in a computer-readable medium, which may refer to any medium that participates in providing instructions to the processor 2614 for execution. Such a medium can take many forms, including, but not limited to, non-volatile, volatile, and transmission media. In various embodiments, non-volatile media include optical or magnetic disks, volatile media include dynamic memory such as the system memory component 2610, and transmission media include coaxial cables, copper wires, and optical fibers, including wires that constitute the bus 2612. In one embodiment, logic is encoded in a non-transient computer-readable medium. In one example, the transmission medium may take the form of acoustic or optical waves, such as those generated during radio, optical, and infrared data communications.

[0162] Some common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROMs, EPROMs, flash EPROMs, any other memory chips or cartridges, or any other media configured to be read by a computer.

[0163] In various embodiments of the Disclosure, the execution of instruction sequences to implement the Disclosure may be performed by a computer system 2600. In various other embodiments of the Disclosure, multiple computer systems 2600 connected by a communication link 2624 to a network (e.g., LAN, WLAN, PTSN, and / or various other wired or wireless networks including telecommunications, mobile, and cellular telephone networks) may execute instruction sequences and cooperate with each other to implement the Disclosure.

[0164] Where applicable, the various embodiments provided herein may be implemented using hardware, software, or a combination of hardware and software. Where applicable, without departing from the spirit of this disclosure, the various hardware and / or software components described herein may be combined to form composite components comprising software, hardware, and / or both. Where applicable, without departing from the spirit of this disclosure, the various hardware and / or software components described herein may be separated into subcomponents comprising software, hardware, and / or both. In addition, where applicable, software components may be implemented as hardware components, and vice versa.

[0165] The software disclosed herein, including program code and / or data, may be stored on one or more computer-readable media. It is also intended that the software identified herein may be implemented using one or more general-purpose or dedicated computers and / or computer systems, networked, and / or otherwise. Where applicable, the order of the various steps described herein may be modified, combined into composite steps, and / or separated into substeps in order to provide the features described herein.

[0166] The various features and steps described herein may be implemented as a system including one or more memories storing the various information described herein, and one or more processors coupled to one or more memories and a network, wherein one or more processors are operable to perform the steps described herein as a non-temporary machine-readable medium containing a plurality of machine-readable instructions, and the machine-readable instructions, when executed by one or more processors, are configured to cause one or more processors to perform the methods including the steps described herein, as well as methods performed by one or more devices such as hardware processors, user devices, servers, and other devices described herein.

[0167] Other examples and variations of the apparatus and methods described herein will be apparent to those skilled in the art in relation to this disclosure.

Claims

1. A computer-based method for performing clinical trial endpoint determination, In a computing system, receiving data from multiple healthcare-related data sources, If the aforementioned data includes unstructured data, the natural language processing model is applied to the unstructured data to obtain embeddings related to features in the unstructured data. If the aforementioned data includes structured data, the process involves extracting features from that data. Applying a machine learning classification model to the embeddings from the unstructured data and the features extracted from the structured data, the classification of whether or not a healthcare event has occurred based on the embeddings and the features extracted from the structured data. Assigning a probability score as an attribute to the classification, wherein the probability score indicates the likelihood that the event occurred, If the aforementioned probability score is below the selected threshold, the user will be notified to review the classification, A computer implementation method including

2. The computer implementation of claim 1, wherein applying a natural language processing model includes applying a plurality of natural language processing models, each including a first dedicated model trained on text available from the data source, together with a second general-purpose model.

3. If the aforementioned data includes unstructured data, a named entity recognition model is applied to the unstructured data to obtain formal event characteristics from the unstructured data. Applying the machine learning classification model to the formal event characteristics obtained via the named entity recognition model, A computer implementation method according to claim 1 or 2, further comprising:

4. (i) assigning a confidence score as an attribute to the data based on the data source and (ii) at least one of the confidence levels identified based on the optical character recognition process applied to the unstructured data, The machine learning classification model uses the confidence score as a weight, The computer implementation method according to any one of claims 1 to 3.

5. The computer implementation method according to claim 4, further comprising excluding data having a confidence score below a selected threshold.

6. A computer-aided method according to any one of claims 1 to 5, wherein extracting features from the data and applying a natural language processing model to the unstructured data to obtain embeddings related to features in the unstructured data includes obtaining a predefined set of features for use in determining the clinical trial endpoint, mapping the extracted features and / or embeddings to the predefined set of features, and discarding features and / or embeddings that are not related to the predefined set of features.

7. The computer implementation method according to any one of claims 1 to 6, wherein the machine learning classification model provides a ranking of the importance of the features related to the classification of whether or not a healthcare event has occurred.

8. The computer implementation method according to claim 7, wherein providing a ranking of the importance of the features includes identifying the SHAP value of each feature.

9. The computer implementation method according to claim 7 or 8, wherein providing the ranking of the importance of the features comprises applying a local surrogate model to the machine learning classification model to identify the relative contribution of each feature to the classification.

10. The computer implementation method according to any one of claims 1 to 9, which is performed when the amount of available data exceeds a selected threshold.

11. The computer implementation method according to any one of claims 1 to 10, wherein the method is performed in response to an event occurrence instruction provided by a user.

12. A method for monitoring the determination of clinical trial endpoints, In a calculation system, receiving multiple notifications of determination decisions from a clinical trial endpoint determination system, wherein the determination decisions include a probability score indicating the likelihood that an event occurred, (i) ranking the notification based on the probability score and (ii) the severity of the event, To obtain the data documents used to perform the aforementioned determination, The process involves providing the user with a list of judgments and corresponding documents for the data, and reviewing the accuracy of the said judgments, wherein the order of the list is based on the said ranking. A method that includes this.

13. To obtain a ranking of the importance of features related to the classification of whether or not a healthcare event occurred, To provide the user with the ranking of the importance of the aforementioned features, along with the list of the determination and the corresponding documents for the data, The method according to claim 12, further comprising:

14. A computer-readable non-temporary storage medium containing a computer program configured to cause a processor to perform the method described in claim 1.

15. A computer-readable non-temporary storage medium containing a computer program configured to cause a processor to perform the method described in claim 12.

Images