Clinical endpoint determination system and method

Abstract

This invention provides a system and method for determining clinical endpoints. [Solution] A computer-based method for performing clinical trial endpoint determination includes: receiving data from multiple healthcare-related data sources; applying a natural language processing model to the unstructured data if the data includes unstructured data to obtain embeddings related to features in the unstructured data; extracting features from the structured data if the data includes structured data; applying a machine learning classification model to the embeddings from the unstructured data and the features extracted from the structured data to classify whether a healthcare event has occurred based on the embeddings and the features extracted from the structured data; assigning a probability score as an attribute to the classification that indicates the likelihood of the event occurring; and providing the user with a notification to review the classification if the probability score is below a selected threshold.

Claims

1. A computer-based method for performing clinical trial endpoint determination, In a computing system, receiving data from multiple healthcare-related data sources, If the aforementioned data includes unstructured data, the natural language processing model is applied to the unstructured data to obtain embeddings related to features in the unstructured data. If the aforementioned data includes structured data, the process involves extracting features from that data. Applying a machine learning classification model to the embeddings from the unstructured data and the features extracted from the structured data, the classification of whether or not a healthcare event has occurred based on the embeddings and the features extracted from the structured data. Assigning a probability score as an attribute to the classification, wherein the probability score provides an indication of the likelihood of the event occurring, and assigning it, If the aforementioned probability score is below the selected threshold, the user will be notified to review the classification, Computer implementation methods including

2. The computer implementation of claim 1, wherein applying a natural language processing model includes applying a plurality of natural language processing models, each including a first dedicated model trained on text available from the data source, together with a second general-purpose model.

3. If the aforementioned data includes unstructured data, a named entity recognition model is applied to the unstructured data to obtain formal event characteristics from the unstructured data. Applying the machine learning classification model to the formal event characteristics obtained via the named entity recognition model, A computer implementation method according to claim 1 or 2, further comprising:

4. (i) assigning a confidence score as an attribute to the data based on the data source and (ii) at least one of the confidence levels identified based on the optical character recognition process applied to the unstructured data, The machine learning classification model uses the confidence score as a weight, The computer implementation method according to any one of claims 1 to 3.

5. The computer implementation method according to claim 4, further comprising excluding data having a confidence score below a selected threshold.

6. A computer-aided method according to any one of claims 1 to 5, wherein extracting features from the data and applying a natural language processing model to the unstructured data to obtain embeddings related to features in the unstructured data includes obtaining a predefined set of features for use in determining the clinical endpoint, mapping the extracted features and / or embeddings to the predefined set of features, and discarding features and / or embeddings that are not related to the predefined set of features.

7. The computer implementation method according to any one of claims 1 to 6, wherein the machine learning classification model provides a ranking of the importance of the features related to the classification of whether or not a healthcare event has occurred.

8. The computer implementation method according to claim 7, wherein providing a ranking of the importance of the features includes identifying the SHAP value of each feature.

9. The computer implementation method according to claim 7 or 8, wherein providing the ranking of the importance of the features comprises applying a local surrogate model to the machine learning classification model to identify the relative contribution of each feature to the classification.

10. The computer implementation method according to any one of claims 1 to 9, which is performed when the amount of available data exceeds a selected threshold.

11. The computer implementation method according to any one of claims 1 to 10, wherein the method is performed in response to an event occurrence instruction provided by a user.

12. A method for training a machine learning classification model to perform clinical trial endpoint determination, In a computing system, receiving data from multiple healthcare-related data sources, wherein the data includes judgment documents from previous clinical trials and judgment decisions related to those judgment documents. Analyze each data source to determine whether the data held by the data source includes structured data and / or unstructured data, If the aforementioned data includes unstructured data, the natural language processing model is applied to the unstructured data to obtain embeddings related to features in the unstructured data. If the aforementioned data includes structured data, the process involves extracting features from that data. To provide instructions for the determination based on the data from the aforementioned determination documents, The machine learning classification model is updated based on the aforementioned determination and the data from the aforementioned determination documents. The updated machine learning classification model is stored in a relational database, A method that includes this.

13. A method for monitoring the determination of clinical trial endpoints, In a calculation system, receiving multiple notifications of determination decisions from a clinical trial endpoint determination system, wherein the determination decisions include probability scores that indicate the likelihood of an event occurring, (i) ranking the notification based on the probability score and (ii) the severity of the event, To obtain the data documents used to perform the aforementioned determination, The process involves providing the user with a list of judgments and corresponding documents for the data, and reviewing the accuracy of the said judgments, wherein the order of the list is based on the said ranking. A method that includes this.

14. To obtain a ranking of the importance of features related to the classification of whether or not a healthcare event occurred, To provide the user with the ranking of the importance of the aforementioned features, along with the list of the determination and the corresponding documents for the data, The method according to claim 13, further comprising:

15. A computer-aided method for harmonizing and collating data from multiple healthcare-related sources for the determination of clinical trial endpoints, In the computing system, each data source is analyzed to determine whether the data held by the data source includes structured data and / or unstructured data. If the aforementioned data includes unstructured data, optical character recognition is performed on one or more regions of the aforementioned data that are not yet in a machine-readable format. (i) assigning a confidence score as an attribute to the data based on the data source and (ii) at least one of the confidence levels identified based on the optical character recognition process, Performing feature analysis on the data to extract features from the data, The extracted features are mapped to a predefined set of features, To perform the aforementioned clinical trial endpoint determination, the extracted and mapped features are made public in JSON format for use by a machine learning model, wherein the confidence score is an attribute of the features. Computer implementation methods including

16. The method according to claim 15, further comprising publishing the extracted and mapped features in JSON format if the confidence score exceeds a selected confidence threshold.

17. The acquisition of a set of features necessary for determining the clinical trial endpoint, wherein the acquisition of the necessary set of features is based on the endpoint. The features obtained from the aforementioned multiple data sources are compared with the set of features necessary for determining the clinical trial endpoint to determine whether any feature is missing or incomplete. If any feature is determined to be missing or incomplete, the user shall be notified of the missing feature, and the notification shall include instructions for the missing or incomplete feature. The method according to claim 15 or claim 16, further comprising:

18. The method according to any one of claims 1 to 17, further comprising performing named entity recognition on the data and selecting formal event characteristics associated with a predefined set of features before performing feature analysis on the data.

19. The method according to any one of claims 1 to 18, further comprising obtaining a set of features to be provided by a data source, determining whether any feature is missing from the data source, and, if a feature is missing from the data source, providing the user with a notification that the feature is missing.

20. The method according to any one of claims 1 to 19, further comprising performing feature analysis on the data and extracting features from the data, checking and removing any duplicate, inconsistent, or inappropriate features.

21. A monitoring system for determining whether or not a healthcare event occurred in a participant in a clinical trial, A communication interface configured to receive data signals related to multiple participants from multiple sources, each of which data signals includes information indicating parameters related to the participant, Processor and Equipped with, For each participant, the processor is configured to process each received data signal and apply a first weight to each data signal based on the source of the data signal. The system is configured such that the processor determines the probability of a healthcare event occurring based on (i) the data signal indicating that a patient-related parameter exceeds a threshold selected for that patient, and (ii) the first weight exceeding a selected trigger threshold.

22. The system according to claim 11, wherein the processor is configured to determine that a healthcare event has occurred based on the identified probability exceeding a selected threshold, and if the processor determines that a healthcare event has occurred, the processor is configured to provide a notification to a user of the monitoring system, and the monitoring system is configured to rank the notifications based on the identified probability of the healthcare event occurring.

23. The system according to claim 21 or 22, wherein the processor is configured to determine the type of healthcare event based on the instructions for the parameters associated with the source and participant of the data signal.

24. The monitoring system is configured to rank the notifications based on the determined type of healthcare event, according to claim 23.

25. The monitoring system according to claim 22, 23, or 24, wherein the monitoring system is configured to rank the notifications based on the participant's known health status.

26. The system according to any one of claims 21 to 25, wherein the processor is configured to determine the probability of a healthcare event occurring based on a plurality of data signals, each of which at least one data signal indicates that (i) a patient-related parameter exceeds a selected threshold and (ii) the weight of the data signal exceeds a selected threshold.

27. The system according to any one of claims 21 to 26, wherein when the processor identifies the probability of a healthcare event occurring based on an received data signal indicating that the parameter exceeds a selected threshold, the processor examines information indicating previous values ​​of the parameter associated with the participant in a selected time interval preceding the identification.

28. The system according to any one of claims 21 to 27, wherein if it is determined that the probability of an event occurring exceeds a selected threshold, the processor is configured to determine whether more information is needed from the patient, and if more information is needed, a notification is provided to the healthcare provider / system administrator to contact the patient.

29. The system according to any one of claims 21 to 28, wherein the processor is also configured to determine the reliability of the data signal and to apply a second weight based on the reliability of the data signal, and the processor is configured to determine the probability of a healthcare event occurring based on the data signal indicating that a patient-related parameter exceeds a selected threshold, as well as the first and second weights.

30. The system according to any one of claims 21 to 29, wherein the processor is configured to determine the probability of a participant experiencing a healthcare event, based on the probability of any event occurring before the participant has been identified.

31. A method for determining whether or not a healthcare event occurred in a participant in a clinical trial, In a computing system, receiving data signals related to multiple participants from multiple sources, each of which includes information indicating parameters related to the participant, For each participant, the received data signal is processed, and a first weight is applied to each data signal based on the source of the data signal. Identifying the probability of a healthcare event occurring based on (i) the data signal indicating that a patient-related parameter exceeds a threshold selected for that participant, and (ii) at least one of the first weights exceeding a selected trigger threshold, A method that includes this.

32. The method according to claim 31, comprising determining that a healthcare event has occurred based on the identified probability exceeding a selected threshold, and providing a notification to the user if a healthcare event has been determined to have occurred, wherein the notification is ranked based on the identified probability of the healthcare event occurring.

33. The method according to claim 31, comprising determining the type of healthcare event based on the source of the data signal and the instructions for the parameters associated with the participant.

34. The method according to claim 33, comprising ranking the notification based on the determined type of the healthcare event.

35. The method according to claim 32, comprising ranking the notifications based on the known health of the participant.

36. The method according to claim 31, comprising identifying the probability of a healthcare event occurring based on a plurality of data signals, each of which at least one data signal indicates that (i) a patient-related parameter exceeds a selected threshold and (ii) the weight of that data signal exceeds a selected threshold.

37. The method according to claim 31, comprising determining the probability of a healthcare event occurring based on an received data signal indicating that a parameter exceeds a selected threshold, wherein the determination is based on information indicating previous values ​​of the parameter associated with the participant in a selected time interval preceding the determination.

38. The method according to claim 31, wherein if it is determined that the probability of an event occurring exceeds a selected threshold, the processor is configured to determine whether more information is needed from the patient, and if more information is needed, a notification is provided to the healthcare provider / system administrator to contact the patient.

39. The method according to claim 31, comprising determining the reliability of the data signal and applying a second weight based on the reliability of the data signal, wherein the method comprises determining the probability of a healthcare event occurring based on the data signal indicating that a patient-related parameter exceeds a selected threshold and the first and second weights.

40. The method according to any one of claims 31 to 39, further comprising determining the probability of a participant experiencing a healthcare event based on the probability of any event occurring prior to the participant's occurrence.

41. A monitoring system for determining whether or not a healthcare event occurred in a participant in a clinical trial, A communication interface configured to receive data signals related to multiple participants from multiple sources, each of which data signals includes information indicating a location associated with a participant, Processor and Equipped with, For each participant, the processor is configured to process each received data signal. The processor is configured to determine the probability of a participant experiencing a healthcare event based on (i) the participant's proximity to a known healthcare center and (ii) the duration of the participant's proximity to the known healthcare center. A monitoring system in which, if the processor determines that the probability of a healthcare event occurring exceeds a selected threshold, the processor is configured to send a notification to the participant requesting confirmation from the participant that a healthcare event has occurred.

42. A method for determining whether or not a healthcare event occurred in a participant in a clinical trial, In a computing system, receiving data signals related to multiple participants from multiple sources, each of which includes information indicating a location associated with a participant, For each participant, the received data signals are processed to identify the participant's probability of a healthcare event occurring based on (i) the participant's proximity to a known healthcare center and (ii) the duration of the participant's proximity to the known healthcare center. If it is determined that the probability of the aforementioned healthcare event occurring exceeds a selected threshold, a notification is sent to the participant requesting confirmation from the participant that the healthcare event has occurred. A method that includes this.

43. A computer-readable non-temporary storage medium containing a computer program configured to cause a processor to perform the method described in claim 1.

44. A computer-readable non-temporary storage medium containing a computer program configured to cause a processor to perform the method described in claim 12.

45. A computer-readable non-temporary storage medium containing a computer program configured to cause a processor to perform the method described in claim 13.

46. A computer-readable non-temporary storage medium containing a computer program configured to cause a processor to perform the method described in claim 15.

47. A computer-readable non-temporary storage medium containing a computer program configured to cause a processor to perform the method described in claim 31.

48. A computer-readable non-temporary storage medium containing a computer program configured to cause a processor to perform the method described in claim 42.

Images