Identifying lines of treatment from clinical documents

Abstract

The method (1300) includes receiving input data (102) including unstructured text representing one or more sequences of terms (104). For each respective sequence of terms, the method includes generating a corresponding Line of Therapy (LoT) pseudo-label (122) indicating whether the respective sequence of terms includes LoT information, generating a corresponding LoT indicator (132) predicting whether the respective sequence of terms includes LoT information, and determining a corresponding LoT-directed loss (242) based on the corresponding LoT pseudo-label and the corresponding LoT indicator. The method also includes fine-tuning a pre-trained Transformer model (300) based on the LoT-directed loss determined for the one or more sequences of terms.

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates to line of treatment identification from clinical documentation. [Background technology]

[0002] Clinical trial design (CTD) is an important part of drug development because it influences the length of the clinical trial, the protocol, patient enrollment, clinical endpoints, and required comparators. Knowledge from previous clinical trials of the same or different drugs can aid in the CTD process. Line of therapy (LoT) refers to the sequence of different treatments administered to a patient during their disease progression. LoT is an important concept in the context of clinical trials because it can relate to the patient population for which a regulatory agency approves a drug. For example, a regulatory agency may grant approval for a first drug only to patients who have already failed standard of care (SoC) therapy (e.g., the first line of therapy). In that case, approval of the first drug is referred to as second-line approval. Similarly, if a second drug is approved only for patients who have failed both SoC therapy and treatment with the first drug, approval of the second drug is referred to as third-line approval. Summary of the Invention

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for using a weakly supervised deep learning model for line of therapy identification from clinical documents. The operations include receiving input data including unstructured text representing a sequence of one or more terms. For each respective sequence of terms, the operations include generating a corresponding line of therapy (LoT) pseudo-label using regular expression rules to indicate whether the respective sequence of terms contains LoT information, generating a corresponding LoT indicator using a pre-trained Transformer model to predict whether the respective sequence of terms contains LoT information, and determining a corresponding LoT-indicated loss based on the corresponding LoT pseudo-label and the corresponding LoT indicator. The operations also include fine-tuning the pre-trained Transformer model based on the LoT-indicated loss determined for the sequence of one or more terms. Embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, the operations further include: for each respective sequence of terms, generating a corresponding classification pseudo-label indicating a classification of the respective sequence of terms using regular expression rules; generating a corresponding LoT classification using a pre-trained Transformer model to predict a classification of the respective sequence of terms; and determining a corresponding LoT classification loss based on the corresponding classification pseudo-label and the corresponding LoT classification; and fine-tuning the pre-trained Transformer model based on the LoT classification loss determined for the one or more sequences of terms. In these embodiments, the classification of each sequence of terms indicates a particular LoT step from a set of LoT steps associated with the respective sequence of terms. The input data may include a plurality of clinical trial documents including unstructured text.

[0004] In some examples, the operations further include updating the regular expression rules based on the corresponding LoT classification loss. The pre-trained Transformer model includes a Bidirectional Encoder from Transformers for Biomedical Text Mining (BioBERT) model pre-trained on a corpus of biomedical text data. Here, the pre-trained BioBERT model may include a stack of multi-head self-attention layers. In some implementations, the operations further include fine-tuning the pre-trained Transformer model using an aggregation of training data including human-annotated labels, labels generated using the regular expression rules, and labels generated by the pre-trained Transformer model. In some examples, the operations further include storing the fine-tuned Transformer model in memory hardware in communication with the data processing hardware. The operations may further include transmitting the fine-tuned Transformer model to one or more computing devices over a network.

[0005] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving input data including unstructured text representing a sequence of one or more terms. For each respective sequence of terms, the operations include generating a corresponding Line of Therapy (LoT) pseudo-label using regular expression rules to indicate whether the respective sequence of terms includes LoT information, generating a corresponding LoT indicator using a pre-trained Transformer model to predict whether the respective sequence of terms includes LoT information, and determining a corresponding LoT-indicated loss based on the corresponding LoT pseudo-label and the corresponding LoT indicator. The operations also include fine-tuning the pre-trained Transformer model based on the LoT-indicated loss determined for the sequence of one or more terms. Embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, the operations further include: for each respective sequence of terms, generating a corresponding classification pseudo-label indicating a classification of the respective sequence of terms using regular expression rules; generating a corresponding LoT classification using a pre-trained Transformer model to predict a classification of the respective sequence of terms; and determining a corresponding LoT classification loss based on the corresponding classification pseudo-label and the corresponding LoT classification; and fine-tuning the pre-trained Transformer model based on the LoT classification loss determined for the one or more sequences of terms. In these embodiments, the classification of each sequence of terms indicates a particular LoT step from a set of LoT steps associated with the respective sequence of terms. The input data may include a plurality of clinical trial documents including unstructured text.

[0006] In some examples, the operations further include updating the regular expression rules based on the corresponding LoT classification loss. The pre-trained Transformer model includes a Bidirectional Encoder from Transformers for Biomedical Text Mining (BioBERT) model pre-trained on a corpus of biomedical text data. Here, the pre-trained BioBERT model may include a stack of multi-head self-attention layers. In some implementations, the operations further include fine-tuning the pre-trained Transformer model using an aggregation of training data including human-annotated labels, labels generated using the regular expression rules, and labels generated by the pre-trained Transformer model. In some examples, the operations further include storing the fine-tuned Transformer model in memory hardware in communication with the data processing hardware. The operations may further include transmitting the fine-tuned Transformer model to one or more computing devices over a network.

[0007] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram of an example system that executes machine learning components to process input data. [Figure 2] Figures 2A and 2B are schematic diagrams of the pre-training and fine-tuning stages for training the BioBERT model. [Figure 3] FIG. 1 is a schematic diagram of an exemplary BioBERT model. [Figure 4] FIG. 1 is a process flow diagram of an example training process for training one or more machine learning components. [Figure 5] 1 is a table providing a description of different trained multi-label LoT classifiers. [Figure 6] FIG. 6 is a table providing the number of data samples for training and testing different trained multi-label LoT classifiers from FIG. 5. [Figure 7] 6 is a table providing the distribution of LoT classes for training different trained multi-label LoT classifiers from FIG. 5. [Figure 8] 10 is a table providing the number of data samples per LoT classification in the ground truth data. [Figure 9] 6 is a table showing precision, recall, and F1 scores of different trained multi-label LoT classifiers from FIG. 5. [Figure 10] 6 is a table showing coverage analysis of different trained multi-label LoT classifiers from FIG. 5 using unseen datasets. [Figure 11] 1 is a graph showing receiver operating characteristic curves illustrating the true positive rate versus the false positive rate for each of the LoT classifications. [Figure 12]FIG. 10 is a schematic diagram of one or more other users executing the trained machine learning component. [Figure 13] 1 is a flowchart of an exemplary configuration of operations for a computer implementation using a weakly supervised deep learning model for LoT identification from clinical documents. [Figure 14] FIG. 1 is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein.

[0009] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE INVENTION

[0010] The drug development process is a complex procedure, of which clinical trials are one of the long and expensive components. Clinical trial documents contain a vast amount of clinical information regarding various entities, including disease entities, drug names, lines of therapy, etc. Accurately extracting and classifying these entities can facilitate the clinical trial design process. A line of therapy (LoT) is an ordered series of treatments administered to a patient during their disease progression. For example, an LoT for cancer treatment may include surgery first, followed by chemotherapy and radiation therapy. In this example, surgery is the first line of treatment, and chemotherapy and radiation therapy are the second line of treatment. LoT is an important concept used in many different clinical scenarios. For example, in clinical trials, LoT information from previous treatments can be used to include or exclude patients, and then the current treatment can be placed as the next LoT. LoT can also be reported in drug regulatory approval applications. Furthermore, physicians can select a patient's LoT based on their condition and provide treatment based on established guidelines related to that LoT.

[0011] LoT information can be used in a variety of contexts, such that LoT information can be applied in several different ways. For example, when designing a clinical trial for a drug, a clinical trial designer may need to gather information from other clinical trials of similar drugs. In particular, a clinical trial designer may need to know which pivotal trials were used for the approval of a particular drug and leverage information from those pivotal trials. However, regulatory approval documents may not explicitly state the names and / or National Clinical Trial (NCT) numbers of the corresponding clinical trials. This is known as a "trial matching problem," in which a clinical trial designer must find the clinical trials used for the approval of a particular drug. For example, if a clinical trial designer wants to know which clinical trials were used for the regulatory approval of the drug "Accrufer" with an approval date of July 25, 2019, the answer may include clinical trials "NCT01340872," "NCT01352221," and "NCT02968368." To that end, various entities from the clinical trial documents and the relationships between those entities common in the regulatory approval can be used to identify the specific clinical trial. In some examples, documents from a particular clinical trial used to approve a particular drug may contain entities such as disease entities, LoT, treatment regimens, drug names, concomitant treatments, drug combinations, biomarkers, disease subtypes, pathologies, risk categories, patient demographics, and age cutoffs. Therefore, accurately extracting and classifying these entities can solve the trial matching problem. Furthermore, when a drug has multiple approvals, different indications may be used for the different approvals. Thus, the "unique indication identification problem" is when a clinical trial designer needs to know which indication was used for a particular approval. LoT information and the other aforementioned entities can be similarly extracted and classified to distinguish between multiple approvals of the same drug.

[0012] Despite the above, clinical trial documents contain text in an unstructured text format. Therefore, any LoT information contained in clinical trial documents is unstructured. As a result, regular expressions and predefined rules may not always accurately identify LoT information from unstructured text formats. Supervised deep learning models may be a useful alternative for extracting LoT information. However, developing a supervised deep learning model is cumbersome because a large amount of labeled data is required. Additionally, accurately manually labeling LoT information is costly, time-consuming, and requires specialized knowledge.

[0013] To that end, embodiments herein are directed to a method and system for a weakly supervised machine learning model for LoT identification from clinical documents. The method includes receiving input data including unstructured text representing a sequence of one or more terms. For each respective sequence of terms, the method includes using regular expression rules to generate a corresponding Line of Therapy (LoT) pseudo-label indicating whether the respective sequence of terms contains LoT information, using a pre-trained Transformer model to generate a corresponding LoT indicator that predicts whether the respective sequence of terms contains LoT information, and determining a corresponding LoT-indicated loss based on the corresponding LoT pseudo-label and the corresponding LoT indicator. The method then includes fine-tuning the pre-trained Transformer model based on the LoT-indicated loss determined for the sequence of one or more terms.

[0014] Further, for each respective sequence of terms, the method may include generating a corresponding classification pseudo-label indicating a classification of the respective sequence of terms using regular expression rules, generating a corresponding LoT classification using a pre-trained Transformer model that predicts a classification of the respective sequence of terms, and determining a corresponding LoT classification loss based on the corresponding classification pseudo-label and the corresponding LoT classification. The method then includes fine-tuning the pre-trained Transformer model based on the LoT classification loss determined for the one or more sequences of terms. As will become apparent, in some implementations, the pre-trained Transformer model is fine-tuned on aggregate training data.

[0015] 1 , in some implementations, an exemplary system 100 includes a processing system 110, which may be a single computer, multiple computers, a user device, or a distributed system (e.g., a cloud computing environment). The processing system 110 has fixed or scalable computing resources (e.g., data processing hardware) 112 and / or storage resources (e.g., memory hardware) 114. The data processing hardware 112 and memory hardware 114 may reside on the user device and / or the cloud computing environment. The processing system 110 includes a machine learning component 170 that processes input data 102 to identify and extract multiple features and identify a line of therapy (LoT). The input data 102 may be stored in the memory hardware 114 of the processing system 110 or may be received from another entity (e.g., another processing system) via a network or other communication channel.

[0016] The input data 102 may include training data, such as a large dataset of clinical trial documents with unstructured text. That is, the input data 102 may include sequences of terms 104 (e.g., sentences), each of which may or may not include LoT information. The processing system 110 trains one or more of the machine learning components 170 by processing the input data 102. For example, the processing system 110 may train a multi-label LoT classifier 160 to identify and extract sequences of terms 104 from the unstructured text of the input data 102 and classify a different LoT for each extracted sequence of terms 104. Here, each sequence of terms 104 corresponds to a sentence from the input data 102 and may represent LoT information. The LoT information may be any textual information related to one or more treatments received by a patient for a particular disease or treatment.

[0017] In some implementations, the machine learning component 170 includes a regular expression (regex) module 120, a LoT sentence detector 130, a multi-label LoT label generator 140, a weakly supervised labeling model 150, and a multi-label LoT classifier 160. The regex module 120 is configured to annotate sequences of terms 104 from unstructured text in the input data 102. That is, not all sequences of terms 104 from the unstructured text contain LoT information; therefore, the regex module 120 determines whether each sequence of terms 104 contains LoT information. Furthermore, the regex module 120 classifies sequences of terms 104 determined to contain LoT information into a particular LoT class.

[0018] In some examples, the regex module 120 is a rule-based, deterministic model that generates pseudo-labels 122, 124 for each sequence of terms 104 from the input data 102. The sequence of terms 104 may include one or more words or sentences from the unstructured text of the input data 102. That is, the regex module 120 is a regex-based annotation module that generates the pseudo-labels 122, 124 by processing each sequence of terms 104 using an initial set of regular expression rules. A regular expression is a sequence of characters / terms that specifies a particular pattern in text. Thus, the regex module 120 generates the pseudo-labels 122, 124 and determines whether the unstructured text of the input data 102 meets the matching threshold of any of the regular expression rules.

[0019] For example, if the regex module 120 determines that the respective sequence of terms 104 satisfies the matching threshold of at least one of the initial regular expression rules, the regex module 120 generates a corresponding LoT pseudo-label 122 indicating that the respective sequence of terms 104 contains LoT information. On the other hand, if the regex module 120 determines that the respective sequence of terms 104 does not satisfy the matching threshold of at least one of the initial regular expression rules, the regex module 120 generates a corresponding LoT pseudo-label 122 indicating that the respective sequence of terms 104 does not contain LoT information. Simply put, the LoT pseudo-label 122 indicates whether a particular sentence from the input data 102 contains LoT information.

[0020] For each sequence of terms 104 for which the regex module 120 has generated a corresponding LoT pseudo-label 122 indicating that the sequence of terms 104 contains LoT information, the regex module 120 also generates a corresponding classification pseudo-label 124 indicating the classification of the respective sequence of terms 104, where the classification of each sequence of terms 104 indicates a particular LoT step from a series of LoT steps associated with the respective sequence of terms 104. For example, each sequence of terms 104 may be associated with the third LoT step from a series of five LoT steps, and thus the classification indicates that the respective sequence of terms 104 is associated with the third LoT step. Thus, the regex module 120 can generate a corresponding classification pseudo-label 124 for each sequence of terms 104 if the respective sequence of terms 104 matches a regular expression rule. For example, the regex module 120 can process a sequence of terms 104 corresponding to "for the treatment of adult patients with multiple myeloma who have received at least three prior lines of therapy, including a proteasome inhibitor (PI) and an immunomodulatory agent as monotherapy, or who are refractory to both a PI and an immunomodulatory agent," detect "at least three" keywords, and generate a corresponding LoT classification pseudo-label 124 representing "LoT4+" as the classification for the sequence of terms 104. In this example, "LoT4+" indicates that the LoT information from the sequence of terms 104 is associated with the fourth LoT step from the series of LoT steps.

[0021] The processing system 110 trains the LoT sentence detector 130 to identify whether a sequence of terms 104 from the input data 102 contains LoT information. For example, the LoT sentence detector 130 can include a pre-trained Bidirectional Encoding Representation by Transformers for Biomedical Text Mining (BioBert) model 300 (e.g., a pre-trained Transformer model), whereby the processing system 110 trains the BioBert model 300 to detect the presence of LoT information. Note that the LoT sentence detector 130 is a neural network model that non-deterministically detects the presence of LoT information (in contrast to the regex module 120). The corresponding LoT pseudo-labels 122 generated for each respective sequence of terms 104 serve as corresponding ground truth labels for training the LoT sentence detector 130.

[0022] 2A and 2B , in some embodiments, the processing system 110 executes a training process 200 to train a BioBERT model 300. The BioBERT model 300 is a domain-specific language representation model. The training process 200 includes a pre-training stage 210 and a fine-tuning stage 220. In the pre-training stage 210, the training process 200 trains the BioBERT model 300 using a pre-training data corpus 212 containing text from popular websites and / or books. The pre-training stage 210 provides parameter weight initialization for the BioBERT model 300. The BioBERT model 300 is also pre-trained using large-scale biomedical text, such as literature abstracts and / or full-text articles from biomedical and life sciences, in addition to, or instead of, popular websites and / or books.

[0023] 2A and 2B , after the BioBert model 300 is pre-trained during the pre-training phase 210, the pre-trained BioBERT model 300 is fine-tuned in the fine-tuning phase 220 using the input data 102 and the pseudo-labels 122, 124. As will become apparent, the pre-trained BioBERT model 300 can be fine-tuned to perform a variety of tasks. For example, the LoT sentence detector 130 uses the BioBert model 300 to generate a corresponding LoT indicator 132 for each sequence of terms 104. The LoT indicator 132 is a binary classification that indicates “true” (e.g., “1”) if the BioBert model 300 of the LoT sentence detector 130 determines that the sequence of terms 104 contains LoT information, and indicates “false” (e.g., “0”) if the LoT sentence detector 130 determines that the sequence of terms 104 does not contain LoT information. Thereafter, a loss module 240 determines a corresponding LoT-indicating loss 242 based on a comparison between the corresponding LoT indicator 132 generated by the BioBert model 300 and the corresponding LoT pseudo-label 122 generated by the regex module 120 (FIG. 1). Here, the corresponding LoT indicator 132 and the corresponding LoT pseudo-label 122 are each generated from the same respective sequence of terms 104, whereby the corresponding LoT pseudo-label 122 serves as a ground truth label. The fine-tuning stage 220 fine-tunes the BioBert model 300 (e.g., updates parameters of the BioBert model 300) based on the LoT-indicating loss 242 determined for the one or more sequences of terms 104. Thus, the fine-tuning stage 220 fine-tunes the BioBert model 300 to detect whether the sequence of terms 104 contains LoT information.

[0024] Referring back to FIG. 1 , the processing system 110 trains the multi-label LoT label generator 140 to classify sequences of terms 104 containing LoT information. Similarly, the multi-label LoT label generator 140 can include a pre-trained BioBert model 300, such that the processing system 110 trains the BioBert model 300 to classify the LoT information into specific classifications. In some examples, the multi-label LoT label generator 140 shares the same BioBert model 300 as the LoT sentence detector 130. In other examples, the multi-label LoT label generator 140 includes a different BioBert model 300 from the LoT sentence detector 130. Notably, the multi-label LoT label generator 140 is a neural network model that non-deterministically classifies LoT information (as opposed to the regex module 120). The corresponding classification pseudo-labels 124 generated for each respective sequence of terms 104 serve as corresponding ground truth labels for training the multi-label LoT label generator 140. The LoT classification 142 output from the multi-label LoT label generator 140 may include a single classification or multiple classifications. For example, the classification may indicate that the LoT information is associated with a second LoT step from a series of LoT steps. On the other hand, the classification may indicate that the LoT information is associated with more than a second LoT step from a series of LoT steps (e.g., a second LoT step, a third LoT step, a fourth LoT step, etc.).

[0025] 2A and 2B , in some implementations, the fine-tuning stage 220 of the training process 200 fine-tunes the BioBert model 300 to predict accurate LoT classifications 142 for sequences of terms 104 that include LoT information. For example, the multi-label LoT label generator 140 uses the BioBert model 300 to generate a corresponding LoT classification 142 for each sequence of terms 104. The LoT classification 142 predicts the classification of each sequence of terms 104 by indicating a particular LoT step from a series of LoT steps. The loss module 240 then determines a corresponding LoT classification loss 244 based on a comparison of the corresponding LoT classification 142 generated by the BioBert model 300 with the corresponding classification pseudo-label 124 generated by the regex module 120 ( FIG. 1 ). Here, the corresponding LoT classification 142 and the corresponding classification pseudo label 124 are each generated from the same respective sequence of terms 104, whereby the corresponding classification pseudo label 124 serves as a ground truth label. The fine-tuning stage 220 fine-tunes the BioBert model 300 (e.g., updates parameters of the BioBert model 300) based on the LoT classification loss 244 determined for the one or more sequences of terms 104. Thus, the fine-tuning stage 220 fine-tunes the BioBert model 300 to classify a particular LoT step associated with the LoT information contained in the particular sequence of terms 104.

[0026] In machine learning, supervised machine learning models require large amounts of labeled training data. However, labeling large amounts of data is time-consuming and expensive because it requires manual annotation by domain experts. In contrast, the fine-tuning stage 220 of the training process 200 utilizes weakly annotated data (e.g., semi-supervised training data) to train the BioBert model 300. That is, the fine-tuning stage 220 relies on pseudo-labels 122, 124 generated by regular expression rules in the regex module 120 (FIG. 1).

[0027] Referring back to FIG. 1 , the weakly supervised labeling model 150 is configured to generate a collection of training data 152 for training a multi-label LoT classifier. In particular, the weakly supervised labeling model 150 obtains a large amount of training data from various sources, such as the LoT classifications 142 generated by the multi-label LoT label generator 140, the pseudo-labels 122, 124 generated by the regex module 120, and externally labeled training data included in the input data 102. Here, the externally labeled training data included in the input data 102 may include labeled training data from an external source, such as a commercial vendor's website. Thus, the externally labeled training data may represent a small portion of data already labeled by human annotators.

[0028] To that end, the weakly supervised labeling model 150 can aggregate labels from these sources using a weakly supervised technique that selects a corresponding pseudo-label for each respective sequence of terms 104 based on the maximum votes received by the label (e.g., MultilabelVoter). That is, for each respective sequence of terms 104, the weakly supervised labeling model 150 can receive one or more different labels from a human annotator, the regex module 120, and / or the LoT sentence detector 130. Thus, the weakly supervised labeling model 150 can employ an external MultilabelVoter model that determines a confidence value associated with the label from each source and selects the label with the highest confidence value. The weakly supervised labeling model 150 then outputs the aggregated training data 152 to the multi-label LoT classifier 160. The multi-label LoT classifier 160, like the multi-label LoT label generator 140, includes a pre-trained BioBert model 300. The pre-trained BioBert model 300 may be the same model as the LoT sentence detector 130 and / or the multi-label LoT label generator 140, or a different model. The loss module 240 then determines a corresponding LoT classification loss 244 based on a comparison of the corresponding LoT classifications 142 generated by the multi-label LoT classifier 160 using the BioBert model 300 with the corresponding pseudo-labels from the aggregated training data 152. That is, the fine-tuning stage 220 trains the multi-label LoT classifier 160 of the BioBert model 300 using the pseudo-labels from the aggregated training data 152. The aggregated training data 152 may include pseudo-labels from human-annotated data included in the input data 102, the LoT classifications 142, and / or the classification pseudo-labels 124. In contrast, the fine-tuning stage 220 trains the multi-label LoT label generator 140 of the BioBert model 300 using ground truth labels that include only the classification pseudo-labels 124.Briefly, the multi-label LoT label generator 140 is trained to classify LoT information based on the classification pseudo-labels 124, while the multi-label LoT classifier 160 is trained to classify LoT information based on an aggregation 152 of training data.

[0029] 2A and 2B , the multi-label LoT classifier 160 can similarly use the BioBert model 300 to predict the corresponding LoT classification 142 for each sequence of terms 104. Then, aggregated labels (e.g., aggregated data) are used. The multi-label LoT classifier 160 can be a fine-tuned BioBert model trained using labels generated by the weakly supervised labeling model 150. The aggregated data can be used, for example, as new training data (e.g., input data 102) for use in training the multi-label LoT classifier 160. The loss module 240 then determines the corresponding LoT classification loss 244 based on a comparison of the corresponding LoT classification 142 generated by the multi-label LoT classifier 160 using the BioBert model 300 with the corresponding pseudo labels from the aggregated training data 152. That is, the fine-tuning stage 220 trains the multi-label LoT classifier 160 of the BioBert model 300 using the pseudo labels from the aggregated training data 152. The aggregated training data 152 may include pseudo-labels from the input data 102, the LoT classifications 142, and / or the human-annotated data included in the classification pseudo-labels 124. In contrast, the fine-tuning stage 220 trains the multi-label LoT label generator 140 using the BioBert model 300 with ground truth labels from the classification pseudo-labels 124 only.

[0030] Referring now to FIG. 3 , in some implementations, the BioBert model 300 includes a Transformer 320 that processes a sequence of terms 104 to identify and extract features related to LoT information from clinical trial documents. Text segments derived from the clinical trial documents (e.g., the sequence of terms 104) are tokenized into a series of tokens 310 (some of which may include a special token SEP indicating separation between adjacent sentences) for input to the Transformer 320. These text segments can be generated using a Coherent Accommodative Text Segmentation (CATS) framework. The Transformer 320 can take various forms, including, for example, a Transformer-based machine learning model or an ensemble of models for natural language processing (NLP), such as BERT. Thus, the Transformer 320 can include a stack of multi-head self-attention layers. For example, the stack of multi-head self-attention layers can include Transformer layers, or conformer layers instead of Transformer layers. NLP can be applied to various biomedical applications, such as for use in identifying tumor status from unstructured medical resonance imaging (MM) reports, lung cancer stage from pathology reports, and cancer staging information from narrative electrical health record (EHR) data to extract features such as a patient's disease, age, sex, and / or racial attributes. The output of the Transformer 320 may include a set of segments 330i-n that can be further labeled using various techniques described in FIG. 1.

[0031] FIG. 4 is a process flow diagram 400 for training one or more machine learning components 170 (FIG. 1). The training process begins with a set of source text 410. The source text 410 may include information such as study participation criteria 412 and / or the full name of the study 414. A LoT detection step 420 then processes the source text 410. For example, in step 422, the regex module 120 annotates the source text 410 using initial regex expression rules. Then, in step 424, the pre-trained BioBERT model can be fine-tuned based on the LoT indicators 132 and corresponding LoT pseudo-labels 122 to detect line of therapy sentences. Optionally, in step 432, sentences from the source text 410 that include LoT information may be filtered or extracted from the source text 410. For example, only studies from clinical documents that have at least LoT sentences can be filtered and used to train the machine learning component 170.

[0032] The extracted sentences with LoT information can be used for further model training 440 (e.g., training the multi-label LoT label generator 140). More specifically, in step 442, the regex module 120 can annotate the extracted sentences using initial regular expression rules. Thus, in step 444, the BioBERT model 300 of the multi-label LoT label generator 140 can be further fine-tuned based on determining the LoT classification loss 244. Validating the classified LoT can include, in step 446, determining whether the classification is appropriate and updating the initial regular expression rules of the regex module 120 accordingly.

[0033] In some examples, the extracted studies are annotated using the weakly supervised labeling model 112d described above in FIG. 1. As described above, the weakly supervised labeling model 150 can generate a training data aggregation 152 that includes external source LoT classifications labeled at the study (e.g., human) level 452, pseudo-labels 122, 124 (e.g., regex labels 545) generated by the regex module 120, and labels generated by the multi-label LoT label generator 140. The line of therapy classification model 456 can also be further trained via the labels generated by the BioBERT model 300. The weakly supervised labeling model 150 generates the training data aggregation 152 using weakly supervised techniques as described in FIG. 1. The BioBERT model 300 of the multi-label LoT classifier 160 can be further fine-tuned in step 464 by training using the training data aggregation 152 to classify LoT information.

[0034] In some embodiments, the classified LoT information may be validated using various datasets in step 466. That is, the validated LoT information is manually validated against various annotated studies. In some examples, precision and recall metrics may be utilized to evaluate the identified line of treatment. The precision score quantifies the number of positive class predictions that actually belong to the positive class. The recall score quantifies the number of positive class predictions made from all positive examples in the dataset. The Fl score provides a single score that combines both the precision score and the recall score. By way of example, use of the machine learning component 170 as described herein may result in precision and recall scores, including, but not limited to, a precision score of approximately 0.81, a recall score of approximately 0.97, and an Fl score of approximately 0.88.

[0035] To verify the functionality of system 100, different versions of system 100 were generated and tested using different training datasets and annotation / labeling functions. As a first example, clinical trial data was collected from publicly available real-time research and development data for the pharmaceutical industry (e.g., Citeline). Approximately 10,205 sentences were extracted from approximately 8,117 studies. These sentences included official text and patient inclusion criteria. The sentences were annotated using an initial version of regex module 112a. Figure 5 is a table 500 providing a description of the different versions of system 100 generated. "Model 1" was trained using approximately 7,677 annotated sentences, with a validation set size of approximately 2,528. "Model 1" achieved a precision score of approximately 0.62, a recall score of approximately 0.79, and an Fl score of approximately 0.69. Figure 6 is a table 600 providing the number of data samples for the training and test sets of the model described in table 500 of Figure 5. FIG. 7 is a table 700 that defines and illustrates the distribution of LoT classifications for different fine-tuned BioBERT models 300 of FIG.

[0036] By analyzing the validation set output, the regex module 120 was updated to create a second version of the model, i.e., "Model 2." Model 2 was trained using approximately 7,677 annotated sentences, with a validation set size of approximately 2,528. "Model 2" achieved a precision score of approximately 0.79, a recall score of approximately 0.86, and an Fl score of 0.82. The regex module 120 was again updated based on the validation set output of Model 2.

[0037] We collected training data for "Model 3" using a weakly supervised labeling model 150. More specifically, for "Model 3," we generated a set of training data using the output from "Model 2." Another set of training data was generated using the updated regex module 120 as the labeling function. The final set of labeled data came from labels created by an external data source, Citeline. The output from "Model 3" was aggregated using a multi-label LoT classifier 160. "Model 3" achieved a precision score of approximately 0.81, a recall score of approximately 0.95, and an Fl score of approximately 0.87. To develop "Model 4," we evaluated the performance of a pre-trained BioBERT model 300 using a separate test set of approximately 160 samples. These samples were manually annotated by clinical trial domain experts. "Model 4" achieved a precision score of approximately 0.81, a recall score of approximately 0.97, and an Fl score of approximately 0.88. FIG. 8 is a table 800 showing the number of data samples in each of the line of treatment classes in the ground truth data.

[0038] Using a weakly supervised, fine-tuned BioBERT model appears to perform better than rule- and regular expression-based models for classifying LoT information. FIG. 9 is a table 800 showing the precision, recall, and Fl scores of the different models in table 500 of FIG. 5. Based on the performance metrics in table 800, it is clear that the weakly supervised model improves the recall score. On a validation dataset of approximately 160 studies, system 100 achieves a precision score of approximately 0.81, a recall score of approximately 0.97, and an Fl score of approximately 0.88.

[0039] Figure 10 is a table 1000 showing a coverage analysis of the various models of table 500 for untouched datasets (i.e., datasets not seen by the various models during training). Figure 11 is a graph 1100 showing receiver operating characteristic curves showing the true positive rate versus the false positive rate for each of the line of treatment classes.

[0040] 12 is a process flow diagram 1200 illustrating a method for classifying LoT information clinical trial documents. In some examples, the trained one or more machine learning components 170 reside and execute on the processing system 110 such that the processing system 110 receives input data 102 from other computing devices associated with the user 1202 that communicate with the machine learning components 170. In other examples, the processing system 110 transmits the trained one or more machine learning components 170 to the other computing devices associated with the user 1202.

[0041] 13 is a flowchart of an exemplary configuration of operations for a computer-implemented method 1300 of a weakly supervised deep learning model for LoT identification from clinical documents. The method 1300 can be executed on data processing hardware 1410 (FIG. 14) using instructions stored in memory hardware 1420 (FIG. 14). The data processing hardware 1410 and memory hardware 1420 can reside on a processing system 110 (e.g., a user device and / or a cloud computing environment) corresponding to the computing device 1400 (FIG. 14).

[0042] At operation 1302, the method 1300 includes receiving input data 102 including unstructured text representing one or more sequences of terms 104. For each respective sequence of terms 104, the method 1300 performs operations 1304-1308. At operation 1304, the method 1300 includes using regular expression rules to generate a corresponding line of therapy (LoT) pseudo-label 122 that indicates whether the respective sequence of terms 104 includes LoT information. At operation 1306, the method 1300 includes using a pre-trained Transformer model 300 to generate a corresponding LoT indicator 132 that predicts whether the respective sequence of terms 104 includes LoT information. At operation 1308, the method 1300 includes determining a corresponding LoT-indicating loss 242 based on the corresponding LoT pseudo-label 122 and the corresponding LoT indicator 132. At operation 1310, the method 1300 includes fine-tuning the pre-trained Transformer model 300 based on the LoT-instructed loss 242 determined for the sequence 104 of one or more terms.

[0043] 14 is a schematic diagram of an exemplary computing device 1400 that can be used to implement the systems and methods described herein. Computing device 1400 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functions are merely exemplary and do not limit the scope of the invention(s) described and / or claimed herein.

[0044] Computing device 1400 includes processor 1410, memory 1420, storage device 1430, high-speed interface / controller 1440 connecting to memory 1420 and high-speed expansion port 1450, and low-speed interface / controller 1460 connecting to low-speed bus 1470 and storage device 1430. Each of components 1410, 1420, 1430, 1440, 1450, and 1460 are interconnected using various buses and may be mounted on a common motherboard or otherwise as desired. Processor 1410 can process instructions for execution within computing device 1400, including instructions stored in memory 1420 or storage device 1430 to display graphical information for a graphical user interface (GUI) on an external input / output device, such as display 1480 coupled to high-speed interface 1440. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and memory types, as desired. Also, multiple computing devices 1400 may be connected, each providing a portion of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0045] The memory 1420 stores information non-transiently within the computing device 1400. The memory 1420 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 1420 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 1400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0046] The storage device 1430 is capable of providing mass storage for the computing device 1400. In some implementations, the storage device 1430 is a computer-readable medium. In various different implementations, the storage device 1430 may be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional implementations, a computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 1420, the storage device 1430, or memory on the processor 1410.

[0047] The high-speed controller 1440 manages bandwidth-intensive operations of the computing device 1400, and the low-speed controller 1460 manages less bandwidth-intensive operations. Such duty allocation is merely exemplary. In some implementations, the high-speed controller 1440 is coupled to a memory 1420, a display 1480 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 1450 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 1460 is coupled to a storage device 1430 and a low-speed expansion port 1490. The low-speed expansion port 1490, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled, for example, via a network adapter, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router.

[0048] The computing device 1400, as shown, can be implemented in several different forms. For example, the computing device may be implemented as a standard server 1400a, or multiple such servers 1400a in a group, as a laptop computer 1400b, or as part of a rack server system 1400c.

[0049] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special-purpose or general-purpose, coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0050] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or in an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0051] The processes and logic flows described herein can be implemented by one or more programmable processors, also referred to as data processing hardware, that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be implemented by special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operably coupled to receive data from or transfer data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0052] To provide for user interaction, one or more aspects of the present disclosure can be implemented on a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to the web browser in response to a request received from the web browser on the user's client device.

[0053] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (1300), when executed on data processing hardware (1410), causing the data processing hardware (1410) to: receiving input data (102) including unstructured text representing a sequence of one or more terms (104); For each respective sequence of terms (104), generating a corresponding Line of Treatment (LoT) pseudo-label (122) using regular expression rules, the pseudo-label indicating whether each sequence of terms (104) contains LoT information; using a pre-trained Transformer model (300) to generate a corresponding LoT indicator (132) that predicts whether each of the sequences of terms (104) contains LoT information; and determining a corresponding LoT-indicated loss (242) based on the corresponding LoT pseudo-label (122) and the corresponding LoT indicator (132); fine-tuning the pre-trained Transformer model (300) based on the LoT-instructed loss (242) determined for the one or more sequences of terms (104); A computer-implemented method for performing an operation, comprising:

2. The operation is For each respective sequence of terms (104), generating a corresponding classification pseudo-label (124) indicating the classification of each of the sequences of terms (104) using the regular expression rules; using the pre-trained Transformer model (300) to generate a corresponding LoT classification (142) that predicts the classification of each of the sequences of terms (104); determining a corresponding LoT classification loss (244) based on the corresponding classification pseudo-label (124) and the corresponding LoT classification (142); fine-tuning the pre-trained Transformer model (300) based on the LoT classification loss (244) determined for the one or more sequences of terms (104); The computer-implemented method (1300) of claim 1 further comprising:

3. 3. The computer-implemented method of claim 2, wherein the classification of the respective sequence of terms indicates a particular LoT step from a set of LoT steps associated with the respective sequence of terms.

4. The computer-implemented method of claim 1 , wherein the input data comprises a plurality of clinical trial documents that include the unstructured text.

5. The computer-implemented method (1300) of any one of claims 1 to 4, wherein the operations further comprise updating the regular expression rules based on the corresponding LoT-indicated losses (242).

6. 6. The computer-implemented method of claim 1, wherein the pre-trained Transformer model comprises a Bidirectional Encoder from Transformers for Biomedical Text Mining (BioBERT) model pre-trained on a corpus of biomedical text data.

7. 7. The computer-implemented method (1300) of claim 6, wherein the pre-trained BioBERT model includes a stack of multi-head self-attention layers.

8. 8. The computer-implemented method of claim 1, wherein the operations further comprise fine-tuning the pre-trained Transformer model using an aggregation of training data including human-annotated labels, labels generated using the regular expression rules, and labels generated by the pre-trained Transformer model.

9. 9. The computer-implemented method (1300) of claim 1, wherein the operations further comprise storing the fine-tuned Transformer model (300) in memory hardware (720) in communication with the data processing hardware (710).

10. 10. The computer-implemented method (1300) of claim 1, wherein the operations further comprise transmitting the fine-tuned Transformer model (300) to one or more computing devices over a network (1208).

11. A system (100), comprising: data processing hardware; memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to: receiving input data (102) including unstructured text representing a sequence of one or more terms (104); For each respective sequence of terms (104), generating a corresponding Line of Treatment (LoT) pseudo-label (122) using regular expression rules, the pseudo-label indicating whether each sequence of terms (104) contains LoT information; using a pre-trained Transformer model (300) to generate a corresponding LoT indicator (132) that predicts whether each of the sequences of terms (104) contains LoT information; determining a corresponding LoT-indicated loss (242) based on the corresponding LoT pseudo-label (122) and the corresponding LoT indicator (132); fine-tuning the pre-trained Transformer model (300) based on the LoT-instructed loss (242) determined for the one or more sequences of terms (104); memory hardware for performing operations, including: A system comprising:

12. The operation is For each respective sequence of terms (104), generating a corresponding classification pseudo-label (124) indicating the classification of each of the sequences of terms (104) using the regular expression rules; using the pre-trained Transformer model (300) to generate a corresponding LoT classification (142) that predicts the classification of each of the sequences of terms (104); determining a corresponding LoT classification loss (244) based on the corresponding classification pseudo-label (124) and the corresponding LoT classification (142); fine-tuning the pre-trained Transformer model (300) based on the LoT classification loss (244) determined for the one or more sequences of terms (104); The system (100) of claim 11, further comprising:

13. The system (100) of claim 12, wherein the classification of the respective sequence of terms (104) indicates a particular LoT step from a set of LoT steps associated with the respective sequence of terms (104).

14. The system (100) of any one of claims 11 to 13, wherein the input data (102) comprises a plurality of clinical trial documents comprising the unstructured text.

15. The system (100) of any one of claims 11 to 14, wherein the operations further comprise updating the regular expression rules based on the corresponding LoT-indicated losses (242).

16. 16. The system of claim 11, wherein the pre-trained Transformer model comprises a Bidirectional Encoder from Transformers for Biomedical Text Mining (BioBERT) model pre-trained on a corpus of biomedical text data.

17. 17. The system (100) of claim 16, wherein the pre-trained BioBERT model includes a stack of multi-head self-attention layers.

18. 18. The system (100) of claim 11, wherein the operations further include fine-tuning the pre-trained Transformer model (300) using an aggregation of training (152) data that includes human-annotated labels, labels generated using the regular expression rules, and labels generated by the pre-trained Transformer model (300).

19. 19. The system (100) of claim 11, wherein the operations further comprise storing the fine-tuned Transformer model (300) in memory hardware (720) in communication with the data processing hardware (710).

20. 20. The system (100) of claim 11, wherein the operations further comprise transmitting the fine-tuned Transformer model (300) to one or more computing devices over a network (1208).

Images