Identifying therapeutic lines from clinical documents

Abstract

A method (1300) includes receiving input data (102), the input data including unstructured text representing one or more word sequences (104). For each respective word sequence, the method comprises: generating a corresponding treatment line (LoT) pseudo-marker (122), the corresponding treatment line pseudo-marker indicating whether the respective word sequence comprises LoT information; generating a corresponding LoT indicator (132), the corresponding LoT indicator predicting whether the corresponding word sequence includes LoT information; and determining a corresponding LoT indication loss based on the corresponding LoT pseudo-marker and the corresponding LoT indicator (242). The method further includes trimming the pre-trained transformer model (300) based on the LoT indication loss determined for the one or more word sequences.

Description

Technical Field

[0001] The present disclosure relates to identifying lines of treatment from clinical documents. Background Art

[0002] Clinical trial design (CTD) is a crucial part of drug development because it influences clinical trial length, protocol, patient recruitment, clinical endpoints, and the required comparators. Knowledge from earlier clinical trials of the same drug or different drugs can aid in the CTD process. Line of Therapy (LoT) refers to the sequence of different treatments given to a patient during their disease progression. LoT is an important concept in the context of clinical trials because it can relate to the patient population for which regulatory agencies approve drugs. For example, a regulatory agency may grant approval for a first drug only for patients who have failed standard of care (SoC) treatment (e.g., first-line treatment), in which case the approval of the first drug would be referred to as a second-line treatment approval. Similarly, if a second drug is approved only for patients who have failed SoC treatment and have also failed treatment with the first drug, the approval of the second drug would be referred to as a third-line treatment approval. Summary of the Invention

[0003] One aspect of the present disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the data processing hardware to perform operations for identifying lines of treatment from clinical documents using a weakly supervised deep learning model. The operations include receiving input data, the input data comprising unstructured text representing one or more word sequences. For each corresponding word sequence, the operations include: generating a corresponding line of treatment (LoT) pseudo-label using a regular expression rule, the corresponding line of treatment pseudo-label indicating whether the corresponding word sequence includes LoT information; generating a corresponding LoT indicator using a pre-trained transformer model, the corresponding LoT indicator predicting whether the corresponding word sequence includes LoT information; and determining a corresponding LoT indication loss based on the corresponding LoT pseudo-label and the corresponding LoT indicator. The operations also include fine-tuning the pre-trained transformer model based on the LoT indication loss determined for the one or more word sequences. Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operation further includes: for each corresponding word sequence: generating a corresponding classification pseudo-label using a regular expression rule, the corresponding classification pseudo-label indicating the classification of the corresponding word sequence; generating a corresponding LoT classification using a pre-trained transformer model, the corresponding LoT classification predicting the classification of the corresponding word sequence; and determining a corresponding LoT classification loss based on the corresponding classification pseudo-label and the corresponding LoT classification; and fine-tuning the pre-trained transformer model based on the LoT classification loss determined for one or more word sequences. In these implementations, the classification indication of the corresponding word sequence comes from a specific LoT step in a series of LoT steps associated with the corresponding word sequence. The input data may include a plurality of clinical trial documents, wherein the plurality of clinical trial documents include unstructured text.

[0004] In some examples, the operation further includes updating the regular expression rules based on the corresponding LoT classification loss. The pre-trained transformer model includes a Bidirectional Encoder from Transformers for Biomedical Text Mining (BioBERT) model pre-trained on a corpus of biomedical text data. Here, the pre-trained BioBERT model may include a stack of multi-head self-attention layers. In some implementations, the operation further includes fine-tuning the pre-trained transformer model using an aggregation of training data, wherein the aggregation of training data includes manually annotated tags, tags generated using regular expression rules, and tags generated by the pre-trained transformer model. In some examples, the operation further includes storing the fine-tuned transformer model in memory hardware that communicates with the data processing hardware. The operation may further include transmitting the fine-tuned transformer model to one or more computing devices via a network.

[0005] Another aspect of the present disclosure provides a system comprising data processing hardware and memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving input data, the input data comprising unstructured text representing one or more word sequences. For each corresponding word sequence, the operations include: generating a corresponding line of therapy (LoT) pseudo-label using a regular expression rule, the corresponding line of therapy pseudo-label indicating whether the corresponding word sequence includes LoT information; generating a corresponding LoT indicator using a pre-trained transformer model, the corresponding LoT indicator predicting whether the corresponding word sequence includes LoT information; and determining a corresponding LoT indication loss based on the corresponding LoT pseudo-label and the corresponding LoT indicator. The operations also include fine-tuning the pre-trained transformer model based on the LoT indication loss determined for the one or more word sequences. Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operation further includes: for each corresponding word sequence: generating a corresponding classification pseudo-label using a regular expression rule, the corresponding classification pseudo-label indicating the classification of the corresponding word sequence; generating a corresponding LoT classification using a pre-trained transformer model, the corresponding LoT classification predicting the classification of the corresponding word sequence; and determining a corresponding LoT classification loss based on the corresponding classification pseudo-label and the corresponding LoT classification; and fine-tuning the pre-trained transformer model based on the LoT classification loss determined for one or more word sequences. In these implementations, the classification indication of the corresponding word sequence comes from a specific LoT step in a series of LoT steps associated with the corresponding word sequence. The input data may include a plurality of clinical trial documents, wherein the plurality of clinical trial documents include unstructured text.

[0006] In some examples, the operation further includes updating the regular expression rules based on the corresponding LoT classification loss. The pre-trained transformer model includes a Bidirectional Encoder from Transformers for Biomedical Text Mining (BioBERT) model pre-trained on a corpus of biomedical text data. Here, the pre-trained BioBERT model may include a stack of multi-head self-attention layers. In some implementations, the operation further includes fine-tuning the pre-trained transformer model using an aggregation of training data, wherein the aggregation of training data includes manually annotated tags, tags generated using regular expression rules, and tags generated by the pre-trained transformer model. In some examples, the operation further includes storing the fine-tuned transformer model in memory hardware that communicates with the data processing hardware. The operation may further include transmitting the fine-tuned transformer model to one or more computing devices via a network.

[0007] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG1 is a diagram of an example system that executes a machine learning component to process input data.

[0009] Figures 2A and 2B are schematic diagrams of the pre-training and fine-tuning stages for training the BioBERT model.

[0010] Figure 3 is a schematic diagram of an example BioBERT model.

[0011] 4 is a process flow diagram of an example training process for training one or more machine learning components.

[0012] FIG5 is a table providing descriptions of different trained multi-label LoT classifiers.

[0013] FIG6 is a table providing the number of data samples used for training and testing different trained multi-label LoT classifiers from FIG5.

[0014] FIG. 7 is a table providing the distribution of LoT classes used to train the different trained multi-label LoT classifiers from FIG. 5 .

[0015] FIG8 is a table providing the number of data samples in each LoT category in the ground truth data.

[0016] FIG9 is a table showing the precision, recall, and F1 scores of different trained multi-label LoT classifiers from FIG5 .

[0017] FIG10 is a table showing coverage analysis of different trained multi-label LoT classifiers from FIG5 using an unseen dataset.

[0018] FIG11 is a graph showing receiver operating characteristic curves illustrating the true positive rate versus the false positive rate for each LoT classification.

[0019] FIG12 is a schematic diagram of one or more other users executing a trained machine learning component.

[0020] 13 is a flow diagram of an example arrangement of computer-implemented operations for identifying LoTs from clinical documents using a weakly supervised based deep learning model.

[0021] 14 is a schematic diagram of an example computing device that may be used to implement the systems and methods described herein.

[0022] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION

[0023] The drug development process is a complex one, with clinical trials being a lengthy and costly component. Clinical trial documents contain extensive clinical information on various entities, including disease entities, drug names, and lines of therapy. Properly extracting and classifying these entities can facilitate the clinical trial design process. A line of therapy (LoT) is a series of sequential treatments administered to a patient during their disease progression. For example, a LoT for cancer treatment might include surgery first, followed by chemotherapy and radiotherapy. In this example, surgery is the first-line treatment, and chemotherapy and radiotherapy are second-line treatments. LoT is a crucial concept used in many different clinical scenarios. For example, in clinical trials, LoT information on previous treatments can be used to include or exclude patients, and the current treatment can then be placed as the next LoT. LoTs can also be reported in drug application materials for regulatory approval. Furthermore, physicians can select a LoT for a patient based on their condition and provide treatment based on established guidelines for that LoT.

[0024] LoT information can be used in various contexts, resulting in a number of different applications. For example, when designing a clinical trial for a drug, clinical trial designers may need to gather information from other clinical trials of similar drugs. Specifically, clinical trial designers may need to know which pivotal studies were used for the approval of a particular drug in order to optimally utilize the information from those pivotal studies. However, regulatory approval documents may not explicitly mention the name and / or National Clinical Trial (NCT) number of the corresponding clinical trial. This is known as the "trial matching problem," in which clinical trial designers need to find the clinical trial that was used for the approval of a specific drug. For example, if a clinical trial designer wants to know which clinical trials were used for the regulatory approval of the drug "Accrufer," which had an approval date of July 25, 2019, the answers could include clinical trials "NCT01340872," "NCT01352221," and "NCT02968368." To this end, specific clinical trials can be identified using relationships between different entities from clinical trial documents and those entities commonly found in regulatory approvals. In some examples, the documentation from a specific clinical trial used to approve a specific drug includes entities such as disease entity, LoT, treatment course, drug name, co-therapy, drug combination, biomarker, disease subtype, pathology, risk category, patient demographics, age cutoff, etc. Therefore, correctly extracting and classifying these entities can solve the trial matching problem. In addition, when a drug has multiple approvals, different indications may be used for different approvals. Therefore, the "unique indication identification problem" is when clinical trial designers need to know which indication was used for a specific approval. LoT information and the other previously mentioned entities can be similarly extracted and classified to distinguish multiple approvals of the same drug.

[0025] Despite this, clinical trial documents include text in unstructured text formats. Therefore, any LoT information included in clinical trial documents is unstructured. Consequently, regular expressions and predefined rules may not always correctly identify LoT information from unstructured text formats. Supervised deep learning models can be a useful alternative for extracting LoT information. However, developing supervised deep learning models is cumbersome because they require large amounts of labeled data. Furthermore, correctly manually labeling LoT information is expensive and time-consuming, and requires expert knowledge.

[0026] To this end, implementations herein are directed to methods and systems for identifying LoTs from clinical documents based on a weakly supervised machine learning model. The method includes receiving input data comprising unstructured text representing one or more word sequences. For each corresponding word sequence, the method includes: generating a corresponding line of therapy (LoT) pseudo-label using a regular expression rule, wherein the corresponding line of therapy pseudo-label indicates whether the corresponding word sequence includes LoT information; generating a corresponding LoT indicator using a pre-trained transformer model, wherein the corresponding LoT indicator predicts whether the corresponding word sequence includes LoT information; and determining a corresponding LoT indication loss based on the corresponding LoT pseudo-label and the corresponding LoT indicator. Thereafter, the method includes fine-tuning the pre-trained transformer model based on the LoT indication loss determined for the one or more word sequences.

[0027] In addition, for each corresponding word sequence, the method may include: generating a corresponding classification pseudo-label using a regular expression rule, the corresponding classification pseudo-label indicating the classification of the corresponding word sequence; generating a corresponding LoT classification using a pre-trained transformer model, the corresponding LoT classification predicting the classification of the corresponding word sequence; and determining a corresponding LoT classification loss based on the corresponding classification pseudo-label and the corresponding LoT classification. Thereafter, the method includes fine-tuning the pre-trained transformer model based on the LoT classification loss determined for the one or more word sequences. As will become clear, in some implementations, the pre-trained transformer model is fine-tuned on the aggregated training data.

[0028] Referring now to FIG. 1 , in some implementations, an example system 100 includes a processing system 110, which can be a single computer, multiple computers, a user device, or a distributed system (e.g., a cloud computing environment). The processing system 110 has fixed or scalable computing resources (e.g., data processing hardware) 112 and / or storage resources (e.g., memory hardware) 114. The data processing hardware 112 and memory hardware 114 can reside on a user device and / or a cloud computing environment. The processing system 110 includes a machine learning component 170 that processes input data 102 to identify and extract multiple features to identify lines of treatment (LoTs). The input data 102 can be stored in the memory hardware 114 of the processing system 110 or received from another entity (e.g., another processing system) via a network or other communication channel.

[0029] The input data 102 may include training data, such as a large dataset of clinical trial documents with unstructured text. That is, the input data 102 may include word sequences 104 (e.g., sentences), each of which may or may not include LoT information. The processing system 110 trains one or more of the machine learning components 170 by processing the input data 102. For example, the processing system 110 may train a multi-label LoT classifier 160 to identify and extract word sequences 104 from the unstructured text of the input data 102, and classify different LoTs for each extracted word sequence 104. Here, each word sequence 104 may correspond to a sentence from the input data 102 and represent LoT information. LoT information can be any textual information related to one or more therapies a patient is receiving for a particular disease or treatment.

[0030] In some implementations, the machine learning component 170 includes a regular expression (regex) module 120, an LoT sentence detector 130, a multi-label LoT tokenizer 140, a weakly supervised tagging model 150, and a multi-label LoT classifier 160. The regex module 120 is configured to annotate word sequences 104 from the unstructured text of the input data 102. That is, not all word sequences 104 from the unstructured text include LoT information, and therefore the regex module 120 determines whether each word sequence 104 includes LoT information. Furthermore, the regex module 120 classifies word sequences 104 determined to include LoT information into a specific LoT category.

[0031] In some examples, the regex module 120 is a rule-based deterministic model that generates pseudo-tokens 122, 124 for each word sequence 104 from the input data 102. The word sequence 104 may include one or more words or sentences from the unstructured text of the input data 102. In other words, the regex module 120 is a regex-based annotation module that generates pseudo-tokens 122, 124 by processing each word sequence 104 using an initial set of regular expression rules. A regular expression is a sequence of characters / words that specifies a specific pattern in text. Therefore, the regex module 120 generates pseudo-tokens 122, 124 to determine whether the unstructured text of the input data 102 meets the matching threshold of any regular expression rule.

[0032] For example, when the regex module 120 determines that the corresponding word sequence 104 satisfies the matching threshold of at least one initial regular expression rule, the regex module 120 generates a corresponding LoT pseudo-tag 122, which indicates that the corresponding word sequence 104 includes LoT information. On the other hand, when the regex module 120 determines that the corresponding word sequence 104 fails to satisfy the matching threshold of at least one initial regular expression rule, the regex module 120 generates a corresponding LoT pseudo-tag 122, which indicates that the corresponding word sequence 104 does not include LoT information. In short, the LoT pseudo-tag 122 indicates whether a particular sentence from the input data 102 includes LoT information.

[0033] For each word sequence 104 for which the regex module 120 generates a corresponding LoT pseudo-tag 122 indicating that the word sequence 104 includes LoT information, the regex module 120 also generates a corresponding classification pseudo-tag 124 indicating a classification of the corresponding word sequence 104. Here, the classification of the words of the corresponding word sequence 104 indicates that they are from a particular LoT step in a series of LoT steps associated with the corresponding word sequence 104. For example, the corresponding word sequence 104 may be associated with the third LoT treatment from a series of five LoT treatments, and thus, the classification would indicate that the corresponding word sequence 104 is associated with the third LoT step. Therefore, if the corresponding word sequence 104 matches the regular expression rule, the regex module 120 may generate a corresponding classification pseudo-tag 124 for the corresponding word sequence 104. For example, the regex module 120 may process the word sequence 104 corresponding to “treatment of adult patients with multiple myeloma for monotherapy who have received at least three prior lines of therapy, including a proteasome inhibitor (PI) and an immunomodulatory agent, or who are double refractory to a PI and an immunomodulatory agent,” detect the keyword “at least three,” and generate a corresponding LoT classification pseudo-label 124 indicating ‘LoT 4+’ as the classification of the word sequence 104. In this example, ‘LoT 4+’ indicates that the LoT information from the word sequence 104 is associated with the fourth LoT step from a series of LoT steps.

[0034] The processing system 110 trains an LoT sentence detector 130 to identify whether a word sequence 104 from the input data 102 includes LoT information. For example, the LoT sentence detector 130 may include a pretrained Bidirectional Encoder Representations from Transformers (BioBert) model for biomedical text mining (e.g., a pretrained transformer model) 300, whereby the processing system 110 trains the BioBert model 300 to detect the presence of LoT information. Notably, the LoT sentence detector 130 is a neural network model that non-deterministically detects the presence of LoT information (in contrast to the regex module 120). The corresponding LoT pseudo-label 122 generated for each corresponding word sequence 104 serves as the corresponding ground truth label for training the LoT sentence detector 130.

[0035] Referring now to Figures 2A and 2B, in some implementations, the processing system 110 performs a training process 200 to train a BioBERT model 300. The BioBERT model 300 is a domain-specific language representation model. The training process 200 includes a pre-training phase 210 and a fine-tuning phase 220. In the pre-training phase 210, the training process 200 trains the BioBERT model 300 using a pre-training data corpus 212 comprising text from general websites and / or books. The pre-training phase 210 provides weight initialization for the parameters of the BioBERT model 300. The BioBERT model 300 is also pre-trained using large-scale biomedical text, which includes abstracts and / or full-text articles of biomedical and life science literature, as a supplement to or alternative to general websites and / or books.

[0036] Continuing with reference to Figures 2A and 2B, after the BioBERT model 300 is pre-trained during the pre-training phase 210, the pre-trained BioBERT model 300 is fine-tuned in the fine-tuning phase 220 using the input data 102 and pseudo-labels 122 and 124. As will become apparent, the pre-trained BioBERT model 300 is fine-tuned and can be fine-tuned to perform various tasks. For example, the LoT sentence detector 130 uses the BioBERT model 300 to generate a corresponding LoT indicator 132 for the corresponding word sequence 104. The LoT indicator 132 is a binary classification that indicates "true" (e.g., '1') when the BioBert model 300 of the LoT sentence detector 130 determines that the word sequence 104 includes LoT information, and indicates "false" (e.g., '0') when the LoT sentence detector 130 determines that the word sequence 104 does not include LoT information. Thereafter, the loss module 240 determines a corresponding LoT indicator loss 242 based on a comparison of the corresponding LoT indicator 132 generated by the BioBert model 300 and the corresponding LoT pseudo-label 122 generated by the regex module 120 ( FIG. 1 ). Here, the corresponding LoT indicator 132 and the corresponding LoT pseudo-label 122 are each generated from the same corresponding word sequence 104 , whereby the corresponding LoT pseudo-label 122 serves as the ground truth label. The fine-tuning stage 220 fine-tunes the BioBert model 300 (e.g., updates parameters of the BioBert model 300 ) based on the LoT indicator loss 242 determined for one or more word sequences 104 . Thus, the fine-tuning stage 220 fine-tunes the BioBert model 300 to detect whether the word sequence 104 includes LoT information.

[0037] Referring back to FIG. 1 , the processing system 110 trains the multi-label LoT tokenizer 140 to classify word sequences 104 comprising LoT information. Similarly, the multi-label LoT tokenizer 140 may include a pre-trained BioBert model, allowing the processing system 110 to train the BioBert model 300 to classify LoT information into a specific category. In some examples, the multi-label LoT tokenizer 140 shares the same BioBert model 300 as the LoT sentence detector 130. In other examples, the multi-label LoT tokenizer 140 includes a different BioBert model 300 than the LoT sentence detector 130. Notably, the multi-label LoT tokenizer 140 is a neural network model that non-deterministically classifies LoT information (in contrast to the regex module 120). The corresponding classification pseudo-label 124 generated for each corresponding word sequence 104 serves as the corresponding ground truth label for training the multi-label LoT tokenizer 140. The LoT classification 142 output from the multi-label LoT label generator 140 may include a single classification or multiple classifications. For example, the classification may indicate that the LoT information is associated with the second LoT step from a series of LoT steps. Alternatively, the classification may indicate that the LoT information is associated with the second LoT step or a higher LoT step (e.g., the second LoT step, the third LoT step, the fourth LoT step, etc.) from a series of LoT steps.

[0038] Referring again to FIG. 2A and FIG. 2B , in some implementations, the fine-tuning phase 220 of the training process 200 fine-tunes the BioBert model 300 to predict accurate LoT classifications 142 for word sequences 104 that include LoT information. For example, the multi-label LoT token generator 140 uses the BioBert model 300 to generate corresponding LoT classifications 142 for the corresponding word sequences 104. The LoT classifications 142 predict the classification of the corresponding word sequence 104 by indicating a specific LoT step from a series of LoT steps. Thereafter, the loss module 240 determines a corresponding LoT classification loss 244 based on comparing the corresponding LoT classifications 142 generated by the BioBert model 300 with the corresponding classification pseudo-labels 124 generated by the regex module 120 ( FIG. 1 ). Here, the corresponding LoT classifications 142 and the corresponding classification pseudo-labels 124 are each generated from the same corresponding word sequence 104, and thus the corresponding classification pseudo-labels 124 serve as the ground truth labels. The fine-tuning stage 220 fine-tunes the BioBert model 300 (e.g., updates parameters of the BioBert model 300) based on the LoT classification loss 244 determined for the one or more word sequences 104. Thus, the fine-tuning stage 220 fine-tunes the BioBert model 300 to classify a particular LoT step associated with the LoT information included in the particular word sequence 104.

[0039] In machine learning, supervised machine learning models require large amounts of labeled training data. However, labeling large amounts of data is time-consuming and expensive because it requires manual annotation by subject matter experts. In contrast, the fine-tuning phase 220 of the training process 200 utilizes weakly annotated data (e.g., semi-supervised training data) to train the BioBert model 300. Specifically, the fine-tuning phase 220 relies on pseudo-labels 122, 124 generated by the regular expression rules of the regex module 120 ( FIG. 1 ).

[0040] Referring back to FIG1 , the weakly supervised labeling model 150 is configured to generate an aggregate 152 of training data to train the multi-label LoT classifier. Specifically, the weakly supervised labeling model 150 obtains a large amount of training data from various sources, such as the LoT classification 142 generated by the multi-label LoT label generator 140, the pseudo labels 122 and 124 generated by the regex module 120, and externally labeled training data included in the input data 102. Here, the externally labeled training data included in the input data 102 may include labeled training data from external sources such as commercial vendor websites. Therefore, the externally labeled training data may represent a small portion of data that has been labeled by human annotators.

[0041] To this end, the weakly supervised labeling model 150 can aggregate labels from these sources using a weakly supervised technique that selects a corresponding pseudo-label for each corresponding word sequence 104 based on the maximum vote received by the label (e.g., a MultilabelVoter). That is, for each corresponding word sequence 104, the weakly supervised labeling model 150 can receive one or more different labels from human annotators, the regex module 120, and / or the LoT sentence detector 130. Thus, the weakly supervised labeling model 150 can employ an external MultilabelVoter model that determines the confidence value associated with the label from each source and selects the label with the highest confidence value. The weakly supervised labeling model 150 then outputs the aggregated training data 152 to the multi-label LoT classifier 160. Similar to the multi-label LoT token generator 140, the multi-label LoT classifier 160 includes the pre-trained BioBert model 300. The pre-trained BioBert model 300 can be the same model as the LoT sentence detector 130 and / or the multi-label LoT token generator 140, or a different model. Thereafter, the loss module 240 determines a corresponding LoT classification loss 244 based on comparing the corresponding LoT classifications 142 generated by the multi-label LoT classifier 160 using the BioBert model 300 with the corresponding pseudo-labels from the aggregated training data 152. That is, the fine-tuning stage 220 trains the multi-label LoT classifier 160 of the BioBert model 300 using the pseudo-labels from the aggregated training data 152. The aggregated training data 152 can include pseudo-labels from manually annotated data included in the input data 102, the LoT classifications 142, and / or the classification pseudo-labels 124. In contrast, the fine-tuning stage 220 trains the multi-label LoT token generator 140 of the BioBert model 300 using the ground truth labels that only include the classification pseudo-labels 124. In brief, the multi-label LoT label generator 140 is trained to classify LoT information based on the classification pseudo labels 124 , while the multi-label LoT classifier 160 is trained to classify LoT information based on the aggregation 152 of training data.

[0042] Referring again to Figures 2A and 2B , the multi-label LoT classifier 160 can similarly use the BioBert model 300 to predict the corresponding LoT classification 142 for the corresponding word sequence 104. Subsequently, the aggregated labels (e.g., aggregated data) are used. The multi-label LoT classifier 160 can be a fine-tuned BioBert model trained using labels generated by the weakly supervised labeling model 150. The aggregated data can be used as new training data (e.g., input data 102) for training, for example, the multi-label LoT classifier 160. Subsequently, the loss module 240 determines the corresponding LoT classification loss 244 based on comparing the corresponding LoT classification 142 generated by the multi-label LoT classifier 160 using the BioBert model 300 with the corresponding pseudo-labels from the aggregated training data 152. That is, the fine-tuning stage 220 uses the pseudo-labels from the aggregated training data 152 to train the multi-label LoT classifier 160 of the BioBert model 302. Aggregated training data 152 may include pseudo-labels from human-annotated data included in input data 102, LoT classifications 142, and / or classification pseudo-labels 124. In contrast, fine-tuning phase 220 uses BioBert model 300 to train multi-label LoT tag generator 140, where the ground truth labels come only from classification pseudo-labels 124.

[0043] Referring now to FIG3 , in some implementations, the BioBert model 300 includes a transformer 320 that processes a word sequence 104 to identify and extract features relevant to LoT information from a clinical trial document. A text segment obtained from the clinical trial document (e.g., the word sequence 104) is tokenized into a series of tokens 310 (some of which may include special tokens (SEPs) that indicate separations between adjacent sentences) for input into the transformer 320. These text segments can be generated using the consistency-aware text segmentation (CATS) framework. The transformer 320 can take various forms, including, for example, a transformer-based machine learning model or a model set for natural language processing (NLP), such as BERT. Thus, the transformer 320 can include a stack of multi-head self-attention layers. For example, the stack of multi-head self-attention layers can include a transformer layer or a transformer layer in place of a transformer layer. NLP can be applied to various biomedical applications, such as identifying tumor status from unstructured medical resonance imaging (MM) reports, identifying lung cancer stage from pathology reports, and identifying cancer stage information from narrative electrical health record (EHR) data to extract features such as the patient's disease, age, gender, and / or ethnicity. The output of transformer 320 may include a series of segments 330i-n, which can be further labeled using the various techniques described in FIG1.

[0044] FIG4 is a flowchart 400 of a process for training one or more machine learning components 170 ( FIG1 ). The training process begins with a collection of source text 410. Source text 410 may include information such as study inclusion criteria 412 and / or study official titles 414. Subsequently, a LoT detection step 420 processes the source text 410. For example, at step 422, the regex module 120 annotates the source text 410 using initial regular expression rules. Thereafter, at step 424, the pre-trained BioBERT model may be fine-tuned to detect treatment line sentences based on the LoT indicators 132 and corresponding LoT pseudo-labels 122. Optionally, at step 432, sentences from the source text 410 that include LoT information may be filtered or extracted. For example, only studies from clinical documents containing at least LoT sentences may be filtered and used to train the machine learning component 170.

[0045] The extracted sentences with LoT information can be used for further model training 440 (e.g., training of the multi-label LoT tokenizer 140). More specifically, at step 442, the regex module 120 can annotate the extracted sentences using the initial regular expression rules. Consequently, the BioBERT model 300 of the multi-label LoT tokenizer 140 can be further fine-tuned at step 444 based on the determination of the LoT classification loss 244. Validation of the classified LoTs can include determining whether the classification is appropriate at step 446 and updating the initial regular expression rules of the regex module 120 accordingly.

[0046] In some examples, the extracted studies are annotated using the weakly supervised labeling model 112d previously described in FIG. As previously described, the weakly supervised labeling model 150 can generate an aggregate 152 of training data, which includes externally sourced LoT classifications labeled at the study (e.g., human) level 452, pseudo-labels 122, 124 generated by the regex module 120 (e.g., regex labels 545), and labels generated by the multi-label LoT label generator 140. The treatment line classification model 456 can also be further trained using the labels generated by the BioBERT model 300. The weakly supervised labeling model 150 generates the aggregate 152 of training data using the weakly supervised technique described in FIG. The BioBERT model 300 of the multi-label LoT classifier 160 can be further fine-tuned at step 464 using the aggregate 152 of training data to classify LoT information.

[0047] In some implementations, the classified LoT information can be verified in step 466 using various data sets. That is, the verified LoT information is manually verified for various annotated studies. In some examples, precision and recall metrics can be used to evaluate the identified treatment lines. The precision score quantifies the number of positive class predictions that actually belong to the positive class. The recall score quantifies the number of positive class predictions made from all positive cases in the data set. The Fl score provides a single score that combines both the precision score and the recall score. By way of example, the use of one or more machine learning components 170 as described herein can result in precision scores and recall scores including, but not limited to, a precision score of approximately 0.81, a recall score of approximately 0.97, and an Fl score of approximately 0.88.

[0048] To validate the functionality of system 100, different versions of system 100 were generated and validated using different training datasets and annotation / labeling functions. As a first example, clinical trial data was collected from publicly available, real-time R&D data from the pharmaceutical industry (e.g., Citeline). Approximately 10,205 sentences were extracted from approximately 8,117 studies. These sentences included formal text and patient inclusion criteria. An initial version of regex module 112a was used to annotate the sentences. Figure 5 is Table 500, providing a description of the different versions of system 100 generated. "Model 1" was trained using approximately 7,677 annotated sentences, and a validation set size of approximately 2,528. "Model 1" achieved a precision score of approximately 0.62, a recall score of approximately 0.79, and an Fl score of approximately 0.69. Figure 6 is Table 600, providing the number of data samples used in the training and test sets for the models described in Table 500 of Figure 5. Figure 7 is Table 700, defining the distribution of LoT classifications for different fine-tuned versions of the BioBERT model 300 of Figure 5.

[0049] By analyzing the output of the validation set, the regex module 120 is updated and a second version of the model, "Model 2," is generated. Model 2 is trained using approximately 7,677 annotated sentences and a validation set size of approximately 2,528. "Model 2" achieves a precision score of approximately 0.79, a recall score of approximately 0.86, and an Fl score of 0.82. The regex module 120 is again updated based on the output of the validation set for Model 2.

[0050] Weak supervision techniques using a weakly supervised labeling model 150 were used to collect training data for "Model 3". More specifically, for "Model 3", a set of training data was generated using the output from "Model 2". Another set of training data was generated using the updated regex module 120 as the labeling function. The final set of labeled data came from labels created by the external data source Citeline. The output from "Model 3" was aggregated using a multi-label LoT classifier 160. "Model 3" achieved a precision score of approximately 0.81, a recall score of approximately 0.95, and an Fl score of approximately 0.87. The pre-trained BioBERT model 300 was used to develop "Model 4". The performance of "Model 4" was evaluated using a separate test set of approximately 160 samples. These samples were manually annotated by experts in the clinical trial field. "Model 4" achieved a precision score of approximately 0.81, a recall score of approximately 0.97, and an Fl score of approximately 0.88. Figure 8 is a table 800 providing the number of data samples in each treatment line category in the ground truth data.

[0051] Compared to rule-based and regular expression-based models, the weakly supervised fine-tuned BioBERT model appears to perform better for classifying LoT information. FIG9 is a table 800 showing the precision, recall, and Fl scores of the different models in Table 500 of FIG5 . Based on the performance metrics in Table 800 , it is clear that the weakly supervised model improves the recall score. On a validation dataset of approximately 160 studies, system 100 achieved a precision score of approximately 0.81, a recall score of approximately 0.97, and an Fl score of approximately 0.88.

[0052] Figure 10 is a table 1000 showing coverage analysis of various models of Table 500 on an untouched dataset (i.e., a dataset not seen by the various models during training). Figure 11 is a graph 1100 showing receiver operating characteristic curves showing the true positive rate versus the false positive rate for each treatment line category.

[0053] FIG12 is a process flow diagram 1200 illustrating a method for classifying LoT information clinical trial documents. In some examples, the trained one or more machine learning components 170 reside and execute on the processing system 110, such that the processing system 110 receives input data 102 from another computing device associated with a user 1202 that communicates with the machine learning component 170. In other examples, the processing system 110 transmits the trained one or more machine learning components 170 to the other computing device associated with the user 1202.

[0054] FIG13 is a flow chart illustrating an exemplary arrangement of operations for a computer-implemented method 1300 for identifying IoTs from clinical documents based on a weakly supervised deep learning model. Method 1300 may be executed on data processing hardware 1410 ( FIG14 ) using instructions stored on memory hardware 1420 ( FIG14 ). Data processing hardware 1410 and memory hardware 1420 may reside on a processing system 110 corresponding to a computing device 1400 ( FIG14 ) (e.g., a user device and / or a cloud computing environment).

[0055] At operation 1302 , method 1300 includes receiving input data 102 comprising unstructured text representing one or more word sequences 104 . For each corresponding word sequence 104 , method 1300 performs operations 1304 - 1308 . At operation 1304 , method 1300 includes generating a corresponding line of therapy (LoT) pseudo-label 122 using a regular expression rule, the corresponding LoT pseudo-label indicating whether the corresponding word sequence 104 includes LoT information. At operation 1306 , method 1300 includes generating a corresponding LoT indicator 132 using a pre-trained transformer model 300 , the corresponding LoT indicator predicting whether the corresponding word sequence 104 includes LoT information. At operation 1308 , method 1300 includes determining a corresponding LoT indication loss 242 based on the corresponding LoT pseudo-label 122 and the corresponding LoT indicator 132 . At operation 1310 , the method 1300 includes fine-tuning the pre-trained transformer model 300 based on the LoT indication loss 242 determined for the one or more word sequences 104 .

[0056] FIG14 is a schematic diagram of an example computing device 1400 that can be used to implement the systems and methods described in this document. Computing device 1400 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are exemplary only and are not meant to limit implementations of the inventions described and / or claimed in this document.

[0057] Computing device 1400 includes a processor 1410, memory 1420, storage device 1430, a high-speed interface / controller 1440 connected to memory 1420 and a high-speed expansion port 1450, and a low-speed interface / controller 1460 connected to a low-speed bus 1470 and storage device 1430. Each of components 1410, 1420, 1430, 1440, 1450, and 1460 is interconnected using various buses and may be mounted on a common motherboard or otherwise installed where appropriate. Processor 1410 may process instructions for execution within computing device 1400, including instructions stored in memory 1420 or on storage device 1430, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 1480 coupled to high-speed interface 1440. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and types of memory, where appropriate. Furthermore, multiple computing devices 1400 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0058] Memory 1420 stores information non-transitorily within computing device 1400. Memory 1420 can be a computer-readable medium, one or more volatile memory units, or one or more non-volatile memory units. Non-transitory memory 1420 can be a physical device used to store programs (e.g., instruction sequences) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 1400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0059] Storage device 1430 can provide mass storage for computing device 1400. In some implementations, storage device 1430 is a computer-readable medium. In various implementations, storage device 1430 can be a floppy disk device, a hard disk device, an optical disk device, a magnetic tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configuration). In other implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as memory 1420, storage device 1430, or memory on processor 1410.

[0060] High-speed controller 1440 manages bandwidth-intensive operations for computing device 1400, while low-speed controller 1460 manages less bandwidth-intensive operations. This division of responsibilities is exemplary only. In some implementations, high-speed controller 1440 is coupled to memory 1420, display 1480 (e.g., via a graphics processor or accelerator), and to high-speed expansion ports 1450, which can accept various expansion cards (not shown). In some implementations, low-speed controller 1460 is coupled to storage device 1430 and low-speed expansion ports 1490. Low-speed expansion ports 1490, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, or networking equipment (such as a switch or router), for example, via a network adapter.

[0061] As shown in the figures, computing device 1400 can be implemented in many different forms. For example, it can be implemented as a standard server 1400a or multiple implementations in a group of such servers 1400a, as a laptop computer 1400b, or as part of a rack server system 1400c.

[0062] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system that includes at least one programmable processor, which can be either special-purpose or general-purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0063] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., a magnetic disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receives machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0064] The processes and logic flows described in this specification can be performed by one or more programmable processors (also known as data processing hardware) that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by dedicated logic circuitry (e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit)). By way of example, processors suitable for executing computer programs include general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, a processor will receive instructions and data from a read-only memory or a random access memory, or both. The basic elements of a computer include a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operatively coupled to the one or more mass storage devices to receive data from them or transfer data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0065] To provide for user interaction, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), an LCD (liquid crystal display) monitor, or a touch screen) for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide for user interaction; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. Furthermore, a computer may interact with a user by sending and receiving documents to and from a device used by the user (e.g., by sending a web page to a web browser in response to a request received from a web browser on the user's client device).

[0066] Many implementations have been described. However, it will be appreciated that various modifications may be made without departing from the spirit and scope of this disclosure. Accordingly, other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method (1300) which, when executed on data processing hardware, causes (1410) the data processing hardware (1410) to perform operations comprising: receiving input data (102), the input data comprising unstructured text representing one or more word sequences (104); For each corresponding word sequence (104): generating a corresponding line of treatment (LoT) pseudo-tag (122) using a regular expression rule, wherein the corresponding line of treatment pseudo-tag indicates whether the corresponding word sequence (104) includes LoT information; generating a corresponding LoT indicator (132) using a pre-trained transformer model (300), wherein the corresponding LoT indicator predicts whether the corresponding word sequence (104) includes LoT information; and determining a corresponding LoT indication loss (242) based on the corresponding LoT pseudo-flag (122) and the corresponding LoT indicator (132); and The pre-trained transformer model (300) is fine-tuned based on the LoT indication loss (242) determined for the one or more word sequences (104).

2. The computer-implemented method (1300) of claim 1, wherein: The operations further include: For each corresponding word sequence (104): generating a corresponding classification pseudo-tag (124) using the regular expression rule, the corresponding classification pseudo-tag indicating the classification of the corresponding word sequence (104); generating a corresponding LoT classification (142) using the pre-trained transformer model (300), wherein the corresponding LoT classification predicts the classification of the corresponding word sequence (104); and determining a corresponding LoT classification loss (244) based on the corresponding classification pseudo-label (124) and the corresponding LoT classification (142); and The pre-trained transformer model (300) is fine-tuned based on the LoT classification loss 244 determined for the one or more word sequences (104).

3. The computer-implemented method (1300) of claim 2, wherein: The classification indication of the corresponding word sequence (104) is from a particular LoT step in a series of LoT steps associated with the corresponding word sequence (104).

4. The computer-implemented method (1300) of any one of claims 1-3, wherein: The input data (102) includes a plurality of clinical trial documents, and the plurality of clinical trial documents includes the unstructured text.

5. The computer-implemented method (1300) of any one of claims 1-4, wherein: The operations further include updating the regular expression rules based on the corresponding LoT-indicated loss (242).

6. The computer-implemented method (1300) of any one of claims 1-5, wherein: The pre-trained transformer model (300) includes a Bidirectional Encoder from Transformers for Biomedical Text Mining (BioBERT) model pre-trained on a corpus (212) of biomedical text data.

7. The computer-implemented method (1300) of claim 6, wherein: The pre-trained BioBERT model consists of a stack of multi-head self-attention layers.

8. The computer-implemented method (1300) of any one of claims 1-7, wherein: The operation further includes fine-tuning the pre-trained transformer model (300) using an aggregation of training (152) data, wherein the aggregation of training data includes manually annotated tokens, tokens generated using the regular expression rules, and tokens generated by the pre-trained transformer model (300).

9. The computer-implemented method (1300) of any one of claims 1-8, wherein: The operations further include storing the fine-tuned transformer model (300) in memory hardware (720) in communication with the data processing hardware (710).

10. The computer-implemented method (1300) of any one of claims 1-9, wherein: The operations further include transmitting the fine-tuned transformer model (300) to one or more computing devices via a network (1208).

11. A system (100) comprising: Data processing hardware; as well as Memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations comprising: receiving input data (102), the input data comprising unstructured text representing one or more word sequences (104); For each corresponding word sequence (104): generating a corresponding line of treatment (LoT) pseudo-tag (122) using a regular expression rule, wherein the corresponding line of treatment pseudo-tag indicates whether the corresponding word sequence (104) includes LoT information; generating a corresponding LoT indicator (132) using a pre-trained transformer model (300), wherein the corresponding LoT indicator predicts whether the corresponding word sequence (104) includes LoT information; and determining a corresponding LoT indication loss (242) based on the corresponding LoT pseudo-flag (122) and the corresponding LoT indicator (132); and The pre-trained transformer model (300) is fine-tuned based on the LoT indication loss (242) determined for the one or more word sequences (104).

12. The system (100) according to claim 11, wherein The operations further include: For each corresponding word sequence (104): generating a corresponding classification pseudo-tag (124) using the regular expression rule, the corresponding classification pseudo-tag indicating the classification of the corresponding word sequence (104); generating a corresponding LoT classification (142) using the pre-trained transformer model (300), wherein the corresponding LoT classification predicts the classification of the corresponding word sequence (104); and determining a corresponding LoT classification loss (244) based on the corresponding classification pseudo-label (124) and the corresponding LoT classification (142); and The pre-trained transformer model (300) is fine-tuned based on the LoT classification loss 244 determined for the one or more word sequences (104).

13. The system (100) according to claim 12, wherein The classification indication of the corresponding word sequence (104) is from a particular LoT step in a series of LoT steps associated with the corresponding word sequence (104).

14. The system (100) according to any one of claims 11 to 13, wherein: The input data (102) includes a plurality of clinical trial documents, and the plurality of clinical trial documents includes the unstructured text.

15. The system (100) according to any one of claims 11 to 14, wherein The operations further include updating the regular expression rules based on the corresponding LoT-indicated loss (242).

16. The system (100) according to any one of claims 11 to 15, wherein The pre-trained transformer model (300) includes a Bidirectional Encoder from Transformers for Biomedical Text Mining (BioBERT) model pre-trained on a corpus (212) of biomedical text data.

17. The system (100) according to claim 16, wherein The pre-trained BioBERT model consists of a stack of multi-head self-attention layers.

18. The system (100) according to any one of claims 11 to 17, wherein The operation further includes fine-tuning the pre-trained transformer model (300) using an aggregation of training (152) data, wherein the aggregation of training data includes manually annotated tokens, tokens generated using the regular expression rules, and tokens generated by the pre-trained transformer model (300).

19. The system (100) according to any one of claims 11 to 18, wherein The operations further include storing the fine-tuned transformer model (300) in memory hardware (720) in communication with the data processing hardware (710).

20. The system (100) according to any one of claims 11 to 19, wherein The operations further include transmitting the fine-tuned transformer model (300) to one or more computing devices via a network (1208).

Images