Generating structured data using machine learning

Abstract

Techniques for generating structured data for progress tracking using machine learning (ML) are disclosed. In an example method, a computing system receives a document including a number of pages including unstructured data or an event table. The computing system inputs, to an LLM, a prompt and the pages, in which the prompt includes instructions for identifying event tables. The computing system receives, from the LLM, a subset of pages including the event tables. The computing system identifies, for each event table, one or more event columns. The computing system inputs, to the LLM, a second prompt and the event columns, the second prompt including instructions for determining information about one or more clinical trial events from the event columns. The computing system receives, from the LLM, the information and outputs structured schedule information generated by mapping the information to a representation of a clinical trials schedule.

Description

GENERATING STRUCTURED DATA USING MACHINE LEARNINGCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to provisional application U.S. Ser. No. 63 / 559,631 entitled “Generating Structured Data Using Machine Learning” and filed on February 29, 2024, the entire disclosure of which is incorporated herein by reference for any purpose.FIELD

[0002] The present application generally relates to tabular information extraction using LLMs and more particularly relates to generating structured data for progress tracking using machine learning (ML).BACKGROUND

[0003] For some applications, data may only be available in formats that are not easily accessible. For example, the data required for an application may be contained in documents such as electronic documents formatted as Portable Document Format (PDF) files. PDF documents may include formatting, such as tables, that is not readily convertible to machine-readable, structured data formats suitable for analysis or other processing.

[0004] Particularly for large amounts of data, it may be desirable to automate the extraction of data from formatted documents such as PDFs. In some cases, however, the tables may be formatted in a non-standardized manner. This may present challenges for data extraction, especially when data accuracy and consistency are paramount.SUMMARY

[0005] Various examples are described for generating structured data for progress tracking using ML. In one example method, a computing system receives a document including a set of pages, in which first individual pages of the set of pages include at least one of unstructured data or an event table. The computing system inputs, to a large language model (LLM), a first prompt and the set of pages, in which the first prompt includes first instructions for identifying one or more eventtables. The computing system receives, from the LLM, a subset of pages of the set of pages, the subset of pages including the one or more event tables and second individual pages of the subset of pages including at least a portion of a first event table. The computing system identifies, for each event table of the one or more event tables, one or more event columns. The computing system inputs, to the LLM, a second prompt and the one or more event columns, in which the second prompt includes second instructions for determining information about one or more clinical trial events based on the one or more event columns. The computing system receives, from the LLM, the information about the one or more clinical trial events. The computing system generates structured schedule information including mapping the information about the one or more clinical trial events to a representation of a clinical trials schedule. The computing system outputs the structured schedule information based on the representation of the clinical trials schedule to cause an action associated with a clinical trial event.

[0006] In another example method, a computing system accesses unlabeled training data, including a set of pages including a first portion of pages with at least one event table on first individual pages and a second portion of pages without event tables on second individual pages. The computing system determines a label for each page of the set of pages. The computing system inputs, to an LLM, a prompt and the set of pages, in which the prompt includes instructions for identifying one or more event tables on the first individual pages, including a first instruction to, for an individual page of the set of pages, perform a first binary classification of the page corresponding to a first presence of at least a portion of a table on the individual page. The computing system receives, from the LLM, an identification of a subset of pages of the set of pages, the subset of pages including the one or more event tables and third individual pages of the subset of pages including the at least the portion of the table. The computing system compares the identification of the subset of pages of the set of pages with the label for each page of the set of pages. The computing system modifies the LLM, based on the comparison, using a fine-tuning technique.

[0007] In some embodiments, an apparatus is provided, which includes means for implementing part or all of the operations and / or methods disclosed herein.

[0008] In some embodiments, a computer program product is provided, which includes computer instructions that, when executed by a processor, implement part or all of the operations and / or methods disclosed herein.

[0009] These illustrative examples are mentioned not to limit or define the scope of this disclosure, but rather to provide examples to aid understanding thereof. Illustrative examples are discussed in the Detailed Description, which provides further description. Advantages offered by various examples may be further understood by examining this specification.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate one or more certain examples and, together with the description of the example, serve to explain the principles and implementations of the certain examples.

[0011] FIG. 1 shows an example system for generating structured data for progress tracking using ML, according to some aspects of the present disclosure.

[0012] FIG. 2 shows a flowchart of an example method for generating structured data for progress tracking using ML, according to some aspects of the present disclosure.

[0013] FIG. 3 shows an example Schedule of Events (SoE) table, according to some aspects of the present disclosure.

[0014] FIG. 4 shows an example of SoE footnotes, according to some aspects of the present disclosure.

[0015] FIG. 5 shows an example of a representation of a clinical trials schedule, according to some aspects of the present disclosure.

[0016] FIG. 6 shows an example of a prompt that may be input to an LLM used to implement some aspects of generating structured data for progress tracking using ML, according to some aspects of the present disclosure.

[0017] FIG. 7 shows an example method for generating structured data for progress tracking using ML, according to some aspects of the present disclosure.

[0018] FIG. 8 shows another example method for fine-tuning LLMs used for generating structured data for progress tracking using ML, according to some aspects of the present disclosure.

[0019] FIG. 9 shows an example computing device suitable for use in example systems or methods for generating structured data for progress tracking using ML, according to some aspects of the present disclosure.

[0020] FIG. 10 shows an example of an SoE table during a verification process, according to some aspects of the present disclosure.

[0021] FIG. 11 shows an example of a user interface that may be used during a verification process, according to some aspects of the present disclosure.DETAILED DESCRIPTION

[0022] Examples are described herein in the context of generating structured data for progress tracking using ML. Those of ordinary skill in the art will realize that the following description is illustrative only and is not intended to be in any way limiting. For example, the examples described herein may be used to generate structured data for applications outside of progress tracking and may be performed using other forms of machine learning models. Reference will now be made in detail to implementations of examples as illustrated in the accompanying drawings. The same reference indicators will be used throughout the drawings and the following description to refer to the same or like items.

[0023] In the interest of clarity, not all of the routine features of the examples described herein are shown and described. It will, of course, be appreciated that in the development of any such actual implementation, numerous implementationspecific decisions must be made in order to achieve the developer’s specific goals, such as compliance with application- and business-related constraints, and that these specific goals will vary from one implementation to another and from one developer to another.

[0024] Automated data extraction from structured documents is an important of the modern “big data” landscape, particularly when large volumes of documents are involved. Automated data extraction can be performed using methods such as optical character recognition (OCR) to convert scanned documents into editable text, natural language processing (NLP) for understanding and extracting relevant information from text, rule-based extraction where specific rules or algorithms are defined to locate and extract data based on its structure and content (e.g., regularexpressions), or machine learning (ML) algorithms to identify and classify data patterns in input documents.

[0025] For example, one approach to automated data extraction involves ML models such as LLMs. LLMs have been found to be useful for a diversity of task types. In particular, LLMs are useful for tasks involving natural language understanding and generation, question-answering, summarization, programming, and even creativity. LLMs are therefore increasingly being explored for their potential applications relating to data extraction from structured data formats in various contexts such as healthcare.

[0026] Identifying and extracting structured data from documents such as PDFs can be a challenging problem to solve in general, particularly when the structured data is in a non-standardized format. Rule-based or algorithmic approaches may fail due to unpredictable variations among input documents. Likewise, traditional approaches to NLP involving, for example, feature-based ML models may be limited to the realm of their available training data that cannot anticipate every possible variation in the input documents.

[0027] The clinical trial context is a representative example. The traditional conduct of progress tracking during clinical trials is fraught with inefficiencies at various stages, in part due to the inefficient communication of scheduling data. For example, tables in a clinical trial scheduling document, that is part of a clinical trial protocol, may outline the plan of care for participants, detailing visits for screening, treatment, and follow-up phases, and so on. Accurate identification and parsing of these tables from clinical trial scheduling documents, often consisting of hundreds of pages, as well as several different kinds of tables, poses a key challenge in digitization of clinical trial protocols.

[0028] For example, clinical trial scheduling data is often published in a tabular format referred to as a Schedule of Events (“SoE”) or Schedule of Activities (“SoA”) (hereinafter, collectively “SoE”). SoE tables may include, for example, descriptions of specific procedures or assessments (referred to generally herein as “events”) to be performed and columns for indicating the timing of each event relative to various touchpoints. While SoE tables may include similar information from document to document, the particular format of the table may vary between0documents and may in fact be unique for a given document. For instance, the SoE tables can have variability in formatting, terminology, and layout across a diverse set of clinical trial protocols.

[0029] As a result, existing data extraction procedures for such documents may involve automated components such as the examples mentioned above but must also generally rely on or be augmented by manual work performed by human extractors, annotators, verifiers, etc., to attain the desired accuracy. The combination of inaccurate automated methods and the need for human involvement significantly slow down the extraction of clinical trials scheduling data from structured documents and are accompanied by explosive costs relating to the required human labor. Accurate and efficient digitization of clinical trials scheduling information such as SoEs is necessary for progress tracking needed to generate structured schedule information that can be used to improve processes relating to recruitment, follow-ups, data acquisition, and data handling.

[0030] These challenges can be addressed using techniques for generating structured data for progress tracking using ML. For example, steps such as SoE table identification, parsing, and interpretation can be modeled as steps to be performed by an LLM given an appropriate prompt. Because some steps performed by LLMs may be accompanied by inaccuracies in extraction or semantic understanding, human feedback can be incorporated to improve accuracy. The human feedback can also be used to “fine tune” the LLMs to promote further improvements in accuracy and efficiency. The information thus extracted and, in some cases, annotated or verified, can be used along with business logic and domain expertise to generate structured schedule information for use in various applications.

[0031] In an example method, a computing system for data extraction receives a multi-page document. The document may be a clinical trials scheduling document including one or more SoE tables along with various other structured or nonstructured data (e.g., text, other tables, lists, illustrations, graphs, etc.). In some cases, the SoE tables may straddle multiple pages, beginning on a first page and ending on a later page. The SoE tables may be augmented by a number of footnotesor other annotations specifying modifications or special considerations for certain aspects of the SoE table data.

[0032] The multi-page document can be converted into a suitable machine- readable format and input to an LLM along with a prompt including instructions for the LLM to identify event tables such as SoEs. The LLM then outputs a subset of pages from the document that were identified as containing an event table or a portion thereof. In some examples, a verification procedure using, for example, nonexpert human annotators can be used to verify the results at this step as well as to generate feedback to fine-tune the LLM for improving future identifications.

[0033] The computing system then identifies, within each of the identified event tables, one or more event columns using an algorithmic approach or using the LLM. The event columns can include information sufficient to define one or more clinical trial events which is typically organized beneath one or more column headers of an event table.

[0034] The computing system inputs the identified event columns and a second prompt to the LLM including instructions for determining information about one or more clinical trial events based the event columns. This may involve, for example, an “expansion” procedure in which the collection of information in an event column is converted into information one or more clinical trial events.

[0035] The LLM then outputs the information about the one or more clinical trial events. For example, the LLM may output a list or data structure including all of the identified clinical trial events and details thereof. In some examples, a verification procedure involving, for example, non-expert human annotators can again be used to verify the results at this step as well as to generate feedback to further fine-tune the LLM.

[0036] The computing system then generates and outputs structured schedule information used for progress tracking by mapping the clinical trial event information to a representation of a clinical trials schedule, such as a calendar or a representation used in clinical trials management software. The structured schedule information can then be used to perform various applications relating to clinical trials such as managing patient schedules, identifying drug interactions, patient monitoring, improving clinical trial design, and so on.

[0037] The techniques disclosed herein constitute significant improvements to and thereby provide an advantageous technical effect by improving the technical field of tabular information extraction using LLMs. Existing techniques utilizing ML technologies may be limited in the accuracy they can achieve. For example, as described above, existing approaches involving ML technologies may have an accuracy limited by the scope of the available training data. Rule- or algorithmbased approaches may be similarly limited due to the lack of standardization among certain classes of documents. The combination of at least customized LLM prompts tailored to the expected semantic content of non-standardized tabular data in concert with human feedback can result in previously unattainable levels of accuracy, improve the efficiency and speed of information extraction, and reduce cost.

[0038] The techniques disclosed herein likewise may result in reduced consumption of computing resources and overall improvements to the functioning of computing systems. For example, existing systems, such as rule- or algorithm-based systems, can involve a significant amount of document processing, including processing of documents in their original, unconverted form. In contrast, the techniques of the present disclosure can, in some examples, involve the extraction of text as a pre-processing step prior to processing, thus reducing the need for processor- and memory-intensive processing of large documents full of graphics or tables. Along the same lines, the use of pre-trained, fine-tuned LLMs to perform tasks based on text and text prompts may consume fewer computational resources than the repetitive processing of documents using rules or the ongoing re-training of ML models to account for previously unseen scenarios.

[0039] This illustrative example is given to introduce the reader to the general subject matter discussed herein and the disclosure is not limited to this example. The following sections describe various additional non-limiting examples illustrating techniques for generating structured data for progress tracking using ML.

[0040] Referring now to FIG. 1, FIG. 1 shows an example system 100 for generating structured data for progress tracking using ML, according to some aspects of the present disclosure. The components of system 100 may beimplemented as hardware or software components of one or more computing systems. The computing system(s) can include locally accessible hardware, remote servers, cloud-based computing services, or any combination thereof. For instance, the document pre-processing component 110 may be implemented as program code executing on a laptop, desktop, tablet, etc. while the ML models 115, 135 may be third-party program code executing on remote, cloud -based servers. In another example, the ML models 115, 135 may be accessible using a web-based application programming interface (API) in such a way that their implementation details are transparent to the client components of system 100.

[0041] Example system 100 includes a document pre-processing component 110 that receives as input a clinical trials scheduling document 105. The clinical trials scheduling document 105 may be a multi-page structured document, such as a PDF file that includes information relating to one or more clinical trials. For example, the clinical trials scheduling document 105 may include clinical trial timeline and scheduling information, eligibility criteria, treatment protocols, and so on.

[0042] The clinical trial timeline and scheduling information may include, for example, a tabular SoE 106. An SoE 106 can specify the sequence and timing of various activities and assessments to be conducted throughout the clinical trial. For example, an SoE 106 may include listings of all clinical procedures, laboratory tests, and assessments scheduled to occur at various time points throughout a clinical trial.

[0043] The SoE 106 can include significant levels of detail. For instance, an SoE 106 may specify the exact timing of some event, such as initial screenings, treatment administrations, follow-up visits, the frequency and duration of each scheduled activity, and final assessments, often in relation to the participant's entry into the study. SoEs can further include conditional statements, caveats, or modifications based on, for example, criteria for participant eligibility, required diagnostic tests, or other criteria. Such modifications may be included in the SoE 106 in the form of footnotes, endnotes, inline notes, and so forth.

[0044] The description herein relating to a clinical trials scheduling document 105 is intended as a non-limiting example. For instance, in some examples, thesystem 100 can be configured to perform data extraction on other kinds of structured, semi-structured, or unstructured documents such as invoices, HTML pages, or news articles.

[0045] The document pre-processing component 110 can convert the input clinical trials scheduling document 105 to a format that is suitable for processing by the ML model 115. For example, the document pre-processing component 110 can convert the input clinical trials scheduling document 105 in the PDF format (sometimes referred to as a “raw” PDF) to a suitable combination of structured and unstructured, machine-readable data.

[0046] The input clinical trials scheduling document 105 may be a composite format including at least a structured data portion and a textual portion. For instance, where the input clinical trials scheduling document 105 includes one or more tables (e.g., SoEs) the tables can be converted into a structured or semistructured format such as the JavaScript Object Notation (JSON) format or the extensible Markup Language (XML) format. The document pre-processing component 110 can employ a software library for PDF table extraction that can extract tabular data from the images of tables in PDFs and convert the extracted data into programmatic data structures such as multidimensional arrays or other formats such as JSON or XML. The unstructured data in the clinical trials scheduling document 105 can likewise be converted into plain text.

[0047] The pre-processed clinical trials scheduling document 105 is input to a ML model 115. The ML model 115 may be, for example, an LLM. LLMs can include ML models that are trained to understand, generate, and translate human language based on large amounts of text training data. Other types of ML models that can be used for or in concert with ML model 115 may include, and are not limited to, decision trees, support vector machines (SVMs), neural networks, convolutional neural networks (CNNs), recurrent neural networks (RNNs), random forests, gradient boosting machines (GBMs), k-nearest neighbors algorithms (KNN), naive Bayes classifiers, autoencoders, transformers, adversarial networks, reinforcement learning models, among others. ML model 115 may be an individual, standalone ML model 115, it may be an ensemble ML model 115 including multiple ML model types, or it may be multiple ML models 115 operating in parallel.

[0048] The ML model 115 outputs a subset of input pages identified as containing event tables 120. For example, the ML model 115 output may be a list or other data structure indicating document pages identified as including an event table or a portion thereof. In some examples, the event tables 120 may be output along with their containing pages, as well as some surrounding context. For instance, pages identified as containing at least an event table portion may be output along with one or more pages immediately before or after (which may not be identified as containing an event table portion). In some examples, human verification or other verification procedures can be used to ensure the accuracy of the ML model 115 and to provide feedback to the ML model 115 to further train or fine-tune the ML model 115.

[0049] The event tables 120 are input to a column identification component 125 configured to identify, for each event table of the event tables 120, one or more event columns 130. An event column, in the context of an event table such as an SoE, can refer generally to a collection of information that is sufficient to define a clinical trial event. For example, a clinical trial event may correspond to an event such as chemotherapy administration, radiation therapy sessions, blood sample collections for laboratory tests, imaging studies (e.g., MRI or CT scans), or physical examinations. The collection of information needed to specify when, where, what, how, and other related details are included in an event column. An event column may thus include information obtained from one or more columns, one or more rows, and one or more cells, as well as additional descriptive information such as headers and footnotes. The event columns 130 may be identified by the column identification component 125 using, for example, a regular expression or other rule- or algorithm-based approach.

[0050] The event columns 130 identified by the column identification component 125 can be input to a second ML model 135. The ML model 135 may be, for example, an LLM (which may be the same LLM as the one used by ML model 115). In some examples, the second ML model 135 can be another kind of ML model 135, such as one of the other types listed above with respect to ML model 115. The second ML model 135 is configured or trained to identify clinical trial event information 140 from the event columns 130. The clinical trial event information140 may be. for example, an output or data structure that includes individual or groups of clinical trial events determined based on the event columns 130. In some examples, the clinical trial event information 140 can include information about extracted footnotes or may be modified by information obtained from extracted footnotes.

[0051] In some examples, human verification or other verification procedures can be used to ensure the accuracy of the events determined by ML model 135 and to provide feedback to the ML model 135 to further train or fine-tune the ML model 135.

[0052] The clinical trial event information 140 can be used to generate structured schedule information 150 by an event mapping component 145. The event mapping component 145 can, for example, map the clinical trial event information 140 to a representation of a clinical trials schedule 151. An example of such a representation 151 is sketched in FIG. 1 and discussed in detail below with respect to FIG. 5. The representation of a clinical trials schedule 151 can be, for example, a data structure or user interface that provides the extracted information in a manner that can be usefully extended to other applications (e.g., as part of a clinical trial management software user interface).

[0053] In some examples, the system 100 can output the structured schedule information 150 to cause an action associated with a clinical trial event. For examples, the output the structured schedule information 150 could be used for optimizing treatment schedules, identifying potential drug interactions through appropriate cross-referencing procedures (e.g., with other clinical trials or courses of treatment), enhancing patient monitoring, improving clinical trial design, or determining certain health interventions.

[0054] Turning now to FIG. 2, FIG. 2 shows a flowchart of an example method 200 for generating structured data for progress tracking using ML, according to some aspects of the present disclosure. FIG. 2 may correspond to a particular embodiment of the example system 100 shown and described with respect to FIG. 1.

[0055] At block 205, a computing system receives a raw PDF. A raw PDF can include PDF files containing the original content, layout, and formatting as intended by the generating entity. The raw PDF can include all elements such astext, images, and vector graphics, without any compression, optimization, or extraction applied to the data. The raw PDF may be a clinical trials scheduling document 105 including one or more SoEs, in which each SoE may extend over one or more pages. While the example of FIG. 2 involves processing of a clinical trials scheduling document 105, it should be emphasized that the techniques of the present disclosure can be applied to a variety of document types and input formats.

[0056] Likewise, although the example of a PDF is used here, other similar structured document formats could be used, such as PostScript (PS) files, Encapsulated PostScript (EPS) files, Adobe Illustrator (Al) files, and so on. Likewise, structured documents such as XML files, Comma-Separated Values (CSV) files, and Microsoft Excel spreadsheets, and the like could also be used in some examples.

[0057] At block 210, the computing system performs SoE detection on the input clinical trials scheduling document 105 using an LLM. The LLM can include components for understanding, interpreting, and generating human language based on certain training processes involving very large training datasets. The LLM may be, for example, built on a transformer architecture. Training an LLM can involve unsupervised learning on extensive corpuses of text. Training may be followed by or augmented by fine-tuning for, for example, specific applications or tasks. In addition to LLMs, some or all functions of ML model 115 can be performed by or in combination with other approaches such as rule-based systems, regular expressions, named entity recognition systems, keyword extraction algorithms, or OCR techniques.

[0058] In some examples, the LLM can be a third-party propriety LLM accessible using a web-based API. In some other examples, the LLM can be a locally deployed and trained LLM based on a publicly available, open-source LLM. Popular examples of commercially available or open-source LLMs include the GPT (Generative Pre-trained Transformer) series by OpenAI, Gemini / Bard by Google, PaLM 2 by Google, Claude 2 by Anthropic, or Llama 2 by Meta.

[0059] The pre-processed clinical trials scheduling document 105 can be accompanied by a prompt for the LLM. The prompt can include text provided to the LLM to initiate or guide its generation of text or responses. The prompt mayinclude, for example, plain language, detailed instructions that define the task and desired response for the LLM with respect to the input pre-processed clinical trials scheduling document 105. The prompt may, for example, include instructions for identifying one or more event tables in the clinical trials scheduling document 105. An example of a prompt that may be used in some cases is shown in FIG. 6.

[0060] The prompt may direct the LLM to perform one or more classification tasks based on the pre-processed clinical trials scheduling document 105. For example, one binary classification task may involve identification of a table such as an SoE. In this example, the output of the LLM may be, for each page, a binary classification such as “Yes, this page contains a table” or “No, this page does not contain a table.” The output of the LLM can be configured, via the prompt, to identify portions of tables in addition to entire tables (e.g., tables that fit entirely on one page).

[0061] In some examples, the LLM can be configured, via the prompt, to conservatively identify pages as including a portion of a table. The LLM may be thus configured to minimize false negatives in favor of false positives. This is because, as will be further discussed below, additional annotation or verification steps may be included in the system 100 to screen out false positives. But false negative errors may not be recoverable without significant manual verification, which is one challenge the innovations of this disclosure are intended to mitigate.

[0062] Another example binary classification task may involve identification of the presence of footnotes associated with a table. For example, an SoE in a clinical trials scheduling document 105 is often followed by a number of footnotes that contain additional information to modify or otherwise caveat some of the information in the SoE for specific situations. In this example, the output of the LLM may be, for each page, a binary classification such as “Yes, the table ending of this page is followed by one or more footnotes” or “No, the table ending of this page is not followed by any footnotes.” The binary classifications just described can be used to determine a subset of input pages of the clinical trials scheduling document 105 containing event tables 120 or a portion thereof.

[0063] At block 215, a verification procedure is applied to the output of block 210. For example, the subset of input pages can be provided to a verificationsubsystem. The verification subsystem can determine the accuracy of the binary classifications made in block 210. In some examples, the verifications subsystem may involve the use of human annotators. The human annotators may be, for example, non-expert human annotators with instructions to perform certain verification tasks.

[0064] The non-expert human annotators may be chosen according to certain criteria. For example, only non-expert human annotators may be selected who have previously participated in annotation for creating an LLM fine-tuning dataset. In some examples, non-expert human annotators may have access to expert digitizers for any questions while performing verifications or annotations. Thus, in this example, the non-expert human annotators may have experience in annotation but are not experts in digitization of the clinical trial protocols or scheduling documents.

[0065] For example, the non-expert human annotators can be provided the identified subset of input pages or a portion thereof using a suitable user interface. The non-expert human annotators can be tasked, by the user interface, with making determinations such as whether a particular page contains an SoE or contains footnotes. The tasks can be posed as questions to the non-expert human annotators. For example, a non-expert human annotator can be shown a particular page and then answer a question such as “Does this page contain an SoE?” using the user interface. In accurate identifications can be corrected before being input to another processing step.

[0066] In some examples, the verifications performed by the human annotators can be used for, in addition to elimination of false positives, generating a feedback loop to improve the output of the LLM. Dotted line 217 depicts the feeding back of the results of the verification procedure of block 215 to the LLM used in block 210. For example, the instances in which human annotators determine that the LLM incorrectly identified a page as containing an SoE portion or incorrectly failed to identify a page as containing an SoE portion can be converted into naturallanguage descriptions of the error and fed back to the LLM to fine-tune the LLM or otherwise modify the LLM to improve its accuracy for future classifications. The use of the verification subsystem, including the use of non-expert human annotators can improve the accuracy and / or efficiency of the system 100 by up to several times.

[0067] At block 220, the computing system performs several table processing operations on the SoE(s) identified and verified in blocks 210 and 215. While the structure of the event tables 120 does not follow any standard, some generalizations can be made. The collection of information may include data extracted from several locations within a given event table. The data thus extracted may be generally organized in a columnar manner. For example, the collection of information defining a clinical trial event may include information from a row header, a column header, a row specifying an activity or other event description, or optional instructions associated with a row. In some examples, the collection of information defining a clinical trial event may be further refined, modified, or narrowed with footnotes that appear before, after, or near the event table.

[0068] At block 220, a column identification component 125 receives the pages with event tables 120 previously identified and verified and then output one or more event columns 130. For example, the column identification component 125 can identify, from the event tables 120, one or more event columns 130 based on a segmentation schema. The segmentation schema may correspond to the tabular layout of the collection of information, as described in the example above. FIG. 4 shows an example of an SoE illustrating one such segmentation schema. The column identification component 125 can also map footnotes associated with the event tables 120 to the corresponding event columns 130.

[0069] The application of the segmentation schema as well as the mapping of the footnotes by the column identification component 125 may be performed using a combination of rules and algorithms configured to identify the event columns 130. For example, regular expressions, string match algorithms, keyword searches, software libraries including specialize parsing functions, or text segmentation and tokenization may be used to identify information defining event columns 130. In some examples, some or all functions of the column identification component 125 may be additionally by performed by the LLM.

[0070] The event columns 130 thus identified can be input to an LLM (which can be the same or different LLM as the one used in block 210) along with a prompt. The prompt can, for example, include instructions for extracting information about one or more clinical trial events based on the input event columns 130. For example,the instructions may include steps for extracting information such as study or activity period, study or activity length, cycle length or duration, the number of cycles to conduct, window information, and so on.

[0071] The instructions may specify processes for “expanding” the input event columns 130. Expanding of an event column can refer generally to determining a periodicity of a clinical trial event or events defined by the event column and generating one or more clinical trial events for each event in the periodic series. For example, if an event column includes a specification of a physical exam that should take place once a week for 10 weeks, then 10 associated clinical trial events may be identified.

[0072] Expansions of clinical trial events can be finite or infinite. Finite expansions include clinical trial events with a well-defined, finite number of occurrences. The number of occurrences may be conditional, but the series of clinical trial events may terminate under well-defined circumstances. For example, a finite series of clinical trial events may include a clinical event, such as a physical exam, scheduled to occur 10 times. In another example, the clinical event may be scheduled to occur 10 times and then to continue to recur “as needed.” Such an expansion may be identified as infinite.

[0073] In some examples, the LLM can be used in concert with rule-based or algorithmic extraction to maximize accuracy. For example, timing components can be extracted using them LLM while the expansion can be performed using a suitable algorithm. The algorithm may receive information or metadata determined by the LLM as input. For example, the LLM may identify and list events as well as associated information about cycles, specific days, number of visits, etc. The identified events, along with determinations about the expansion required (e.g., finite or infinite) can be passed into a suitable algorithm to transform the identified events into expanded clinical trial event information 140. In another example, the LLM can be used to identify collections of information about events and then the LLM can be used in concert with rule-based methods or algorithms to extract timing information.

[0074] At block 225, the computing system classifies the expansions of the clinical events as simple or complex using an LLM. In some examples, during theexpansion process of block 220, the LLM may determine a confidence score based on a determination of the accuracy of the expansion. For example, a higher confidence score may correspond to greater estimated accuracy. Based on the determined confidence scores or other criteria, each expansion can be classified as simple or complex.

[0075] The classification as simple or complex can be used to determine whether addition verifications or annotations are required in, for example, block 230. For example, the LLM may classify an expansion as complex based on criteria such as low confidence in the accuracy of the expansion or identification of features that may benefit from downstream human annotation or review. In contrast, the LLM may classify an expansion as simple based on criteria such as a higher confidence in the accuracy of the expansion or a determination that the expansion is less likely to contain any errors.

[0076] At block 230, a verification procedure such as using human annotators is applied to the output of block 225. In some examples, the verification procedure is only applied to clinical trial event expansions identified as complex. Non-expert human verification or annotation can be used to, for example, verify clinical trial event expansions as accurate, correct inaccurate clinical trial event expansions, or reject clinical trial event expansions as non-processable. Verifications may include verifications of details such as window information, cycle length (where applicable), study period, number of visits, cycle number, timing number, and so on.

[0077] Dotted line 232 depicts the feeding back of the results of the verification procedure of block 230 to the LLM used in block 220. For example, for expansions identified as of complex scope, human annotators can identify the expansion as correctly or incorrectly performed. The human annotators may, for example, determine that the expansion was based on misconstrued timing information or failed to take into account an exceptional circumstance not yet encountered by the LLM or anticipated by algorithmic approaches.

[0078] In such cases, the expansion can be revised and described using a natural-language description of the error. The description can be fed back to the LLM to fine-tune the LLM or otherwise modify the LLM to improve its accuracy for performing future expansions. In some examples, the human annotators can revisethe expansion and proceed with processing in block 235 without providing feedback to the LLM to expedite processing.

[0079] At block 235, the computing system performs post-processing of the output of block 230 and outputs clinical trial event information 140 that can be used to generate structured schedule information 150 by, for example, the event mapping component 145. Post-processing may include, for example, parsing by the LLM of any additional event table details such as footnotes or instructions that may override a clinical trial event determined based solely on the corresponding event column. In some examples, these or other post-processing steps may be performed using rule- or algorithm-based approaches.

[0080] Turning now to FIG. 3, FIG. 3 shows an example SoE 300 table, according to some aspects of the present disclosure. The example SoE 300 shows an example arrangement of columns, rows, information, footnotes, etc. It should be emphasized, however, that this is merely an example intended to illustrate some examples of information that may be extracted by some embodiments of the present disclosure. Because there is, in general, no standardization used by the drafters of SoEs and similar documents, this example is also intended to highlight the complexity of the information extraction problem and the challenges overcome by the techniques of the present disclosure.

[0081] The example SoE 300 includes a number of columns. The columns can be used to indicate a correspondence of a particular timing or scheduling aspect of the trial with one or more clinical trials elements 355 listed in the rows. For example, a safety assessment requirement 302 (e.g., a skin biopsy) may be required a several times during a given clinical trial protocol, which can be indicated with “X’ marks 360 in the certain corresponding columns.

[0082] The example SoE 300 includes a study period column 305. The procedure column 305 may correspond to rows for, for example, specific phases, durations, or dates of clinical trial activities for clinical trial participants. A screening column 310 may correspond to rows for, for example, include criteria or procedures for determining participant eligibility for the clinical trial. Treatment columns 315 may correspond to rows for specific therapeutic interventions, medications, or procedures participants will receive during the clinical trial. Somecolumns may have one or more sub-columns 335 that correspond to a row, such as the study day row 340, described below. An end-of-study column 320 may correspond to rows for visits and assessments for monitoring participants' responses following treatment or procedures or for ongoing monitoring of participants to assess long-term treatment effects and safety.

[0083] The example columns of SoE 300 above are arrayed against various rows, the rows listing the various clinical trials elements 355 which may be grouped together. For example, the example SoE 300 includes a study day row 340. The study day row 340 precedes elements 355 that may include timing information (e.g., periodicity, duration, number, etc.) for one or more of the elements represented in the columns. As another example, the safety assessments row 345 precedes elements 355 that may include particular assessments, treatments, and protocols that can be performed at various stages of the clinical trial relating to protocol safety as may be shown using a correspondence with a column.

[0084] Example SoE 300 illustrates one possible segmentation schema. For example, identification of one or more event columns prior to processing the information therein can be based on a segmentation schema associated with an SoE table. In this example, the segmentation schema may include, for example, the header row 304, a column header (e.g., screening column header 310), a row (e.g., requirement 302), one or more “X’ marks 360, and / or footnotes (shown in FIG. 4).

[0085] Turning now to FIG. 4, FIG. 4 shows an example of SoE footnotes 400, according to some aspects of the present disclosure. In this example, the footnotes immediately follow an SoE 405 but other arrangements of footnotes are also possible. For example, footnotes could appear before or to the side of an SoE 405. Alternatively, footnotes 400 could appear on another page or in an appendix to the document.

[0086] The footnotes 400 include footnote listings 415. In the example footnote listings 415, a number of example footnotes are show, indicated using lowercase letters of the alphabet. However, the footnote listings 415 may use numbers, upper case letters, bullet points, and so on.

[0087] These examples are included to illustrate the utility of using an ML model such as an LLM to apply the variable semantic meaning of footnote listings400 to the information extracted from an SoE. For example, an LLM prompt may include instructions to determine a modification to an expanded clinical trial event based on one or more footnotes identified in associated with the SoE. Because the content of the footnotes is essentially unconstrained, the flexibility of an LLM can be used to ensure that their content is accurately applied to the extracted information.

[0088] Turning now to FIG. 5, FIG. 5 shows an example of a representation 500 of a clinical trials schedule, according to some aspects of the present disclosure. The example representation 500 may be displayed and utilized while using, for example, a clinical trial management system that includes features for planning, executing, and monitoring clinical trials. This may used as part of, for example, mapping information about clinical trial events to the representation 500 of a clinical trials schedule to obtain structured schedule information. The structured schedule information can be used in various clinical applications such as optimizing treatment schedules, identifying potential drug interactions through appropriate cross-referencing procedures (e.g., with other clinical trials or courses of treatment), enhancing patient monitoring, improving clinical trial design, determining certain health interventions, and so on. Representation 500 illustrates one possible way to use the information about clinical trial events to obtain structured schedule information that is intended to be non-limiting.

[0089] Representation 500 includes descriptive information 505 such as dates, schedule types, timing information, masking information, and so on. Other representations may should different or additional descriptive information 505. In the representation 500, a protocol graph 509 representation is selected by way of a mode selector switch 507. A protocol graph may be a visual representation or diagram of a clinical trial's timeline including indications of procedures, visits, and assessments, and so on, as defined in the study protocol.

[0090] Protocol graph 509 may include a number of nodes connected by edges, in analogy to a mathematical graph object. The nodes may represent clinical trial events while the edges may represent logical relations (e.g., temporal relations) between those events. For example, the protocol graph 509 may include a top level node 510 corresponding to the beginning of a clinical trial. Arrows 515 can indicatea directed relationship to subsequent nodes, indicating, for example, the passage of time and / or a prerequisite requirement for event(s) in preceding nodes to occur before advancing to subsequent nodes. The protocol graph 509 can include, for example, screening nodes 520. These nodes may correspond to screening events, similar to the screening events denoted in FIG. 3 in the screening column 310. The protocol graph may include one or more feedback arrows 525 indicating, for example, a requirement to repeat actions when certain criteria are fulfilled. The protocol graph 509 can also include, for example, clinical nodes 530 corresponding to procedures, assessments, treatments, and so on. Information about the symbols in the protocol graph 509 can be described in legend 535. It should be emphasized that the protocol graph 509 is merely an example of a representation 500. Representation 500 and the associated mapping from clinical trial event information can take other forms and be used in a variety of contexts.

[0091] Turning now to FIG. 6, FIG. 6 shows an example of a prompt 600 that may be input to implement some aspects of generating structured data for progress tracking using ML, according to some aspects of the present disclosure. An LLM can used, in some examples, to implement some or all functions of the ML model 115 and / or ML model 135. The example prompt 600 includes instructions 610. The instructions 610 can specify the desired output of the LLM as well as specifications on how the output should be formatted. The instructions 610 may further include guidelines on the length or verbosity of the LLM's output, dictate the use of certain language styles or tones, such as formal, informal, or technical, tailored to the particular application, or other constraints expressed as plain, natural language.

[0092] The example prompt 600 also includes supplemental instructions 620 that provide additional detail, context, and guidance for responding to the instructions 610. For example, the supplemental instructions 620 include descriptions of SoE tables, example terminology found in SoE tables, counterexamples, and so on. The supplemental instructions 620 may include any additional information pertinent for or relevant to complying with or otherwise responding to the instructions 610.

[0093] The example prompt 600 of FIG. 6 includes the example instructions 610 and example supplemental instructions 620 as described above, but these aremerely examples of prompt substructure or parts provided to introduce certain concepts. Prompts, as used in various examples of the present disclosure, can vary widely, and may include any language or other information (e.g., tabular data, images, audio, video, binary data, etc.) required to implement the desired function.

[0094] Referring now to FIG. 7, FIG. 7 shows an example method 700 for generating structured data for progress tracking using ML, according to some aspects of the present disclosure. The method 700 will be described with respect to the example system 100 shown in FIG. 1; however, any suitable system according to this disclosure may be similarly used.

[0095] It should be appreciated that method 700 provides a particular method for generating structured data for progress tracking using ML. Other sequences of operations may also be performed according to alternative examples. For example, alternative examples of the present disclosure may perform the steps outlined above in a different order. Moreover, the individual operations illustrated by method 700 may include multiple sub-operations that may be performed in various sequences as appropriate to the individual operation. Furthermore, additional operations may be added or removed depending on the particular applications. Further, the operations described in method 700 may be performed by different devices.

[0096] Method 700 may include block 710. At block 710, a computing system receives a document including a set of pages, in which first individual pages of the set of pages include at least one of unstructured data or an event table. For example, the document may be a clinical trials scheduling document 105 in the PDF format. Some subset of the pages of the document may include an event table such as an SoE or a portion thereof. The pages may likewise include a variety of unstructured data including text, graphics, tables, charts, metadata, and so on.

[0097] At block 720, the computing system inputs, to an LLM, a first prompt and the set of pages, in which the first prompt includes first instructions for identifying one or more event tables. The first instructions may include instructions to, for example, perform a binary classification of individual input pages corresponding to the presence or non-presence of an event table or a portion thereof.

[0098] In some examples, the instructions to perform a binary classification may include sub-instructions. For example, a first sub -instruction may involveidentifying a first event column of the individual event columns of the one or more event columns. A second sub -instruction may involve identifying a header of the first event column, the header including a header text string. A third subinstruction may involve determining a first clinical trial event using a regular expression applied to the header text string. For example, a regular expression can be designed to identify columns that include the word “treatment” alone or in combination with other words.

[0099] At block 730, the computing system receives, from the LLM, a subset of pages of the set of pages, the subset of pages including the one or more event tables in which the individual pages of the subset of pages each include at least a portion of a first event table. For example, the subset of pages may include pages with a complete table, pages with a portion of a table, and some pages inaccurately flagged that contain no table or any portion thereof. In the latter case, in some examples, a verification subsystem can be used to review and identify such cases. For example, the verification system may involve non-expert human annotators to ensure the accuracy of the pages flagged by the LLM or to provide feedback to LLM.

[0100] At block 740, the computing system identifies, for each event table of the one or more event tables, one or more event columns. For example, a column identification component 125 can identify, from the event tables identified in block 720 and verified in block 730, one or more event columns 130 based on a segmentation schema. The segmentation schema may correspond to a tabular layout of a collection of information that defines a clinical event or group of events. A segmentation schema may include, for example, a header row and text therein, a column header and the text therein, activities, optional instructions, optional footnotes, and one or more notations (e.g., check or X marks) that indicate correspondences between the same.

[0101] The application of the segmentation schema as well as the mapping of the footnotes by the column identification component 125 may be performed using a combination of rules and algorithms configured to identify the event columns 130. For example, regular expressions, string match algorithms, keyword searches, software libraries including specialize parsing functions, or text segmentation and tokenization may be used to identify information defining event columns.

[0102] At block 750, the computing system inputs, to the LLM, a second prompt and the one or more event columns, in which the second prompt includes second instructions for determining information about one or more clinical trial events based on the one or more event columns. The second instructions may include a first instruction to, for individual event columns of the one or more event columns, identify one or more clinical trial events. For example, the first instructions may include language describing the contents of an event column and how to identify or assemble a clinical trial event from the contents of an event column.

[0103] The second instructions may include a second instruction to, for the individual event columns of the one or more event columns, determine a periodicity associated with individual clinical trial events of the one or more clinical trial events. For example, the event column may include header information, labels, instructions, or footnotes that assign or otherwise associate a periodicity (e.g., weekly, once every 10 days, etc.) with the clinical trial event.

[0104] The second instructions may include a third instruction, for the individual clinical trial events of the one or more clinical trial events, to generate expanded clinical trial event information for the individual clinical trial events, in which the expanded clinical trial event information is based on the determined periodicities associated with the individual clinical trial events. For example, the third instruction may include language explaining the distinction between finite and infinite expansions or simple and complex expansions.

[0105] The second instructions may include a fourth instruction to generate information about the one or more clinical trial events based on the expanded clinical trial event information for the individual clinical trial events of the one or more clinical trial events. For example, the fourth instruction may include language explaining how to combine, collate, or otherwise generate a data structure or output format that includes the information about the identified and expanded clinical trial events.

[0106] At block 760, the computing system receives, from the LLM, the information about the one or more clinical trial events. In some examples, the information about the one or more clinical trial events may be output to averification subsystem that can be used to determine the accuracy of the information about the one or more clinical trial events as determined by the LLM. As before, the verification subsystem may include the use of non-expert annotators to determine the accuracy of the information about the one or more clinical trial events generated by the LLM in block 750.

[0107] In some examples, the verification subsystem or the non-expert annotators employed therein, can provide modify or correct the information about the one or more clinical trial events. In some examples, the LLM prompt can be modified to include information based on the accuracy of the information about the one or more clinical trial events, as determined using the verification sub-system. This is one example of fine-tuning the LLM based on the output of the verification subsystem.

[0108] At block 770, the computing system generates structured schedule information including mapping the information about the one or more clinical trial events to a representation of a clinical trials schedule. For example, the representation of a clinical trials schedule may be a calendar or a representation provided in the user interface of a clinical trials management software program. The information about the one or more clinical trial events may be provided in a format that is easily mappable such as JSON or a widely used calendar format such as the ICS format.

[0109] At block 780, the computing system outputs the structured schedule information based on the representation of the clinical trials schedule to cause an action associated with a clinical trial event. For example, the calendar or representation mapped in block 770 can be used to perform various applications relating to clinical trials such as managing patient schedules, identifying drug interactions, patient monitoring, improving clinical trial design, and so on.

[0110] Referring now to FIG. 8, FIG. 8 shows an example method 800 for fine- tuning LLMs used for generating structured data for progress tracking using ML, according to some aspects of the present disclosure. The method 800 will be described with respect to the example system 100 shown in FIG. 1; however, any suitable system according to this disclosure may be similarly used.

[0111] It should be appreciated that method 800 provides a particular method for generating structured data for progress tracking using ML. Other sequences of operations may also be performed according to alternative examples. For example, alternative examples of the present disclosure may perform the steps outlined above in a different order. Moreover, the individual operations illustrated by method 800 may include multiple sub-operations that may be performed in various sequences as appropriate to the individual operation. Furthermore, additional operations may be added or removed depending on the particular applications. Further, the operations described in method 800 may be performed by different devices.

[0112] Method 800 may include block 810. At block 810, a computing system accesses unlabeled training data, comprising a number of pages including a first portion of pages with at least one event table on first individual pages and a second portion of pages without event tables on second individual pages. The unlabeled training data may thus be a collection of clinical trials scheduling documents intended for use in fine-tuning an LLM.

[0113] At block 820, the computing system determines a label for each page of the plurality of pages. For example, training data for fine-tuning can be generated using the set of unlabeled clinical trial protocol documents for the fine-tuning task. The unlabeled training data can be divided into fine-tuning, validation, and test portions.

[0114] The unlabeled training data can be processed using a base LLM that has not been fine-tuned and a suitable prompt to generate preliminary labeled training data. For example, the base LLM may be based on GPT-4 or PaLM 2, but other LLM types may be used.

[0115] A subset of the preliminary labeled training data can be examined and labeled by non-expert annotators to generate noisy training data. Noisy training data can be used in preference to expertly labeled (e.g., near-100% accurate labels) training data to preserve the training data’s independence and prevent any potential overfitting or bias towards the characteristics of the expertly labeled training data. Use of noisy training data can further enable generalizability and strong performance on unseen data, thus providing greater applicability in diverse clinical trial settings.

[0116] In some examples, ground truth training data can be used for comparison or evaluation. For example, ground truth training data can be generated from a diverse set of clinical trials with pages labeled as containing SoE tables by expert digitizers. The expert digitizers can be specifically trained to manually label and digitize clinical protocol documents using specialized software.

[0117] At block 830, the computing system inputs, to the LLM, a prompt and the plurality of pages. The prompt may include instructions for identifying one or more event tables on the first individual pages. The instructions may include an instruction to, for an individual page of the set of pages, perform a first binary classification of the page corresponding to a first presence of at least a portion of a table on the individual page.

[0118] For example, a prompt may be developed in consultation with expert digitizers. The effectiveness of the developed prompt can be checked using examples from the ground truth training data determined in block 820. An example prompt as may be used in block 830 is shown in FIG. 6.

[0119] At block 840, the computing system receives, from the LLM, an identification of a subset of pages of the plurality of pages, the subset of pages including the one or more event tables and third individual pages of the subset of pages including the at least the portion of the table. For example, the LLM can be directed to perform a binary classification task that indicates, for each page, whether or not it includes an event table, similar to the process described in block 730 with respect to FIG. 7.

[0120] At block 850, the computing system compares the identification of the subset of pages of the plurality of pages with the label of each page. For example, an output may be generated that indicates the identifications that match the label, the identifications that do not match the label, as well as descriptions and surrounding context of the event tables involves with each comparison.

[0121] At block 860, the computing system modifies the LLM based on the comparison using a fine-tuning technique. For example, one example technique may involve modifying the prompt with information about mis-identifications. For instance, the prompt can be appended with mis-identified tables in a suitable format (e.g., JSON) along with instructions to identify SoEs that are thus formattedas SoEs. Use fine-tuning techniques such as this one can significantly improve the recall, precision, accuracy, and other metrics associated with LLMs performing binary classifications.

[0122] Referring now to FIG. 9, FIG. 9 shows an example computing device 900 suitable for use in example systems or methods for generating structured data for progress tracking using ML according to this disclosure, according to some aspects of the present disclosure. The example computing device 900 includes a processor 910 which is in communication with the memory 920 and other components of the computing device 900 using one or more communications buses 902. The processor 910 is configured to execute processor-executable instructions stored in the memory 920 to perform one or more methods for training ML models or generating virtually stained images according to different examples, such as part or all of the example methods 700 or 800 described above with respect to FIG. 7 or FIG. 8. The computing device 900 also includes one or more user input devices 950, such as a keyboard, mouse, touchscreen, microphone, etc., to accept user input: however, in some examples, the computing device 900 may lack such user input devices, such as remote servers or cloud servers. The computing device 900 also includes a display 940 to provide visual output to a user.

[0123] The computing device 900 also includes a communications interface 930. In some examples, the communications interface 930 may enable communications using one or more networks, including a local area network (“LAN”): wide area network (“WAN”), such as the Internet: metropolitan area network (“MAN”); point-to-point or peer-to-peer connection; etc. Communication with other devices may be accomplished using any suitable networking protocol. For example, one suitable networking protocol may include the Internet Protocol (“IP”), Transmission Control Protocol (“TCP”), User Datagram Protocol (“UDP”), or combinations thereof, such as TCP / IP or UDP / IP.

[0124] The computing device 900 also includes one or more ML models 960. The ML models 960 may include one or a number of ML models 960 operating in parallel or in an ensemble configuration. The ML models 960 can include ML models 960 deployed locally as well as ML models 960 accessed remotely, such as via web-based API, or a combination thereof. Types of ML models 960 includeneural networks such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), Long Short-Term Memory networks (LSTMs), transformer models, decision trees, support vector machines (SVMs), Generative Pre-trained Transformer-series models (GPT-x), Bidirectional Encoder Representations from Transformers (BERT), and so on.

[0125] While some examples of methods and systems herein are described in terms of software executing on various machines, the methods and systems may also be implemented as specifically configured hardware, such as field- programmable gate array (FPGA) specifically to execute the various methods according to this disclosure. For example, examples can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in a combination thereof. In one example, a device may include a processor or processors. The processor comprises a computer-readable medium, such as a random-access memory (RAM) coupled to the processor. The processor executes computer-executable program instructions stored in memory, such as executing one or more computer programs. Such processors may comprise a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), field programmable gate arrays (FPGAs), and state machines. Such processors may further comprise programmable electronic devices such as PLCs, programmable interrupt controllers (PICs), programmable logic devices (PLDs), programmable read-only memories (PROMs), electronically programmable read-only memories (EPROMs or EEPROMs), or other similar devices.

[0126] Such processors may comprise, or may be in communication with, media, for example one or more non-transitory computer-readable media, that may store processor-executable instructions that, when executed by the processor, can cause the processor to perform methods according to this disclosure as carried out, or assisted, by a processor. Examples of non-transitory computer-readable medium may include, but are not limited to, an electronic, optical, magnetic, or other storage device capable of providing a processor, such as the processor in a web server, with processor-executable instructions. Other examples of non-transitory computer- readable media include, but are not limited to, a floppy disk, CD-ROM, magnetic disk, memory chip, ROM, RAM, ASIC, configured processor, all optical media, allmagnetic tape or other magnetic media, or any other medium from which a computer processor can read. The processor, and the processing, described may be in one or more structures, and may be dispersed through one or more structures. The processor may comprise code to carry out methods (or parts of methods) according to this disclosure.

[0127] FIG. 10 shows an example of an SoE table 1000 during a verification process, according to some aspects of the present disclosure. Table 1000 includes a column 1005 that has been selected for verification. For example, in block 225 of FIG. 2, expansions of identified clinical trial events can be classified as simple or complex and in block 230 a verification procedure can be applied to the output of block 225 for expansions classified as complex. In this case, the column 1005 has been identified as encapsulating a complex expansion. Column 1005 is shown highlighted with a thick outline as may be seen in some user interfaces, such as the one described in FIG. 11 below. The complex expansion may include identification of information 1010 in an event column that is converted into one or more clinical trial events. For instance, the information 1010 may be converted into a suitable data structure that defines one or more clinical events.

[0128] In some examples, the expansion is classified as complex by the LLM, which may correspond to a low estimated accuracy of the conversion of the information 1010 into a clinical trial event or a suitable data structure containing the information 1010. In this case, the identified column 1005, the contained information 1010, and the one or more clinical trial events derived therefrom can be verified using a verification process, an example of which is shown in FIG. 11 below.

[0129] FIG. 11 shows an example of a user interface 1100 that may be used during a verification process, according to some aspects of the present disclosure. For example, user interface 1100 could be used by a non-expert human annotator to verify the identified column 1005, the contained information 1010, and the one or more clinical trial events derived therefrom. User interface 1100 includes a column identifier 1105. Column identifier 1105 can be used by a non-expert human annotator to identify the associated SoE data to verify against, such as the SoE table 1000 shown in FIG. 10 above. In some examples, the table data may be integrated into the user interface 1100.

[0130] User interface 1100 includes verifications 1110. Verifications 1110 can include questions, selections, identifications, options, etc. for the non-expert human annotator using the user interface 1100 to determine correlations between the clinical events determined by the expansion performed by the LLM and the information 1010 expressed in the SoE table 1000. The non-expert human annotator can provide answers to the verifications 1110 using suitable controls that indicate the accuracy of the determined expansion and subsequently determined clinical trial events. User interface 1100 includes a trash control 1115 that can be used to indicate that the identification of, for example, a complex expansion was in error and that the subsequent determinations should be deleted. The selections and identifications made by the non-expert human annotator using user interface 1100 can be used to improve the accuracy of the generated schedule information and can additionally be fed back to the LLM to improve the accuracy of subsequent determinations and to fine-tune the LLM.

[0131] Embodiments may be implemented by using a computer program product, comprising computer program / instructions which, when executed by a processor, cause the processor to perform any of the methods described in the disclosure.

[0132] The foregoing description of some examples has been presented only for the purpose of illustration and description and is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Numerous modifications and adaptations thereof will be apparent to those skilled in the art without departing from the spirit and scope of the disclosure.

[0133] Reference herein to an example or implementation means that a particular feature, structure, operation, or other characteristic described in connection with the example may be included in at least one implementation of the disclosure. The disclosure is not restricted to the particular examples or implementations described as such. The appearance of the phrases “in one example,” “in an example,” “in one implementation,” or “in an implementation,” or variations of the same in various places in the specification does not necessarily refer to the same example or implementation. Any particular feature, structure, operation, or other characteristic described in this specification in relation to oneexample or implementation may be combined with other features, structures, operations, or other characteristics described in respect of any other example or implementation.

[0134] Use herein of the word “or” is intended to cover inclusive and exclusive OR conditions. In other words, A or B or C includes any or all of the following alternative combinations as appropriate for a particular usage: A alone; B alone; C alone; A and B only: A and C only; B and C only; and A and B and C.EXAMPLES

[0135] These illustrative examples are mentioned not to limit or define the scope of this disclosure, but rather to provide examples to aid understanding thereof. Illustrative examples are discussed above in the Detailed Description, which provides further description. Advantages offered by various examples may be further understood by examining this specification.

[0136] As used below, any reference to a series of examples is to be understood as a reference to each of those examples disjunctively (e.g., “Examples 1-4” is to be understood as “Examples 1, 2, 3, or 4”).

[0137] Example 1 is a method, comprising: receiving a document comprising a plurality of pages, wherein first individual pages of the plurality of pages comprise at least one of unstructured data or an event table: inputting, to a large language model (LLM), a first prompt and the plurality of pages, wherein the first prompt includes first instructions for identifying one or more event tables; receiving, from the LLM, a subset of pages of the plurality of pages, the subset of pages including the one or more event tables and second individual pages of the subset of pages including at least a portion of a first event table; identifying, for each event table of the one or more event tables, one or more event columns; inputting, to the LLM, a second prompt and the one or more event columns, wherein the second prompt includes second instructions for determining information about one or more clinical trial events based on the one or more event columns; receiving, from the LLM, the information about the one or more clinical trial events; generating structured schedule information comprising mapping the information about the one or more clinical trial events to a representation of a clinical trials schedule; and outputting the structured schedule information based on the representation of the clinical trials schedule to cause an action associated with a clinical trial event.

[0138] Example 2 is the method of example(s) 1, wherein individual tables of the one or more event tables comprise at least an activity component and a corresponding temporal component.

[0139] Example 3 is the method of example(s) 2, wherein the individual tables of the one or more event tables further comprise a footnote component.

[0140] Example 4 is the method of example(s) 1, wherein the first instructions for identifying the one or more event tables comprise: a first instruction to, for an individual page of the plurality of pages, perform a first binary classification of the individual page corresponding to a first presence of the at least the portion of the event table on the individual page.

[0141] Example 5 is the method of example(s) 4, wherein the first instructions for identifying the one or more event tables further comprise: a second instruction to, for the individual page of the plurality of pages, perform a second binary classification of the individual page corresponding to a second presence of one or more footnotes.

[0142] Example 6 is the method of example(s) 5, further comprising: outputting the subset of pages to a verification subsystem; and receiving, for the second individual pages of the subset of pages, from the verification subsystem, a first indication of the accuracy of the first binary classification and a second indication of the accuracy of the second binary classification.

[0143] Example 7 is the method of example(s) 6, wherein: the first indication of the accuracy of the first binary classification is based on a first determination generated by a non-expert annotator; and the second indication of the accuracy of the second binary classification is based on a second determination generated by the non-expert annotator.

[0144] Example 8 is the method of example(s) 1, wherein the document is a portable document format (PDF) file.

[0145] Example 9 is the method of example(s) 1, wherein inputting, to the LLM, the first prompt and the plurality of pages, comprises converting the document to a composite format comprising a structured data portion and a textual portion.

[0146] Example 10 is the method of example(s) 1, wherein the information about the one or more clinical trial events comprises, for individual clinical trial events, one or more of: an event description, an event timing, an event window, or a footnote.

[0147] Example 11 is the method of example(s) 10, wherein the second instructions comprise: a first instruction, for individual event columns of the one ormore event columns, to identify the one or more clinical trial events; a second instruction, for the individual event columns of the one or more event columns, to determine a periodicity associated with individual clinical trial events of the one or more clinical trial events; a third instruction, for the individual clinical trial events of the one or more clinical trial events, to generate expanded clinical trial event information for the individual clinical trial events, wherein the expanded clinical trial event information is based on the determined periodicities associated with the individual clinical trial events; and a fourth instruction to generate information about the one or more clinical trial events based on the expanded clinical trial event information for the individual clinical trial events of the one or more clinical trial events.

[0148] Example 12 is the method ofexample(s) 11, further comprising: outputting the information about the one or more clinical trial events to a verification subsystem; and receiving, from the verification subsystem, a first indication of the accuracy of the information about the one or more clinical trial events.

[0149] Example 13 is the method ofexample(s) 12, further comprising: receiving, from the verification subsystem, a second indication of a modification to the information about the one or more clinical trial events.

[0150] Example 14 is the method of example(s) 12, further comprising modifying the second prompt to include information about the first indication of the accuracy of the information about the one or more clinical trial events.

[0151] Example 15 is the method ofexample(s) 12, wherein: the first indication of the accuracy of the information about the one or more clinical trial events is generated by a non-expert annotator.

[0152] Example 16 is the method of example(s) 11, wherein the second instructions further comprise a fourth instruction, for the individual clinical trial events of the one or more clinical trial events, to determine a modification to the expanded clinical trial event information based on at least one footnote included in the individual clinical trial events.

[0153] Example 17 is the method of example(s) 11, wherein the first instruction to identify the one or more clinical trial events comprises sub-instructions including: a first sub-instruction to identify a first event column of the individual event columns of the one or more event columns; a second sub -instruction to identify a header of the first event column, comprising a header text string; and a third sub -instruction to determine a first clinical trial event using a regular expression applied to the header text string.

[0154] Example 18 is the method of examp le(s) 11, wherein identifying the one or more event columns is based on a segmentation schema associated with at least one of the one or more event tables.

[0155] Example 19 is the method of example(s) 1, wherein the LLM is based on a transformer architecture.

[0156] Example 20 is a system, comprising: one or more processors; and one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform the method of any of example(s)s 1-19.

[0157] Example 21 is one or more computer-readable media storing computerexecutable instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any of example(s)s 1-19.

[0158] Example 22 is a method, comprising: accessing unlabeled training data, comprising a plurality of pages including a first portion of pages with at least one event table on first individual pages and a second portion of pages without event tables on second individual pages; determining a label for each page of the plurality of pages; inputting, to a large language model (LLM), a prompt and the plurality of pages, wherein the prompt includes instructions for identifying one or more event tables on the first individual pages, comprising a first instruction to, for an individual page of the plurality of pages, perform a first binary classification of the page corresponding to a first presence of at least a portion of a table on the individual page; receiving, from the LLM, an identification of a subset of pages of the plurality of pages, the subset of pages including the one or more event tables and third individual pages of the subset of pages including the at least the portion of the table; comparing the identification of the subset of pages of the plurality of pages with the label for each page of the plurality of pages; and modifying the LLM, based on the comparison, using a fine-tuning technique.

Claims

CLAIMSThat which is claimed is:

1. A method, comprising: receiving a document comprising a plurality of pages, wherein first individual pages of the plurality of pages comprise at least one of unstructured data or an event table: inputting, to a large language model (LLM), a first prompt and the plurality of pages, wherein the first prompt includes first instructions for identifying one or more event tables; receiving, from the LLM, a subset of pages of the plurality of pages, the subset of pages including the one or more event tables and second individual pages of the subset of pages including at least a portion of a first event table: identifying, for each event table of the one or more event tables, one or more event columns; inputting, to the LLM, a second prompt and the one or more event columns, wherein the second prompt includes second instructions for determining information about one or more clinical trial events based on the one or more event columns; receiving, from the LLM, the information about the one or more clinical trial events: generating structured schedule information comprising mapping the information about the one or more clinical trial events to a representation of a clinical trials schedule; and outputting the structured schedule information based on the representation of the clinical trials schedule to cause an action associated with a clinical trial event.

2. The method of claim 1, wherein individual tables of the one or more event tables comprise at least an activity component and a corresponding temporal component.

3. The method of claim 2, wherein the individual tables of the one or more event tables further comprise a footnote component.

4. The method of any one of claims 1-3, wherein the first instructions for identifying the one or more event tables comprise: a first instruction to, for an individual page of the plurality of pages, perform a first binary classification of the individual page corresponding to a first presence of the at least the portion of the event table on the individual page.

5. The method of any one of claims 4, wherein the first instructions for identifying the one or more event tables further comprise: a second instruction to, for the individual page of the plurality of pages, perform a second binary classification of the individual page corresponding to a second presence of one or more footnotes.

6. The method of claim 5, further comprising: outputting the subset of pages to a verification subsystem; and receiving, for the second individual pages of the subset of pages, from the verification subsystem, a first indication of the accuracy of the first binary classification and a second indication of the accuracy of the second binary classification.

7. The method of claim 6, wherein: the first indication of the accuracy of the first binary classification is based on a first determination generated by a non-expert annotator: and the second indication of the accuracy of the second binary classification is based on a second determination generated by the non-expert annotator.

8. The method of any one of claims 1-7, wherein the document is a portable document format (PDF) file.

9. The method of any one of claims 1-8, wherein inputting, to the LLM, the first prompt and the plurality of pages, comprises converting the document to a composite format comprising a structured data portion and a textual portion.

10. The method of any one of claims 1-9, wherein the information about the one or more clinical trial events comprises, for individual clinical trial events, one or more of: an event description, an event timing, an event window, or a footnote.

11. The method of claim 10, wherein the second instructions comprise: a first instruction, for individual event columns of the one or more event columns, to identify the one or more clinical trial events; a second instruction, for the individual event columns of the one or more event columns, to determine a periodicity associated with individual clinical trial events of the one or more clinical trial events: a third instruction, for the individual clinical trial events of the one or more clinical trial events, to generate expanded clinical trial event information for the individual clinical trial events, wherein the expanded clinical trial event information is based on the determined periodicities associated with the individual clinical trial events: and a fourth instruction to generate information about the one or more clinical trial events based on the expanded clinical trial event information for the individual clinical trial events of the one or more clinical trial events.

12. The method of claim 11, further comprising: outputting the information about the one or more clinical trial events to a verification subsystem; and receiving, from the verification subsystem, a first indication of the accuracy of the information about the one or more clinical trial events.

13. The method of claim 12, further comprising: receiving, from the verification subsystem, a second indication of a modification to the information about the one or more clinical trial events.

14. The method of claim 12, further comprising modifying the second prompt to include information about the first indication of the accuracy of the information about the one or more clinical trial events.

15. The method of claim 12, wherein: the first indication of the accuracy of the information about the one or more clinical trial events is generated by a non-expert annotator.

16. The method of claim 11, wherein the second instructions further comprise a fourth instruction, for the individual clinical trial events of the one or more clinical trial events, to determine a modification to the expanded clinical trial event information based on at least one footnote included in the individual clinical trial events.

17. The method of claim 11, wherein the first instruction to identify the one or more clinical trial events comprises sub -instructions including: a first sub-instruction to identify a first event column of the individual event columns of the one or more event columns; a second sub -instruction to identify a header of the first event column, comprising a header text string: and a third sub-instruction to determine a first clinical trial event using a regular expression applied to the header text string.

18. The method of claim 11, wherein identifying the one or more event columns is based on a segmentation schema associated with at least one of the one or more event tables.

19. The method of any one of claims 1-18, wherein the LLM is based on a transformer architecture.

20. A system, comprising:one or more processors; and one or more memories storing computer-executable instructions that, when executed by the one or more processors, cause the system to perform the method of any of claims 1-19.

21. One or more computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any of claims 1-19.

22. An apparatus, comprising: means for implementing the operations of the method of any of claims 1-19.

23. A computer program product comprising computer instructions that, when executed by a processor, implement the operations of the method of any of claims 1- 19.

24. A method, comprising: accessing unlabeled training data, comprising a plurality of pages including a first portion of pages with at least one event table on first individual pages and a second portion of pages without event tables on second individual pages: determining a label for each page of the plurality of pages; inputting, to a large language model (LLM), a prompt and the plurality of pages, wherein the prompt includes instructions for identifying one or more event tables on the first individual pages, comprising a first instruction to, for an individual page of the plurality of pages, perform a first binary classification of the page corresponding to a first presence of at least a portion of a table on the individual page; receiving, from the LLM, an identification of a subset of pages of the plurality of pages, the subset of pages including the one or more event tables and third individual pages of the subset of pages including the at least the portion of the table;comparing the identification of the subset of pages of the plurality of pages with the label for each page of the plurality of pages; and modifying the LLM, based on the comparison, using a fine-tuning technique.

Images