Automated and semi-automated extraction of data from tables and graphs in scientific literature

Abstract

A system and method for automatically extracting data from tables and graphs in scientific literature, particularly in life sciences and healthcare, is presented. The invention employs a hybrid approach combining computer vision, natural language processing (NLP), and a large language model (LLM)-based data extraction module. A neural network enhances accuracy by providing contextual information. The system performs table structure detection, optical character recognition (OCR), Vision Transformers (ViTs) for text recognition, and graph-to- table conversion. An LLM refines the extracted data using advanced prompt engineering. A user interface enables data review, validation, and iterative refinement. The system employs a novel two-stage approach: de-rendering tables and graphs into a machine-readable format, followed by interpretation and user-defined mapping. A knowledge graph enhances extraction by resolving ambiguities and inferring relationships. Designed for scalability and continuous improvement, the invention significantly enhances data extraction efficiency and accuracy, accelerating research and knowledge discovery.

Description

PATENT COOPERATION TREATY (PCT) PATENT APPLICATIONTITLE: AUTOMATED AND SEMI- AUTOMATED EXTRACTION OF DATA FROM TABLES AND GRAPHS IN SCIENTIFIC LITERATUREAPPLICANT: Evidence Prime sp. z o.o.INVENTORS: Artur Nowak, Pawel KunstmanABSTRACTA system and method for automatically extracting data from tables and graphs in scientific literature, particularly in life sciences and healthcare, is presented. The invention employs a hybrid approach combining computer vision, natural language processing (NLP), and a large language model (LLM)-based data extraction module. A neural network enhances accuracy by providing contextual information. The system performs table structure detection, optical character recognition (OCR), Vision Transformers (ViTs) for text recognition, and graph-to- table conversion. An LLM refines the extracted data using advanced prompt engineering. A user interface enables data review, validation, and iterative refinement. The system employs a novel two-stage approach: de-rendering tables and graphs into a machine-readable format, followed by interpretation and user-defined mapping. A knowledge graph enhances extraction by resolving ambiguities and inferring relationships. Designed for scalability and continuous improvement, the invention significantly enhances data extraction efficiency and accuracy, accelerating research and knowledge discovery.CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 626,575, filed on Jan 30, 2024, the entire contents of which are incorporated herein by reference.FIELD OF THE INVENTION[2] The present invention relates to the automatic and semi-automatic extraction of data from tables and graphs present in scientific literature, particularly (but not limited to) the context of life sciences and healthcare publications.BACKGROUND OF THE INVENTION[3] The volume of scientific literature, particularly in the life sciences and healthcare domains, is growing exponentially. This vast repository of knowledge holds immense potential for advancing research, improving patient care, and developing new therapies. However, a significant portion of this valuable information is presented in tables and graphs, which are often not readily accessible for automated analysis.[4] Extracting data from tables and graphs in scientific literature presents several challenges. These include the wide variety of table and graph formats, the lack of standardization across publications, the use of domain-specific terminology and abbreviations, the importance of understanding the surrounding textual context, and image quality issues in scanned documents.[5] Several approaches have been developed to address the challenge of data extraction from tables and graphs. These include template-based systems, rule-based systems, and early machine learning techniques. However, these approaches have significant limitations. Template-based systems are often brittle and fail when confronted with variations in layout or formatting. Rule-based systems, while effective for consistent formats, struggle with complex or irregular layouts and require significant manual effort to develop and maintain. Early machine learning techniques often require large amounts of labeled training data, which can be difficult and expensive to obtain, and they may not generalize well to new or unseen formats. While optical character recognition (OCR) and natural language processing (NLP) techniques have advanced, they still struggle with the complexities of scientific text, particularly when dealing with complex table structures and the inherent ambiguity of scientific data.[6] More recently, multi-modal Large Language Models (LLMs) have shown promise in understanding and processing both text and images. However, applying them effectively to scientific literature presents unique challenges. LLMs can struggle with the accurate interpretation of highly specialized terminology, resolving ambiguities that require deep domain knowledge, and handling the wide variability in table and graph formats. They alsooften fail to integrate information from multiple tables and graphs within a document to create a coherent understanding of the data.[7] Crucially, existing systems, including those employing LLMs, lack the ability to engage in an iterative refinement process with the user. They operate on a "one-shot" basis, taking an image or document as input and producing a fixed output. This "black box" approach often results in outputs that do not meet the specific needs of researchers, who may have particular interpretations or require data to be mapped to specific ontologies or formats. Without the ability to interact with the system and refine the extraction, researchers are forced to either manually correct errors or abandon automated extraction altogether.[8] In the life sciences and healthcare domains, these challenges are further compounded by the complexity of clinical data representation, variability in reporting standards, and data privacy concerns.[9] The present invention addresses these limitations by providing a novel system and method that enables accurate, efficient, and user-guided data extraction from tables and graphs in scientific literature.SUMMARY OF THE INVENTION

[0010] To address the foregoing problems, in whole or in part, and / or other problems that may have been observed by persons skilled in the art, the present disclosure provides methods and systems as described by way of example as set forth below.

[0011] The present invention addresses the limitations of prior art by providing methods and systems for automated and semi-automated extraction of data from tables and graphs within scientific literature, particularly in the life sciences and healthcare fields. The invention employs a novel two-stage, iterative approach that allows for user interaction and refinement, ensuring the accuracy and relevance of the extracted data. This two-stage approach, with its emphasis on user interaction and iterative refinement, is a key innovation of the present invention. It allows researchers to guide the extraction process, ensuring that the results meet their specific needs and interpretations, which is not possible with existing automated data extraction systems.

[0012] The two-stage approach comprises de-rendering stage and interpretation Stage. The de-rendering stage focuses on converting tables and graphs from their visual representation in the scientific document into a standardized, machine-readable intermediate format (e.g., a structured data representation like a table or a graph-like data structure). This stage utilizes ahybrid approach employing computer vision and deep learning techniques, including Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) for tasks such as table structure detection and graph type classification, along with traditional computer vision algorithms for tasks like axis and data point extraction. It also incorporates rule-based systems that refine the results of deep learning models and incorporate domain-specific knowledge about table and graph structures. A neural network processes the document's textual content to generate contextual embeddings, which are used to guide the de-rendering process, improving the accuracy of table structure detection, graph type classification, and data point extraction.

[0013] The interpretation stage takes the intermediate data representation produced by the de-rendering stage and maps it to a user-defined output format, performing semantic interpretation, data normalization, and other necessary transformations. This stage leverages the power of multi-modal Large Language Models (LLMs), guided by advanced prompt engineering tailored to the specific characteristics of scientific literature. The LLM receives multi-modal input, including the intermediate data representation, images of the original tables and graphs, and relevant text segments identified using the contextual embeddings. The system employs user-defined and potentially LLM-generated code snippets to ensure transparent and reproducible data transformation, allowing users to understand and customize how data is mapped to the desired output format. Importantly, the system allows for iterative refinement of the LLM's output through a user interface.

[0014] Key to the invention is the user interface, which enables researchers to review, validate, and refine the extracted data at both the de-rendering and interpretation stages. This iterative feedback loop allows researchers to correct errors in OCR output, adjust table structure interpretations, modify graph-to-table conversion results, refine the mapping between the intermediate data representation and the user-defined output format, and provide feedback on the LLM's interpretations and transformations.

[0015] This iterative process, facilitated by the user interface, ensures that the extracted data aligns with the user's specific needs and interpretations, addressing a critical limitation of existing systems. The user feedback is also used to continuously improve the system's performance through machine learning, enhancing the accuracy and adaptability of both the de- rendering and interpretation stages.

[0016] The invention is designed to handle the wide variety of table and graph formats encountered in scientific literature, including complex layouts, multi-level headers, and merged cells. While applicable to scientific literature broadly, the invention is particularlyoptimized for the nuances of life sciences and healthcare data, including specialized terminology, data representation conventions, and the integration of domain-specific ontologies and knowledge bases.

[0017] The system can optionally construct a knowledge graph from extracted data and relationships, enabling advanced reasoning, inference, and enhanced data exploration. The system also incorporates mechanisms for estimating and handling uncertainty in the extracted data, including assigning confidence scores and using the LLM to resolve ambiguities.

[0018] The system is designed to be scalable, handling large datasets and high-throughput data extraction tasks, leveraging cloud computing, parallel processing, and other optimization techniques. It is also designed for continuous improvement, learning from user feedback and new data to adapt to evolving research needs and maintain high accuracy.

[0019] By enabling a two-stage, iterative extraction process with user refinement, the present invention represents a significant advancement in automated data extraction from scientific literature, empowering researchers to efficiently and accurately unlock the valuable information contained within tables and graphs.

[0020] Additional features of the invention will be or will become apparent to one with skill in the art upon examination of the following figures and detailed description. It is intended that all such additional features and advantages be included within this description, be within the scope of the invention, and be protected by the accompanying claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Having thus described the subject matter of the present invention in general terms, reference will now be made to the accompanying drawings, and wherein:

[0022] Figure 1 shows system architecture diagram illustrating the modules and data flow for automated and semi-automated data extraction from scientific literature, highlighting the two-stage approach (de-rendering and interpretation), in accordance with an embodiment of the present invention;

[0023] Figure 2 shows flowchart illustrating the overall data extraction process, highlighting the two-stage approach (de-rendering and interpretation) and iterative refinement, in accordance with an embodiment of the present invention;

[0024] Figure 3 shows user Interface showing the table mapping functionality, allowing users to map table rows and columns to entities or concepts, in accordance with an embodiment of the present invention;

[0025] Figure 4 shows user interface showing the extracted data review and validation process, with confidence scores and options for user corrections, in accordance with an embodiment of the present invention;

[0026] Figure 5 illustrates the structure of the knowledge graph, with nodes representing entities and edges representing relationships, in accordance with an embodiment of the present invention;

[0027] Figure 6 illustrates user Interface showing the knowledge graph exploration functionality, allowing users to search, visualize, and query the extracted knowledge, in accordance with an embodiment of the present invention;

[0028] Skilled artisans will appreciate that the elements in the drawings are illustrated for clarity and simplicity, focusing on the functional aspects of the software design. The drawings emphasize components and workflows pertinent to understanding the embodiments of the present invention, omitting extraneous details that would be readily apparent to those of ordinary skill in the art.DETAILED DESCRIPTION OF THE INVENTION

[0029] The subject matter of the present invention now will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the subject matter of the present invention are shown. Like numbers refer to like elements throughout. The subject matter of the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Indeed, many modifications and other embodiments of the subject matter of the present invention set forth herein will come to mind to one skilled in the art to which the subject matter of the present invention pertains having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. All illustrations of the drawings are for the purpose of describing selected versions of the present invention and are not intended to limit the scope of the present invention. Therefore, it is to be understood that the subject matter of the present invention is not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims.

[0030] As a preliminary matter, it wifi readily be understood by one having ordinary skill in the relevant art that the present disclosure has broad utility and application. As should beunderstood, any embodiment may incorporate only one or a plurality of the above-disclosed aspects of the disclosure and may further incorporate only one or a plurality of the abovedisclosed features. Furthermore, any embodiment discussed and identified as being “preferred” is considered to be part of a best mode contemplated for carrying out the embodiments of the present disclosure. Other embodiments also may be discussed for additional illustrative purposes in providing a full and enabling disclosure. Moreover, many embodiments, such as adaptations, variations, modifications, and equivalent arrangements, will be implicitly disclosed by the embodiments described herein and fall within the scope of the present disclosure.

[0031] Accordingly, while embodiments are described herein in detail in relation to one or more embodiments, it is to be understood that this disclosure is illustrative and example of the present disclosure and are made merely for the purposes of providing a full and enabling disclosure. The detailed disclosure herein of one or more embodiments is not intended, nor is to be construed, to limit the scope of patent protection afforded in any claim of a patent issuing here from, which scope is to be defined by the claims and the equivalents thereof. It is not intended that the scope of patent protection be defined by reading into any claim a limitation found herein that does not explicitly appear in the claim itself.

[0032] Thus, for example, any sequence(s) and / or temporal order of steps of various processes or methods that are described herein are illustrative and not restrictive.Accordingly, it should be understood that, although steps of various processes or methods may be shown and described as being in a sequence or temporal order, the steps of any such processes or methods are not limited to being carried out in any particular sequence or order, absent an indication otherwise. Indeed, the steps in such processes or methods generally may be carried out in various different sequences and orders while still falling within the scope of the present invention. Accordingly, it is intended that the scope of patent protection is to be defined by the issued claim(s) rather than the description set forth herein.

[0033] Additionally, it is important to note that each term used herein refers to that which an ordinary artisan would understand such term to mean based on the contextual use of such term herein. To the extent that the meaning of a term used herein - as understood by the ordinary artisan based on the contextual use of such term - differs in any way from any particular dictionary definition of such term, it is intended that the meaning of the term as understood by the ordinary artisan should prevail.

[0034] Furthermore, it is important to note that, as used herein, “a” and “an” each generally denotes “at least one”, but does not exclude a plurality unless the contextual use dictatesotherwise. When used herein to join a list of items, “or” denotes “at least one of the items”, but does not exclude a plurality of items of the list. Finally, when used herein to join a list of items, “and” denotes “all of the items of the list”.

[0035] The present invention relates to a system for automatically and semi-automatically extracting data from tables and graphs in scientific literature, particularly in the life sciences and healthcare domains, using a hybrid approach that integrates computer vision, natural language processing (NLP), and large language models (LLMs).

[0036] The exponential growth of scientific literature, especially in life sciences and healthcare, presents a significant challenge for researchers seeking to synthesize and analyze the vast amounts of data contained within. Much of this crucial information is presented in tables and graphs, which, while visually informative for humans, are not readily accessible for automated analysis by existing systems. Current automated data extraction methods often struggle to accurately and reliably extract data from these visual representations due to several key limitations.

[0037] Scientific publications employ a wide range of table and graph formats, often with complex layouts, multi-level headers, nested structures, and inconsistent formatting. Existing systems often rely on rigid templates or rules that fail to adapt to this variability, leading to inaccurate or incomplete data extraction.

[0038] Unlike some other domains, scientific literature generally lacks strict standardization in table and graph design, making it difficult to develop universally applicable extraction methods.

[0039] The specialized language used in life sciences and healthcare, including complex terminology, acronyms, and abbreviations, poses a significant challenge for general-purpose data extraction tools, leading to misinterpretations and errors.

[0040] The meaning of data within tables and graphs is often heavily dependent on the surrounding textual context of the article. Existing systems often fail to adequately incorporate this context, leading to ambiguities and misinterpretations.

[0041] Many scientific articles, especially older ones, are only available as scanned images, which may have low resolution, noise, or other artifacts that hinder optical character recognition (OCR) and make it difficult to accurately identify table structures and extract text.

[0042] Crucially, existing systems lack the ability to engage in an iterative refinement process with the user. They operate on a "one-shot" basis, taking an image or a document as input and producing a fixed output without allowing the user to review, correct, or guide theextraction process. This "black box" approach often results in outputs that do not meet the specific needs of researchers, who may have particular interpretations or require data to be mapped to specific ontologies or formats.

[0043] Without the ability to interact with the system and refine the extraction, researchers are forced to either manually correct errors or abandon automated extraction altogether, leading to significant inefficiencies and hindering the progress of scientific discovery. The inability of researchers to guide the system and ensure the extracted data aligns with their specific needs and interpretations is a major limitation that the present invention addresses.

[0044] The present invention provides a system and method for automatically extracting data from tables and graphs in scientific literature, particularly in life sciences and healthcare. It employs a two-stage data extraction process consisting of a de-rendering stage and an interpretation stage. In the de-rendering stage, tables and graphs are converted into a standardized intermediate data representation using computer vision techniques and a neural network that processes document context. The interpretation stage then maps this intermediate representation to a user-defined output format using a large language model (LLM), guided by prompts tailored to scientific data. A user interface module allows users to review and modify both the intermediate representation and the final mapping, ensuring iterative refinement. The dynamic update module integrates user modifications to continuously improve the system’s accuracy and adaptability, enhancing the de-rendering and interpretation processes over time.

[0045] By addressing key limitations in existing automated extraction systems, this invention significantly improves accuracy, adaptability, and efficiency. Unlike conventional "black box" methods, it enables researchers to interactively refine extracted data, ensuring it aligns with specific needs and domain-specific terminology. The hybrid approach - leveraging deep learning, computer vision, neural networks, and LLMs - effectively handles data variety, lack of standardization, and contextual dependencies. Moreover, the ability to process low-quality scanned images and resolve ambiguities through contextual analysis and knowledge graph integration further enhances the system’s robustness. Ultimately, the invention accelerates research and knowledge discovery by making complex scientific data more accessible and structured for automated analysis.

[0046] In accordance with an embodiment of the present invention, the system for automatically extracting data from tables and graphs in scientific literature is designed with a modular architecture, as illustrated in Figure 1. This modularity allows for flexibility, scalability, and ease of maintenance. The system employs a novel two-stage approach,comprising a de-rendering stage 102 and an interpretation stage 103. This two-stage approach, with its emphasis on user interaction and iterative refinement, is a key innovation of the present invention. It allows researchers to guide the extraction process, ensuring that the results meet their specific needs and interpretations, which is not possible with existing automated data extraction systems. The system also includes a preprocessing module 101, a post processing and data integration module 104, a user interface module 105, an uncertainty handling module 106, and an optional knowledge graph construction and utilization module 107. These modules work together to perform the data extraction process, with data flowing sequentially from input to output.

[0047] The preprocessing module 101 is responsible for the initial processing of the input scientific documents. It accepts scientific articles in various formats, including PDF files and scanned images 100. The module performs the following key functions:® Text Extraction: The preprocessing module employs Optical Character Recognition (OCR) and other computer vision techniques to extract the textual content of the document. The OCR engine is enhanced with a dynamically adaptable language model tailored for scientific literature, particularly in the life sciences and healthcare domains. It is capable of handling a variety of fonts, special characters, and formatting conventions commonly found in scientific publications. It can also be substituted or supported by other computer vision techniques, such as Vision Transformers (ViTs) to improve accuracy. Furthermore, to address the issue of poor-quality scans, the text extraction component incorporates advanced image preprocessing techniques, such as adaptive binarization, noise reduction, and skew correction. This ensures that the OCR engine or other techniques receive the highest quality image data possible, leading to more accurate text recognition.® Table and Graph Identification: The preprocessing module utilizes computer vision algorithms to detect and isolate tables and graphs within the document. This component employs a combination of techniques, including deep learning models such as Faster R-CNN or YOLO, trained on large datasets of scientific documents, and rule-based systems that incorporate domain-specific knowledge about the structure and layout of tables and graphs. For example, the system might use heuristics based on the presence of horizontal and vertical lines, the density of text in certain regions, and the presence of captions or titles that often accompany tables and graphs. The identified regions of interest (i.e., potential tables and graphs) are thenpassed to the de-rendering module 102 for further processing. The module outputs extracted text and identified tables and graphs to the de-rendering module 102.

[0048] In an embodiment of the present invention, the de-rendering module is responsible for converting the identified tables and graphs into a standardized, machine-readable intermediate representation. It employs a hybrid approach, combining deep learning techniques with rule-based systems and traditional computer vision algorithms. The primary goal of this module is to accurately capture the structural and semantic information from the visual representations of tables and graphs, transforming them into a format that can be further processed by the interpretation module.

[0049] In an embodiment of the present invention, the system employs a combination of deep learning models and rule-based systems for accurate table structure detection. Deep learning models, such as Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs), are trained to identify table boundaries, rows, columns, and cells with high accuracy. These models can handle a wide variety of table formats, including those with complex layouts and nested structures. They are trained on large datasets of annotated scientific tables, such as PubTables-lM and other domain-specific datasets. ViTs are particularly well-suited for this task due to their ability to capture global context through self-attention mechanisms.

[0050] The table structure detection process incorporates contextual information from a neural network, which processes the document's textual content (described in the below paragraphs). This provides semantic information to guide the algorithms. For instance, if the text mentions a "table summarizing clinical trial results," the system can prioritize table structures commonly used in clinical trials and use this information to inform the selection of an optimal OCR strategy, leveraging the recognized accuracy advantages of Vision Transformers in processing complex visual layouts.

[0051] Following the initial deep learning-based detection, an adaptive rule-based system refines the results. This system incorporates domain-specific knowledge about table structures commonly found in life sciences and healthcare literature. The rules are dynamically adjusted based on user feedback and contextual clues. The system learns from user corrections and adjustments made through the user interface. For instance, if users frequently merge cells that the system initially identified as separate, the system can learn to adjust its merging rules accordingly. The rule-based system also considers the table's context, as interpreted by the neural network. For example, if the surrounding text indicates that thetable presents data from a specific type of study (e.g., a meta-analysis), the system can apply rules tailored to the typical table formats used in that type of study.

[0052] The system may employ a hybrid approach, combining the outputs of the deep learning model and the rule-based system. An uncertainty estimation mechanism assigns a confidence score to each detected table component. Regions with high uncertainty are flagged for review by the user or further analyzed using the LLM-based module in the interpretation stage.

[0053] In an embodiment, the system utilizes advanced OCR and text recognition techniques to extract text from the identified table cells. Recognizing the challenges of accurately extracting text from scientific tables, especially in the presence of low-quality scans, specialized terminology, and varied formatting. For doing so, the system employs various strategies.

[0054] The system employs a high-accuracy OCR engine capable of handling a variety of fonts, languages, and special characters commonly found in scientific literature. The specific OCR engine can be selected based on the characteristics of the document and the desired performance. Examples of suitable engines include, but are not limited to, Tesseract (especially versions 4.x or 5.x with LSTM models), Google Cloud Vision API, and Amazon Textract. To improve accuracy in the life sciences and healthcare domain, the OCR engine may be further enhanced through:

[0055] Further, the engine may be fine-tuned on a large corpus of scientific text, with a particular focus on life sciences and healthcare publications. This fine-tuning process helps the engine learn the specific characteristics of scientific writing, including common abbreviations, symbols, and formatting conventions.

[0056] The OCR engine may integrate a dynamically adaptable language model. This language model is initially trained on a broad corpus of scientific literature but is then further customized based on document context and user feedback.

[0057] In an embodiment, the neural network, which analyzes the document's textual context, provides information to the language model about the specific topic and terminology used in the article. This allows the language model to better predict and correct OCR output based on the expected vocabulary.

[0058] The language model is continuously updated based on user corrections and validations. For example, if users frequently correct a particular OCR error, the language model learns to adjust its predictions accordingly.

[0059] Recognizing the superior performance of Vision Transformers in various computer vision tasks, including OCR, the system may incorporate ViT-based models to enhance text recognition accuracy, especially in cases of complex table structures or low-quality images. The identified table structure or graph type can be used to inform the selection of an optimal OCR strategy, leveraging the recognized accuracy advantages of Vision Transformers in processing complex visual layouts.

[0060] Before OCR, the table cell images undergo a series of preprocessing steps to enhance image quality and improve OCR accuracy. These steps may include: o Adaptive Binarization: Instead of using a fixed threshold, an adaptive binarization technique is employed to convert the cell image to black and white, taking into account local variations in illumination and contrast. Examples include Otsu's method, Niblack, or Sauvola binarization. o Noise Reduction: Filters such as Gaussian blur, median filtering, or non-local means denoising are applied to reduce noise and artifacts that can interfere with OCR. The specific filter and parameters are chosen based on the characteristics of the scanned images. Deep learning models trained to remove noise from scanned documents can also be used as a complementary or alternative approach to traditional filtering. o Skew Correction and Deskewing: Algorithms are used to detect and correct any skew or rotation in the table cell images, ensuring that text lines are horizontal. Examples include Hough transform-based methods or projection profile analysis. o Layout specific corrections: For example, removing lines that might interfere with the characters, but are part of the table structure.

[0061] In an embodiment, the raw OCR output is further processed to correct errors and improve text quality. This includes dictionary-based correction, where a specialized dictionary containing scientific terms, drug names, gene names, and other domain-specific vocabulary is used to identify and correct potential OCR errors.

[0062] The system also employs contextual error correction by leveraging the neural network and the LLM to identify and correct errors based on the surrounding text. For example, if the OCR output for a cell is "100 mg" (with an incorrect "1" instead of "1"), but the context indicates that the cell should contain a dosage value, the system can correct the error using information from the neural network or by querying the LLM with a prompt like:"What is the most likely correct dosage value, given that the OCR output is '100 mg' and the context is [brief description of the context from the neural network]?"

[0063] Statistical language models, such as n-gram models trained on scientific text, are also utilized to identify and correct improbable word sequences or character combinations.

[0064] Further, the system may employ a hybrid OCR approach, combining the outputs of multiple OCR engines and using an uncertainty estimation mechanism to assign confidence scores. Low-confidence results are flagged for review or further analysis by the LLM. The system may dynamically select the most appropriate OCR technique based on thespecific characteristics of the document or the detected table / graph structure.

[0065] In an embodiment, the system converts graphical data representations into a tabular format. The system automatically classifies the type of graph using a deep learning model, such as a Convolutional Neural Network (CNN) or a Vision Transformer (ViT), trained on a large dataset of various graph types, including bar charts, line graphs, scatter plots, box plots, heatmaps, survival curves (e.g., Kaplan-Meier plots), forest plots, flow charts, pie charts, and area plots. The model is fine-tuned for the life sciences and healthcare domain to improve classification accuracy. Additionally, the classification process may leverage information from the surrounding text, extracted by the neural network, to enhance the accuracy of graph type identification.

[0066] For axis and data point extraction, the system employs various computer vision techniques. Axis detection and scaling are performed using the Hough transform to identify axes, while OCR extracts axis labels and numerical values. An adaptive scaling detection algorithm determines the scaling function (e.g., linear or logarithmic) of each axis. Data point extraction relies on algorithms tailored to the specific graph type. For bar charts, connected component analysis identifies individual bars, with bar height or length measured relative to the axis scale. Line graphs use line detection and interpolation techniques to trace lines and extract data points. Scatter plots employ algorithms such as circle detection and blob detection to identify individual data points. For other graph types, specialized techniques are applied based on their unique structural characteristics.

[0067] To improve accuracy, the system incorporates several innovative techniques, including color segmentation to separate data points of different categories, shape recognition to differentiate data point markers, and interactive refinement by presenting extracted data points to the user for review and correction.

[0068] In an embodiment, the system extracts text from the graph legend and other labels using OCR. To accurately associate legend entries with data series, the system employs advanced layout analysis and natural language processing (NLP) techniques.

[0069] The system utilizes spatial reasoning by analyzing the spatial proximity of legend entries to data points or lines within the graph. For example, legend items that are closer to a particular line or set of data points are more likely to be associated with that series. Additionally, semantic similarity techniques compare the text of legend entries with axis labels, figure captions, and the surrounding text of the article. This involves using word embeddings or other semantic similarity measures to determine the relatedness of different text elements.

[0070] To further improve accuracy, the system leverages information from the neural network’s contextual embeddings to resolve ambiguities. For instance, if a legend entry is a single word like "Control," the neural network can analyze the broader context of the article and the specific experiment being described to determine its meaning.

[0071] In cases where the layout is complex or the text is ambiguous, the system may also utilize a large language model (LLM) to assist in legend and label extraction, ensuring more precise associations between legends, labels, and their corresponding data points.

[0072] In an embodiment, the system reconstructs tables by organizing extracted data from graphs, including data points, axis information, and legend entries, into a structured tabular format. The system is designed to handle complex graph types by structuring tables accordingly. For instance, a graph with multiple Y-axes might be reconstructed as atable with multiple columns or headers corresponding to each Y-axis. Additionally, metadata such as graph type, axis scaling functions, and extracted legend information are preserved alongside the tabular data. This preservation ensures accurate interpretation and allowsfor the potential reconstruction of the original graph if needed.

[0073] In an embodiment, the system estimates uncertainty for each extracted data element during the graph-to-table conversion process. Uncertainty can arise due to various factors such as low-resolution images, overlapping data points, or ambiguous labeling. The uncertainty estimation mechanism considers multiple factors, including the clarity of visual features in the graph, confidence scores from OCR or other extraction techniques, and the consistency of the extracted data with the overall context. In cases where the system encounters high uncertainty or conflicting information, an LLM can be queried for assistance. For example, if the system is unsure about the value of a data point due to overlappingelements, it can prompt the LLM to estimate the value based on surrounding data points, axis scales, and relevant contextual information extracted from the text.

[0074] In an embodiment, a neural network processes the extracted textual content of the document to generate contextual embeddings. These embeddings capture the semantic meaning of words and phrases within the document’s context. While various neural network architectures can be used, a transformer-based architecture is preferred for its ability to capture long-range dependencies and generate rich contextual representations. A transformerbased architecture such as a pre-trained BERT, RoBERT a, or a domain-specific model like BioBERT or SciBERT is employed and fine-tuned for the task. The network is trained on a vast corpus of scientific literature, potentially including millions of articles from diverse life sciences and healthcare domains. The training data is carefully curated to ensure it represents the target domain, incorporating various writing styles, journal formats, and research areas. Data sources may include PubMed Central, open-access journals, and other relevant scientific databases. Before training, the text undergoes preprocessing steps such as tokenization using specialized scientific text tokenizers, optional lowercasing, and potentially stemming or lemmatization. Numerical tokens may be normalized or replaced with special placeholders. The training objective involves a masked language model (MLM) or a similar approach that encourages learning rich contextual representations.

[0075] The system provides contextual understanding through several functions. One such function is ambiguity resolution, where the neural network helps resolve ambiguities in the extracted data by analyzing surrounding text. For instance, if the OCR process extracts the term "OR" from a table, the network can determine whether it stands for "odds ratio" or "operating room" based on context. If the text discusses a clinical trial and mentions terms like "treatment group," "control group," and "statistical significance," it is more likely that "OR" refers to "odds ratio." Similarly, if a graph legend contains "HR," the network can determine whether it means "hazard ratio" or "heart rate" by analyzing the surrounding text and identifying the study type, such as survival analysis or a clinical trial.

[0076] The system also enhances acronym and abbreviation expansion by identifying and expanding abbreviations based on their contextual usage within the document. Instead of relying solely on dictionary lookups, it considers the specific context in which an abbreviation appears. For example, it can differentiate between "HR" as "hazard ratio" in a clinical trial context and "heart rate" in a physiological study by referencing surrounding text, such as mentions of "survival analysis" or "cardiac output." Additionally, it can linkabbreviations to their expanded forms if they appear elsewhere in the document, even if the expanded form is not explicitly defined near the abbreviation's first occurrence.

[0077] In an embodiment, the system also performs relationship extraction toinfer connections between data elements across different tables or between tables and graphs. For example, if one table lists the baseline characteristics of patients in a clinical trial and another presents trial results, the network can identify references to both tables in the text and understand the relationships between the variables in each table, such as age, treatment group, and outcomes. The system may use these extracted relationships to construct aknowledge graph representing key findings from the study. This knowledge graph can then be leveraged to answer complex queries about the data or generate summaries of the study’s results.

[0078] The contextual embeddings generated by the network are not only used for internal analysis but are also made available to other modules in the system. Theseembeddings enhance performance in several areas, including table structure detection, where embeddings guide the algorithms in accurately identifying table components. In OCR andtext recognition, the embeddings assist in error correction and language model adaptation for the OCR engine. In graph-to-table conversion, they help classify graph types and interpret data points, ensuring higher accuracy in extraction. Lastly, in LLM-based data extraction, contextual embeddings provide crucial background information to the LLM, improvingits ability to understand extracted data and accurately respond to complex queries.

[0079] In an embodiment, the interpretation module is responsible for the second stage of data extraction, where it interprets the intermediate data representation produced by the derendering module and maps it to a user-defined output format. This module leverages a multimodal Large Language Model (LLM) and advanced prompt engineering to ensure accurate extraction and transformation of scientific data.

[0080] The LLM receives multiple inputs, including the intermediate data representation (such as tables in a standardized format) from the de-rendering module, images of the original tables and graphs, and relevant text segments identified using contextual embeddings generated by the neural network. Additionally, it incorporates the definition of the output data format (data extraction form), references already extracted data, such as names of outcomes or study arms, and utilizes a list of available parsing and transformation tools to ensure precise mapping of extracted information.

[0081] In an embodiment, carefully designed prompts are used to instruct the LLM to perform specific tasks such as mapping the intermediate data representation to a user-defined output format, resolving ambiguities in the extracted data, inferring relationships betweendata elements, answering user-defined queries, and generating explanations for its reasoning. These prompts guide the LLM to extract meaningful insights from scientific tables and graphs by aligning data with the surrounding context.

[0082] The system utilizes various example prompts to achieve accurate data extraction and interpretation. For instance, the LLM can be instructed to identify the column representing a treatment group and extract survival rates based on contextual text. It may also be asked to analyze bar charts displaying percentages of patients achieving a target systolic blood pressure or refine OCR outputs when confidence scores are low. Additionally, the model can compare and contrast data from different tables, evaluate dose-response relationships, assess patterns in adverse event data, and generate plain-language summaries of study findings for non-expert audiences.

[0083] By applying these structured prompts, the interpretation module ensures that scientific data is accurately extracted, logically interpreted, and mapped to structured formats in a way that aligns with user-defined requirements. This approach enhances data accessibility, facilitates complex analysis, and enables meaningful insights from scientific literature.

[0084] In an embodiment, the system employs various Large Language Models (LLMs), including large pre-trained models such as GPT-4, PaLM 2, LLaMA 2, and DeepSeek, or fine-tuned versions of these models. The choice of LLM and the fine-tuning strategy is dynamically adapted based on the specific task and document characteristics to optimize extraction and interpretation accuracy. The fine-tuning process involves training models on datasets of scientific publications, focusing on tasks related to data extraction and interpretation. The system dynamically selects the appropriate LLM and fine-tuning strategy depending on the document’s content and complexity.

[0085] To handle prompt length limitations, the system applies several techniques. Large tables or text segments are divided into smaller chunks to facilitate processing. The LLM may first analyze a summarized version and then refine its results iteratively by processing more detailed chunks. Additionally, the system identifies the most relevant text segments, ensuring that the LLM receives only the most critical information while staying within prompt length constraints.

[0086] In an embodiment, the system maintains a library of callable functions designed to perform common data transformations encountered in scientific literature. These functions handle domain-specific transformations, such as converting between different units of measurement, calculating risk ratios from odds ratios, and normalizing data to a commonscale. Researchers can also define and add their own custom transformation functions through the user interface, ensuring that the system can adapt to new or specialized transformations beyond the built-in functions.

[0087] When a required transformation is not available in the library, the system can leverage LLM-assisted code generation. The LLM receives a description of the transformation, input data examples, and the desired output format, and generates a Python function to perform the conversion. For example, if a user needs to convert a biochemical unit that is not covered by existing functions, the LLM can generate an appropriate code snippet. The user interface then displays the LLM-generated code for review and validation, allowing users to test and modify the code as needed. Users can provide feedback to improve future code generation, and once validated, the approved function is added to the callable functions library. Metadata such as the function’s purpose, author, and validation status is also stored for future reference.

[0088] The system ensures transparency and reproducibility by relying on explicitly defined and user-validated functions for data transformation. This contrasts with opaque, "black box" transformations that are commonly used in other systems, providing users with clear insights into how their data is processed.

[0089] In an embodiment, the LLM-based module works in concert with the traditional module, refining and validating its results. A feedback loop exists between the LLM, the traditional module, and the user interface. User corrections and validations are used to enhance prompt engineering, fine-tune the LLM, adjust parameters in the traditional module, and refine the system’s contextual understanding of documents. This iterative process ensures continuous improvement in data extraction accuracy and relevance.

[0090] In an embodiment, the postprocessing and data integration module handles the processing and integration of extracted data after it has been interpreted and mapped to the user-defined format by the interpretation module. This module performs key functions such as data normalization, validation, integration, transformation, and output generation.

[0091] Data normalization ensures that the extracted data is standardized into a consistent format. This includes unit conversion, where numerical values are converted into standard units, such as converting different pressure units to mmHg. Abbreviation expansion is performed using domain-specific dictionaries and contextual information from the neural network to ensure clarity. Additionally, data format standardization converts different formats into a uniform representation, such as standardizing date formats.

[0092] Data validation employs multiple techniques to detect and correct errors or inconsistencies in the extracted data. Heuristic rules based on domain knowledge flag values that fall outside expected ranges, such as physiological parameters with improbable values. Statistical methods analyze extracted data to identify outliers or inconsistencies, for instance, by calculating the standard deviation and flagging extreme values. External knowledge bases and ontologies, such as RxNorm for drug names, are leveraged to validate extracted data. The system may also cross-reference extracted values with the surrounding document text to ensure consistency.

[0093] Data integration combines extracted data from multiple tables and graphs within a document, linking related data elements and resolving inconsistencies. Relationships identified by the neural network during contextual analysis help facilitate integration. While basic transformations occur during the interpretation stage, this module performs more complex transformations requiring a broader view of extracted data. It utilizes a library of callable functions, including user-defined and LLM-generated code snippets, to execute transformations.

[0094] The module outputs processed data in structured formats suitable for further analysis or integration with external systems. Supported formats include Comma-Separated Values (CSV) files, JavaScript Object Notation (JSON) files, relational databases (e.g., SQL), and other structured data formats as required.

[0095] In an embodiment, the uncertainty handling module ensures the reliability and accuracy of extracted data by identifying, quantifying, and managing uncertainty throughout the data extraction process. Confidence scores are assigned to each extracted data element, reflecting the system’s certainty about its accuracy. These scores consider various factors, including agreement between different algorithms, clarity of visual features in tables and graphs, OCR confidence scores, contextual consistency determined by the neural network, and the LLM’s assessment of its own confidence.

[0096] When confidence scores fall below a predetermined threshold, the module flags those elements for further processing. Low-confidence data may be refined by prompting the LLM to re-evaluate the extraction using its reasoning capabilities and contextual information. Alternatively, the system may present flagged data for user review, allowing manual verification and correction through the user interface. Additionally, the module tracks and propagates uncertainty through the data extraction pipeline, ensuring that any calculations or transformations using low-confidence elements are also assigned appropriate confidence scores.

[0097] In an embodiment, the user interface module provides researchers with an interactive platform for reviewing, validating, refining, and controlling the data extraction process. As a critical component of the system’s iterative, two-stage approach, it allows users to provide feedback and adjustments at both the de-rendering and interpretation stages.

[0098] For data review and validation, the UI presents extracted data in an organized tabular format. Confidence scores are displayed alongside data elements, visually highlighting low-confidence values or potential errors using color-coded indicators. The UI includes direct links to the original tables or graphs in the scientific document, allowing users to verify extracted data against the source material. Figure 4 illustrates this process, showing the extracted data in a tabular format with confidence scores, editable cell contents for corrections, and side-by-side viewing of the original document for validation.

[0099] In an embodiment, the UI enables interactive refinement of data extraction. During the de-rendering stage, users can adjust detected table structures, merge or split cells, add or delete rows and columns, and correct header row identification. Users can also modify graph- to-table conversion parameters, such as axis scaling or data point extraction, and manually correct errors in legend associations. OCR corrections allow direct editing of text fields to fix recognition errors, while ambiguous text elements can be presented with alternative interpretations for user selection.

[0100] During the interpretation stage, users can refine mappings between the intermediate data representation and the user-defined output format, ensuring accurate transformations. They can also review and modify callable function parameters or provide their own code snippets for specialized transformations. Additionally, users can give feedback on LLM- generated interpretations, either confirming their accuracy or suggesting alternative insights.

[0101] In an embodiment, the mapping interface allows users to associate extracted data elements with concepts in standard ontologies or user-defined entities. Users can manually map table rows and columns to specific concepts or utilize automated system-generated suggestions. This feature is particularly useful for maintaining consistency across multiple studies and integrating extracted data into larger datasets. Figure 3 illustrates this interface, displaying a table alongside its extracted data, with dropdown menus and text fields for mapping, along with visual cues representing established relationships.

[0102] The user interface depicted in Figure 3 is designed to facilitate the mapping of table rows and columns to relevant entities or concepts within a scientific article. This interface is intuitive and interactive, enabling users to easily organize and interpret extracted data. The figure features a viewing pane 301 displaying a scientific article, or a portion of it, with aspecific table detected within the content. This table is highlighted 302 to draw the user's attention to the area where data mapping will take place.

[0103] Below the article, the extracted table data is presented in a table view 303, which shows the raw data as it was parsed from the article. This view provides a clear structure, allowing users to identify how the information is represented before any mapping adjustments are made. The mapping controls 304 are central to the functionality of the interface. These controls include dropdown menus or text fields 305 where users can either select or manually enter entities or concepts that correspond to the rows and columns of the table. This step is crucial for linking the raw data to meaningful, domain-specific entities, such as genes, proteins, diseases, or other scientific concepts.

[0104] Additionally, the interface provides users with options to modify the structure of the table itself. The controls allow for merging cells, splitting cell contents, or defining data types to ensure the extracted data aligns with the intended use. The system offers flexibility, enabling users to customize how the data is represented. A visual representation of the mapping 306 is also displayed, typically through color-coded highlighting or connecting lines, which visually link the table's rows and columns to the selected entities or concepts. This makes it easy for users to verify that the mappings are correct and to see how each piece of data is associated with the broader context.

[0105] To enhance user control and ensure accuracy, the interface includes several buttons for actions such as "Save Mapping," "Undo," and "Redo" 307. These buttons allow users to save their progress, revert any changes, or redo actions, providing a seamless and errorresistant workflow. This combination of elements makes the user interface a powerful tool for users to map and refine extracted data, ensuring that it is correctly interpreted and linked to the relevant scientific knowledge.

[0106] In accordance with an embodiment of the present invention, Figure 4 shows user interface showing the extracted data review and validation process, with confidence scores and options for user corrections.

[0107] Figure 4 illustrates a user interface designed for the extracted data review and validation process, offering users an interactive platform to assess, correct, and refine the data that has been extracted from scientific articles. At the core of this interface is a table view 401 that presents the extracted data, enabling users to clearly see how the information was parsed and represented. This table allows users to review the data systematically and assess whether the extraction process has accurately captured the content from the original document.

[0108] Each data element in the table is accompanied by a confidence score 402, which indicates the reliability of the extracted data. These confidence scores may be displayed as color-coded cells or numerical values, making it visually clear which elements have a high or low degree of certainty. For instance, data with high confidence might be shown in green, while data with lower confidence could be highlighted in yellow or red. This feature helps users quickly identify areas that may require further scrutiny or correction.

[0109] The interface offers several options for user corrections 403, providing a robust mechanism for improving the extracted data. Users can edit cell contents directly, allowing them to make precise modifications if any extracted data is inaccurate or incomplete. Additionally, users are presented with the option to accept or reject system suggestions, which may arise from the system’s automated analysis or proposed corrections based on learned patterns. In cases of ambiguous or unclear data, such as outputs from optical character recognition (OCR) that may be difficult to interpret, users can choose from alternative interpretations suggested by the system, providing flexibility in addressing inconsistencies. Moreover, the interface includes a feature for users to provide free-text feedback, enabling them to clarify, suggest improvements, or document observations for further refinement of the system.[HO] To facilitate thorough validation, the interface also offers a mechanism 404 that allows users to view the original table or graph alongside the extracted data. This side-by-side comparison provides users with a reference point to ensure that the extracted data aligns accurately with the source material, reducing errors and improving the overall quality of the data. In addition, navigation controls 405 such as "Next Table" and "Previous Table" enable users to move seamlessly between different tables or sections of the article. These controls make the review process more efficient, especially when working with large datasets or multiple pages of extracted information.[Hl] Further, if applicable, the system also includes a knowledge graph exploration feature within the UI. Users can visualize relationships between extracted entities, search for specific concepts, and query the knowledge graph to gain deeper insights into the extracted data. This enhances the system’s ability to provide structured knowledge representation and facilitate complex data analysis.

[0112] In accordance with an embodiment of the present invention, Figure 5 illustrates the structure of the knowledge graph, with nodes representing entities and edges representing relationships. Figure 5 provides a visual representation of the knowledge graph structure, illustrating how various entities and their interrelationships are organized within the system.At the core of this structure are the nodes 501, which represent different types of entities extracted from scientific data, such as genes, proteins, diseases, drugs, and biological pathways. These nodes are visually distinct, with each entity type being assigned a specific shape or color to aid in quick identification. For example, genes might be represented as circles, proteins as squares, diseases as diamonds, and pathways as triangles. This color- coding or use of shapes helps users easily differentiate between the various types of entities within the graph, improving the overall clarity and usability of the knowledge graph.

[0113] The edges 502 in the knowledge graph represent the relationships between the entities, highlighting how they are connected to one another. These relationships can take various forms, such as "encodes," "interacts with," "associated with," or "treats." Each relationship is depicted with a specific line style or color, further emphasizing the nature of the connection between the entities. For instance, a solid line may be used for direct associations (e.g., "Gene encodes Protein"), while a dashed line might represent more complex or indirect relationships (e.g., "Gene associated with Disease"). The different line styles or colors make it easy for users to interpret the type of interaction or relationship between any two entities at a glance.

[0114] In addition to the nodes and edges, labels 503 are used to clearly identify both the entities and the relationships within the graph. Each node is labeled with the name of the entity it represents, such as a specific gene or protein, while each edge is labeled with the relationship type, providing context to the connection. These labels ensure that the meaning of the nodes and edges is immediately understandable to users, eliminating ambiguity and facilitating a clearer comprehension of the knowledge graph's structure.

[0115] A key aspect of the UI module is its feedback loop, which continuously improves the system’s performance. All user interactions and corrections are captured and used to enhance various components, including the neural network in the de-rendering module, the LLM’s fine-tuning, the rule-based systems, and the callable function library. By incorporating iterative user feedback, the system evolves over time, improving its accuracy, adaptability, and overall effectiveness in extracting and interpreting scientific data.

[0116] In an embodiment, the knowledge graph construction and utilization module provides the capability to build and use a knowledge graph based on extracted data and relationships. This module enhances data organization, interpretation, and exploration by representing extracted entities and their interconnections in a structured graph format.

[0117] The system constructs the knowledge graph by extracting entities and relationships from processed documents. Entities include relevant scientific concepts such as genes,proteins, diseases, drugs, and pathways, as well as any other concepts identified by the system or defined by the user. These entities are represented as nodes within the graph. Relationships define the connections between these entities, such as "interacts with," "causes," "treats," or "is a," and are represented as edges. The graph is built using information from multiple sources, including extracted data from tables and graphs, textual content analyzed by the neural network in the de-rendering module and the LLM in the interpretation module, user input added via the user interface, and external knowledge bases that enrich the graph with additional scientific information.

[0118] In an embodiment, the knowledge graph is stored in a graph database, such as Neo4j, which is optimized for storing and querying graph- structured data. The database schema is designed to efficiently represent various entity and relationship types, ensuring smooth integration with the broader system.

[0119] The knowledge graph is leveraged to enhance multiple aspects of the system. It assists in ambiguity resolution by helping to disambiguate extracted terms or concepts. For instance, if the term "HR" appears in a document, the system can query the graph to determine whether it refers to "hazard ratio" or "heart rate" based on the surrounding entities and relationships. The graph also enables relationship inference by traversing connections and applying graph algorithms to infer indirect relationships, such as linking a gene to a disease through a shared biological pathway. Additionally, the knowledge graph provides contextual information to the LLM during the interpretation stage by retrieving relevant subgraphs and incorporating them into the model's input, thereby enhancing its understanding of extracted data.

[0120] The user interface module includes tools for exploring and querying the knowledge graph, allowing researchers to uncover new insights and connections within the extracted data. By structuring extracted knowledge in a graph format, the system enables more effective data interpretation, facilitates advanced queries, and enhances the overall research and analysis process.

[0121] In an embodiment, the knowledge graph is continuously updated with new information from processed documents and user feedback. The system incorporates mechanisms for updating the graph structure, adding new entities and relationships, and revising existing information based on new evidence.

[0122] The User Interface Module provides tools for interacting with the knowledge graph, as described in the above paragraphs. Figure 6 illustrates how users can explore the knowledge graph through the user interface.

[0123] The Dynamic Update Module is responsible for the continuous improvement and adaptation of the system. It manages the incorporation of user feedback, the retraining of machine learning models, and the updating of system components based on new data and evolving research needs.

[0124] In an embodiment, the module processes user feedback captured through the User Interface Module. Corrections, validations, and mapping decisions are analyzed to identify patterns and areas for improvement. The module also manages the retraining of the neural network in the De-rendering Module and the fine-tuning of the LLM in the Interpretation Module. User feedback provides valuable training data for this process.

[0125] Additionally, the module adjusts the rules in the rule-based systems of the Derendering Module based on user interactions. For example, if users frequently modify table boundaries detected by the system, the rules governing table structure detection are refined accordingly.

[0126] The module also oversees the dynamic updating of the knowledge graph, incorporating new entities and relationships extracted from newly processed documents, as well as any corrections or modifications provided by users.

[0127] In an embodiment, the knowledge graph is continuously updated with new information from processed documents and user feedback. The system incorporates mechanisms for updating the graph structure, adding new entities and relationships, and revising existing information based on new evidence.

[0128] The User Interface Module provides tools for interacting with the knowledge graph, as described in above paragraphs. Figure 6 illustrates how users can explore the knowledge graph through the user interface. This figure shows a user interface designed to facilitate the exploration of the knowledge graph, providing users with a powerful set of tools to search, visualize, and query the extracted knowledge. At the heart of this interface is the visualization of the knowledge graph 601, which displays the graph itself, or a selected portion of it, with clearly marked nodes and edges. The nodes represent various biological entities, such as genes, proteins, and diseases, while the edges depict the relationships between these entities. This graphical representation enables users to intuitively explore the structure and interconnections of the knowledge base, allowing them to visually comprehend the relationships and navigate through the vast amounts of data.

[0129] The search and query interface provides several functionalities to enhance user interaction with the graph. A prominent search bar 602 is included, allowing users to quickly find specific entities within the graph, such as particular genes, proteins, or pathways. Thisfeature is crucial for pinpointing relevant data in a large knowledge graph, improving efficiency in navigating the graph. Additionally, users can filter the nodes and edges displayed in the graph through filtering options 603, which allow them to sort or narrow down the displayed information based on entity type or other properties, such as the relationship type or the functional characteristics of the entities. This ensures that users can focus on the most relevant information based on their query.

[0130] The query builder 604 is another key feature, enabling users to construct more complex and refined queries to explore deeper connections within the knowledge graph. This tool provides an intuitive interface for creating queries that combine multiple conditions, such as finding all proteins that interact with a specific gene or exploring the pathways associated with a particular disease. The query builder simplifies the process of conducting advanced searches, allowing users to easily access the specific data they need without requiring advanced technical skills.

[0131] The Dynamic Update Module is responsible for the continuous improvement and adaptation of the system. It manages the incorporation of user feedback, the retraining of machine learning models, and the updating of system components based on new data and evolving research needs.

[0132] In an embodiment, this module processes user feedback captured through the User Interface Module. Corrections, validations, and mapping decisions are analyzed to identify patterns and areas for improvement. It also manages the retraining of the neural network in the De-rendering Module and the fine-tuning of the LLM in the Interpretation Module, leveraging user feedback as valuable training data for this process.

[0133] Additionally, the module adjusts the rules in the rule-based systems of the Derendering Module based on user interactions. For example, if users frequently modify table boundaries detected by the system, the rules governing table structure detection are refined accordingly.

[0134] The module also oversees the dynamic updating of the knowledge graph, incorporating new entities and relationships extracted from newly processed documents, as well as any corrections or modifications provided by users.

[0135] In an embodiment, the system employs the following methodology, with specific technical details provided for each step to ensure an enabling disclosure.

[0136] The overall data extraction process is further illustrated in the flowchart shown in Figure 2. The process begins with the input of a scientific document (step 201) and proceeds through the de-rendering stage (steps 202-205) and the interpretation stage (steps 206-208).The de-rendering stage involves table structure detection, text recognition, graph-to-table conversion, and contextual analysis using a neural network. The interpretation stage utilizes a large language model (LLM) to map the de-rendered data to a user-defined output format. The system also includes mechanisms for handling uncertainty (step 209), user feedback (step 210), and iterative improvement (step 211). An optional knowledge graph can be constructed and utilized to enhance the extraction process (steps 212-213).

[0137] In an embodiment, the system utilizes a hybrid approach for data extraction, combining traditional computer vision and NLP techniques with a Large Language Model (LLM)-based module. This approach leverages the strengths of both methodologies to achieve high accuracy and robustness. The process is divided into two primary stages: de- rendering and interpretation.

[0138] In an embodiment, the de-rendering stage focuses on converting the identified tables and graphs into a standardized, machine-readable intermediate representation. This involves a combination of computer vision techniques, deep learning models, rule-based systems, and a neural network for contextual analysis.

[0139] The system employs a combination of deep learning models and rule-based systems for accurate table structure detection. Deep learning models, such as ViTs and CNNs, are trained to identify table boundaries, rows, columns, and cells with high accuracy. These models can handle a wide variety of table formats, including those with complex layouts and nested structures. They are trained on large datasets of annotated scientific tables, such as PubTables- 1M and other domain-specific datasets. ViTs are particularly well-suited for this task due to their ability to capture global context through self-attention mechanisms.

[0140] In an embodiment, the table structure detection process incorporates contextual information from a neural network, which processes the document's textual content. This provides semantic information to guide the algorithms. For instance, if the text mentions a "table summarizing clinical trial results," the system can prioritize table structures commonly used in clinical trials and use this information to inform the selection of an optimal OCR strategy, leveraging the recognized accuracy advantages of Vision Transformers in processing complex visual layouts.

[0141] Following the initial deep learning-based detection, an adaptive rule-based system refines the results. This system incorporates domain-specific knowledge about table structures commonly found in life sciences and healthcare literature. The rules are dynamically adjusted based on:o User Feedback: The system learns from user corrections and adjustments made through the user interface. For instance, if users frequently merge cells that the system initially identified as separate, the system can learn to adjust its merging rules accordingly. o Contextual Clues: The rule-based system considers the table's context, as interpreted by the neural network. For example, if the surrounding text indicates that the table presents data from a specific type of study (e.g., a metaanalysis), the system can apply rules tailored to the typical table formats used in that type of study.

[0142] In an embodiment, the system may employ a hybrid approach, combining the outputs of the deep learning model and the rule-based system. An uncertainty estimation mechanism assigns a confidence score to each detected table component. Regions with high uncertainty are flagged for review by the user or further analyzed using the LLM-based module in the interpretation stage.

[0143] The system utilizes advanced OCR and text recognition techniques to extract text from the identified table cells. Recognizing the challenges of accurately extracting text from scientific tables, especially in the presence of low-quality scans, specialized terminology, and varied formatting, the following strategies are employed such as high-Accuracy OCR Engine with Domain Adaptation. The system employs a high-accuracy OCR engine capable of handling a variety of fonts, languages, and special characters commonly found in scientific literature. The specific OCR engine can be selected based on the characteristics of the document and the desired performance. Examples of suitable engines include, but are not limited to, Tesseract (especially versions 4.x or 5.x with LSTM models), Google Cloud Vision API, and Amazon Textract. To improve accuracy in the life sciences and healthcare domain, the OCR engine may be further enhanced through: o Fine-tuning on Scientific Text: The engine may be fine-tuned on a large corpus of scientific text, with a particular focus on life sciences and healthcare publications. This fine-tuning process helps the engine learn the specific characteristics of scientific writing, including common abbreviations, symbols, and formatting conventions. o Dynamic Language Model Integration: The OCR engine may integrate a dynamically adaptable language model. This language model is initially trained on a broad corpus of scientific literature but is then further customized based on document context and user feedback.ss Document Context: The neural network, which analyzes the document's textual context, provides information to the language model about the specific topic and terminology used in the article. This allows the language model to better predict and correct OCR output based on the expected vocabulary. s User Feedback: The language model is continuously updated based on user corrections and validations. For example, if users frequently correct a particular OCR error, the language model learns to adjust its predictions accordingly.

[0144] In an embodiment, the system leverages Vision Transformers (ViTs) to enhance text recognition accuracy, particularly in cases involving complex table structures or low-quality images. Recognizing the superior performance of Vision Transformers in various computer vision tasks, including OCR, the system may incorporate ViT-based models to optimize OCR strategies. The identified table structure or graph type informs the selection of an optimal OCR approach, utilizing the accuracy advantages of Vision Transformers in processing intricate visual layouts.

[0145] Preprocessing techniques are employed to improve OCR accuracy by enhancing image quality before text extraction. Adaptive binarization is applied instead of using a fixed threshold, converting cell images to black and white while accounting for local variations in illumination and contrast. Methods such as Otsu’s, Niblack, or Sauvola binarization may be used. Noise reduction is achieved through filters like Gaussian blur, median filtering, or nonlocal means denoising, with filter parameters chosen based on the characteristics of the scanned images. Deep learning models trained to remove noise from scanned documents can also serve as an alternative or complementary approach. Additionally, skew correction and deskewing techniques, such as Hough transform-based methods or projection profile analysis, ensure proper text alignment. Layout-specific corrections, such as removing lines that may interfere with character recognition but are part of the table structure, are also applied.

[0146] In an embodiment, postprocessing and error correction are performed to refine OCR output and improve text quality. Dictionary-based correction utilizes specialized dictionaries containing scientific terms, drug names, gene names, and other domain-specific vocabulary to identify and correct OCR errors. Contextual error correction is enabled by leveraging neural networks and the LLM to detect and rectify errors based on surrounding text. For instance, if the OCR output for a cell is “100 mg” with an incorrect “1” instead of “1,” the system can correct it by analyzing the context or querying the LLM with a structured prompt. Statisticallanguage models, such as n-gram models trained on scientific text, further enhance correction by identifying improbable word sequences or character combinations.

[0147] A hybrid OCR approach is employed, combining multiple OCR engines and utilizing an uncertainty estimation mechanism to assign confidence scores to extracted text. Low-confidence results are flagged for review or further analysis by the LLM. The system may dynamically select the most suitable OCR technique based on document characteristics or the detected table / graph structure.

[0148] The system converts graphical data representations into a tabular format through a graph-to-table conversion process. Additionally, it classifies the type of graph using a deep learning model, such as a CNN or ViT, trained on a large dataset covering various graph types, including bar charts, line graphs, scatter plots, box plots, heatmaps, survival curves (e.g., Kaplan-Meier plots), forest plots, flow charts, pie charts, and area plots. The model is fine-tuned specifically for the life sciences and healthcare domain and may leverage information from the surrounding text, extracted by the neural network, to enhance classification accuracy.

[0149] Axis and data point extraction is performed using computer vision techniques such as the Hough transform for axis detection. OCR is applied to extract axis labels and numerical values, while an adaptive scaling detection algorithm determines whether each axis follows a linear or logarithmic scale. Data points are extracted using specialized algorithms depending on the graph type. For bar charts, connected component analysis identifies individual bars, measuring their height or length relative to the axis scale. Line graphs are processed using line detection and interpolation techniques to trace lines and extract data points. Scatter plots employ circle detection or blob detection to identify data points. Additional graph types utilize specialized extraction techniques.

[0150] In an embodiment, advanced methods are used to improve data extraction accuracy. Color segmentation helps separate data points belonging to different categories, while shape recognition differentiates data point markers. An interactive refinement mechanism presents extracted data points to users for review and correction, ensuring higher accuracy.

[0151] Legend and label extraction is facilitated through OCR, with advanced layout analysis and NLP techniques employed to associate legend entries with data series. Spatial reasoning analyzes the proximity of legend entries to data points, while semantic similarity compares legend text with axis labels and surrounding text. The system also leverages neural network insights to resolve ambiguities in legend interpretation.

[0152] Extracted data is organized into a tabular format through table reconstruction. The system handles complex graph types by structuring tables appropriately, such as using multilevel headers for graphs with multiple Y-axes. Metadata, including graph type and axis scaling information, is preserved to maintain context.

[0153] In an embodiment, uncertainty handling is integrated with LLM-based assistance. The system estimates uncertainty levels for each extracted element and, in cases of high uncertainty or conflicting information, queries the LLM to refine or validate results.

[0154] In an embodiment, the interpretation stage focuses on mapping the intermediate data representation produced by the de-rendering stage to a user-defined output format and performing semantic interpretation.

[0155] In an embodiment, the interpretation stage focuses on mapping the intermediate data representation produced by the de-rendering stage to a user-defined output format and performing semantic interpretation. The LLM-based module refines the extracted data and handles complex cases, leveraging the advanced reasoning and language understanding capabilities of LLMs. The LLM is strategically integrated to complement the traditional module, maximizing strengths and mitigating weaknesses. It is employed for refining uncertain extractions by clarifying ambiguous outputs from the traditional module, handling complex queries that require reasoning and inference beyond the traditional module's capabilities, and validating relationships between entities while providing explanations.

[0156] The LLM receives multi-modal input, including images of tables and graphs, extracted data from the traditional module, and relevant text segments. Sophisticated prompt engineering techniques are used, specifically tailored to scientific literature in life sciences and healthcare. These prompts provide clear instructions, leverage domain knowledge, incorporate contextual information from the document as identified by the neural network in the de-rendering module, and elicit explanations for the LLM’s reasoning.

[0157] Below are some of the example prompts: s "Given this table and the statement in the text that 'patients treated with DrugX showed a statistically significant improvement in survival,' identify the column representing the group treated with Drug X and extract the survival rates for each time point. Explain your reasoning." s "This bar chart shows the percentage of patients achieving target systolic blood pressure. The text mentions that 'Drug X is a new treatment for hypertension.' Based on this information, identify the bar corresponding to thegroup treated with Drug X and extract its corresponding percentage. Justify your answer. " "The OCR output for this cell is '15.2 ± 2. S mmHg,' but the confidence score is low. The table presents data on systolic blood pressure changes from baseline in a clinical trial for Drug X. What is the most likely correct value, considering the surrounding values and the context provided by the Bi-LSTM, which indicates this is a study on a new drug for hypertension? Explain your reasoning." "This table presents data on the efficacy of different treatments for a specific disease. Based on the data in the table and the information extracted from the text, which treatment appears to be the most effective? Explain your reasoning, taking into account the sample sizes, p-values, and confidence intervals." "Compare and contrast the data presented in Table 2 (Treatment Group Assignment and Dosage) and Table 3 (Systolic Blood Pressure Changes from Baseline) of this clinical trial. What conclusions can you draw about the relationship between Drug X dosage and the change in systolic blood pressure? Provide evidence from the tables to support your conclusions." "Figure 1 shows the percentage of patients achieving target blood pressure in each treatment group, while Table 4 lists the adverse events reported in each group. Analyze the data presented in these two figures and discuss the tradeoffs between efficacy and safety for each treatment group. Cite specific data points from the figures to support your analysis." "Table 1 presents the baseline characteristics of patients in this study, while Figure 2 shows the change in a biomarker over time for each treatment group. Based on the data in Table 1 and Figure 2, are there any baseline characteristics that appear to be associated with the response to treatment as measured by the biomarker? Explain your reasoning and provide evidence from the data. " "Analyze the data presented in Figure 2, which shows the change in systolic blood pressure over time for each treatment group. Describe any trends or patterns you observe in the data, and discuss their potential implications for the efficacy of Drug X." "Examine the data in Table 3, which shows the changes in systolic blood pressure from baseline for different treatment groups. Is there a dose-responserelationship between Drug X and the reduction in systolic blood pressure? If so, describe the nature of this relationship and provide evidence from the table.""Review the adverse events data presented in Table 4. Are there any patterns or trends in the types or frequency of adverse events reported across the different treatment groups? What might these patterns suggest about the safety profile of Drug X?""Summarize the key findings of this study on the efficacy and safety of Drug X for treating hypertension, based on the data presented in Tables 1-4 and Figures 1-2. Write your summary in plain language, suitable for a patient with no medical background. Limit your summary to 200 words." s "Explain the results shown in Figure 2 (change in systolic blood pressure over time) to a non-expert audience. Describe the trends observed in the graph and their implications for patients with hypertension. Use simple terms and avoid technical jargon." s "Based on the data presented in this article, create a short paragraph explaining the main findings of this research for a general audience. Imagine you are writing a news article for a non-scientific publication. Highlight the most important results and their potential impact on the treatment of hypertension."

[0158] The system may utilize various LLMs, including large pre-trained models such as GPT-4, PaLM 2, and LLaMA 2, or fine-tuned versions tailored for specific tasks. The fine- tuning process involves training models on datasets of scientific publications with tasks relevant to data extraction and interpretation. An innovative approach is employed where the choice of LLM and fine-tuning strategy are dynamically adapted based on the task and document characteristics.

[0159] To handle prompt length limitations, the system implements several strategies. Large tables or text segments are divided into smaller chunks for processing. The LLM may first analyze a summarized version and then iteratively refine its understanding by processing more detailed chunks. Additionally, the system identifies the most relevant text chunks to ensure the LLM receives the most important information within the prompt length constraints.

[0160] The LLM-based module is integrated with the traditional module, refining and validating its results. A feedback loop exists between the LLM, the traditional module, and the user interface. User corrections and validations contribute to improving prompts, fine-tuning the LLM, adjusting parameters of the traditional module, and refining the contextual understanding of the document.

[0161] Contextual understanding is crucial for resolving ambiguities, inferring relationships, and ensuring the overall coherence of the extracted data. The system achieves this through a neural network in the de-rendering module and the advanced reasoning capabilities of the LLM in the interpretation module.

[0162] In an embodiment, the system employs a neural network, specifically within the derendering module, to process the entire textual content of the document. While various architectures can be used, a transformer-based architecture is preferred for its ability to capture long-range dependencies and generate rich contextual representations. This network generates contextual embeddings for each word, capturing its meaning in its specific context, considering the surrounding words and the overall meaning of the sentence and the document.

[0163] In an embodiment, a transformer-based architecture is employed for the system, trained on a vast corpus of scientific literature, potentially including millions of articles from diverse life sciences and healthcare domains. The training data is carefully curated to ensure it represents the target domain and encompasses a wide variety of writing styles, journal formats, and research areas. Sources for this training data could include PubMed Central, open-access journals, and other relevant scientific databases. Before training, the text undergoes preprocessing steps such as tokenization (using specialized tokenizers for scientific text), lowercasing (optional), and potentially stemming or lemmatization. Numerical tokens might be normalized or replaced with special tokens. The network is trained using a masked language model (MLM) objective or a similar approach that encourages the model to learn rich contextual representations.

[0164] The network provides key functions for contextual understanding, including ambiguity resolution. It helps resolve ambiguities in extracted data by analyzing the surrounding text. For example, if the OCR process extracts the term "OR" from a table, the network can determine whether it stands for "odds ratio" or "operating room" based on the context. If the text discusses a clinical trial and mentions terms like "treatment group," "control group," and "statistical significance," it is more likely that "OR" refers to "odds ratio." Similarly, if a graph legend contains "HR," the network can determine whether it stands for "hazard ratio" or "heart rate" by analyzing the surrounding text and identifying the type of study being described, such as survival analysis or a clinical trial.

[0165] A key feature of the system is its ability to identify and expand acronyms and abbreviations within the document. Unlike simple dictionary lookups, the model considers the context in which an abbreviation appears. For instance, "HR" can be interpreted as "hazard ratio" in a clinical trial context or "heart rate" in a physiological study based on nearby sentences mentioning "survival analysis" or "cardiac output." Additionally, if an expanded form of an abbreviation is mentioned elsewhere in the document, the network can link the abbreviation to it, even if it is not explicitly defined near the first occurrence.

[0166] Another major function is relationship extraction, where the network infers relationships between data elements in different tables or between tables and graphs. For example, if one table lists the baseline characteristics of patients in a clinical trial and another presents the trial results, the system can link these datasets by identifying references between them and understanding variable relationships (e.g., age, treatment group, outcome). The system may further leverage these extracted relationships to construct a knowledge graph that represents the study’s key findings. This knowledge graph can be used to answer complex queries and generate summaries of the study’s results.

[0167] The contextual embeddings generated by the network are not only used for internal analysis but are also available to other modules in the system. These embeddings provide valuable semantic information that enhances the performance of several key components. They assist in table structure detection by guiding table detection algorithms to identify table components more accurately. For OCR and text recognition, the embeddings help with error correction and language model adaptation for the OCR engine. In graph-to-table conversion, they assist in classifying graph types and interpreting data points. Additionally, for LLM- based data extraction, the embeddings provide crucial context, allowing the LLM to better understand the meaning of extracted data and to answer complex queries with higher accuracy.

[0168] The LLM module is employed for advanced reasoning and inference tasks that require a deeper understanding of a document’s context. Unlike operating in isolation, the LLM leverages the contextual understanding already developed by the network in the derendering module. The network’s output, including contextual embeddings and identified relationships, is provided as input to the LLM along with prompts. This integration enables the LLM to analyze information with a richer understanding of the document’ s context, leading to more accurate and nuanced interpretations. For example, when extracting data from a complex table, the LLM can utilize the network’s embeddings to quickly identify themost relevant text sections describing the table and its context. This allows the LLM to focus its analysis and provide more precise answers.

[0169] Beyond simple data extraction, the LLM is capable of answering complex, open- ended queries that require synthesizing information from multiple sources and applying domain-specific knowledge. For instance, a user might ask, "What are the main findings of this study regarding the efficacy of Drug X in treating Disease Y, considering the data presented in Table 3 and Figure 2, as well as the discussion in the text?" The LLM can integrate insights from different sections of the document to generate a comprehensive response. Additionally, the LLM is prompted to provide explanations and justifications for its answers, making its reasoning transparent and helping users understand the basis for its conclusions. For example, if asked to identify the most effective treatment in a table, the LLM might respond: "Based on the data in Table 2, Treatment A appears to be the most effective. It has the highest response rate (75%) and a statistically significant p-value (p<0.01) compared to the control group. This is consistent with the statement in the text that ‘Treatment A demonstrated superior efficacy in improving patient outcomes.’”

[0170] The system also supports continuous learning and adaptation, where both the neural network and the LLM are improved based on user feedback and new data. User feedback, such as corrections and validations provided through the interface, is used to fine-tune the network, refine the LLM’s prompts, and enhance the LLM’s responses through reinforcement learning or few-shot learning. For instance, if a user corrects the network’s interpretation of an abbreviation or its identification of a relationship between a table and a text section, the system adjusts accordingly to improve future accuracy. Similarly, if users frequently indicate that the LLM is misinterpreting a particular type of question, the corresponding prompts can be refined to guide the LLM toward better understanding. Additionally, when users repeatedly correct LLM responses, these corrections can be incorporated as examples for fine-tuning or included in prompts for few-shot learning. They may also serve as a reward signal for reinforcement learning, allowing the LLM to improve over time.

[0171] To stay up to date with evolving scientific knowledge, the system is designed to incorporate new research and adapt to changes in language and terminology within the life sciences and healthcare domains. The neural network can be periodically retrained on new scientific publications, ensuring it remains aligned with the latest research. Similarly, the LLM can be fine-tuned on new datasets or prompted with updated information, ensuring that its reasoning and contextual understanding remain relevant as scientific literature continues to evolve.

[0172] The disclosed system is designed for continuous improvement, learning from user feedback and new data to adapt to evolving research needs and maintain high accuracy.

[0173] The system captures and analyzes user interactions, including corrections, validations, mapping choices, and other feedback provided through the user interface. This feedback is crucial for identifying areas where the system can be improved.

[0174] In an embodiment, the neural network in the De-rendering Module is periodically retrained using accumulated user feedback and newly available scientific literature. This allows the network to improve its ability to generate accurate contextual embeddings, identify relationships, and adapt to changes in language and terminology.

[0175] The LLM in the Interpretation Module is fine-tuned using corrected outputs and user- approved responses. This process can involve techniques like reinforcement learning or few-shot learning, where the LLM learns from a limited number of examples provided by users or from the feedback signals it receives.

[0176] The parameters of the traditional data extraction modules (e.g., thresholds for table structure detection, weights in the rule-based systems) are adjusted based on user interactions. For instance, if users frequently modify the system's table structure detection results, the parameters governing cell merging or splitting can be adjusted to better align with user preferences.

[0177] The disclosed system is designed to incorporate new scientific publications on an ongoing basis. These new articles serve to expand the training data for the neural network and LLM, keeping the system up-to-date with the latest research findings, terminology, and data representation conventions.

[0178] In an example, the new data can also be used to identify emerging patterns and trends in scientific literature, allowing the system to adapt to new types of tables, graphs, and data reporting practices.

[0179] In an embodiment, the system's performance is continuously monitored using a variety of metrics to ensure its effectiveness and accuracy. These metrics include accuracy, which is the percentage of correctly extracted data elements; precision, which is the proportion of extracted data elements that are actually correct; recall, which is the proportion of actual data elements that are successfully extracted; and user satisfaction, which is measured through feedback forms, ratings, and other user interactions. These metrics are tracked over time, allowing for the identification of areas where the system is performing well and areas where improvement is needed. The gathered information is then used to guidethe iterative improvement process, prioritize development efforts, and ensure that the system continues to meet the needs of researchers.

[0180] In an embodiment, the system incorporates multiple mechanisms for detecting, handling, and reporting errors to ensure the reliability and accuracy of the extracted data.

[0181] As described in the above paragraphs, the system assigns confidence scoresto extracted data elements, reflecting the uncertainty associated with each element.

[0182] Data elements with low confidence scores are flagged for potential refinement by the LLM. The LLM can be prompted to re-evaluate the extraction, providealtemative interpretations, or request further information from the user.

[0183] The user interface allows researchers to review and correct any errors in the extracted data, including OCR errors, incorrect table structure interpretations, and inaccurate graph-to-table conversions.

[0184] Further, the system performs various consistency checks to identify potential errors, such as comparing extracted data with the surrounding text to ensure consistency, checking extracted data against external knowledge bases and ontologies to identify potential inconsistencies or errors. For example, flagging cases where numerical values in a table do not add up correctly or where extracted data contradicts information presented elsewhere in the document.

[0185] In an embodiment, the system logs all detected errors, including the type of error, the location of the error in the document, and any relevant contextual information. These logs can be used by developers to identify and fix bugs, improve algorithms, and monitor the overall performance of the system. Error reports can also be generated to provide users with insights into the reliability of the extracted data and to help them identify potential issues.

[0186] The disclosed system is designed for scalability, enabling it to handle large datasets and high-throughput data extraction tasks efficiently.

[0187] The system's modular design allows for individual components to be scaled independently based on specific needs. For instance, the computationally intensive deep learning models can be deployed on dedicated hardware with powerful GPUs, while other modules can be scaled using more general-purpose computing resources.

[0188] The disclosed system can be deployed on cloud platforms like AWS, Google Cloud, or Azure, leveraging their vast computing resources and scalability features. This allows the system to handle large-scale data extraction tasks by dynamically allocating resources as needed.

[0189] Further, the system utilizes parallel processing techniques to accelerate data extraction. For example, multiple documents can be processed simultaneously, and different parts of a single document (e.g., different tables or graphs) can be processed in parallel.

[0190] In an embodiment, the system employs efficient algorithms for tasks like table structure detection, graph-to-table conversion, and text recognition. These algorithms are optimized for performance and can handle large amounts of data without significant slowdowns. Further, the system uses asynchronous processing for tasks that may take longer to complete, such as LLM-based refinement. This allows the system to continue processing other data while waiting for the LLM to complete its analysis, improving overall throughput.

[0191] The system may employ caching mechanisms to store frequently accessed data, such as contextual embeddings or the results of common transformations. This reduces the need to recompute these values, improving performance. Furthermore, it uses appropriate database technologies (e.g., graph databases for the knowledge graph, relational databases for structured data) and optimizes database schemas and queries for efficient data storage and retrieval.

[0192] The following paragraphs provide details about the software and hardware components used to implement the system. The system is designed to be flexible and adaptable, allowing for the use of different software libraries, frameworks, and hardware configurations depending on specific needs and available resources.

[0193] The system is primarily implemented in Python, a widely used language for scientific computing, data analysis, and machine learning, thanks to its extensive ecosystem of libraries and frameworks. Key libraries used in the system include TensorFlow / Keras or PyTorch for deep learning tasks. These frameworks provide the foundation for building and training deep learning models, including those used in generating contextual embeddings and powering the table structure detection, OCR, graph-to-table conversion, named entity recognition (NER), and relationship extraction modules. The Hugging Face Transformers library is also utilized to access a wide range of pre-trained transformer models like BERT, RoBERTa, and others, which can be fine-tuned for specific tasks such as NER, relationship extraction, and contextual analysis.

[0194] For computer vision tasks, OpenCV is employed for image processing, such as manipulation, feature extraction, and object detection, particularly in the table structure detection and graph-to-table conversion modules. Additional specialized libraries like Mahotas, Scikit-image, or SimplelTK may be used for image segmentation, feature extraction, and image registration, depending on the specific algorithms being implemented.

[0195] The system also uses natural language processing (NLP) libraries such as SpaCy or NLTK for tasks like tokenization, part-of-speech tagging, dependency parsing, and named entity recognition. Gensim is used for topic modeling and document similarity analysis, which help in understanding the context of scientific articles. For numerical computation and data manipulation, NumPy and Pandas are employed, while Scikit-learn is used for general machine learning tasks like clustering, classification, and regression. The Requests library is used to make HTTP requests to external APIs, such as those for LLMs or knowledge bases.

[0196] Regarding large language models (LLMs), the system is designed to leverage various models for the refinement module. OpenAI models such as GPT-3 and GPT-4 can be accessed through the OpenAI API, while Google’s PaLM 2 and Meta’s LLaMA 2 models can be accessed through Google Cloud's Vertex Al or run locally or on cloud platforms, respectively. The system also supports DeepSeek models, including DeepSeek-Coder and DeepSeek-LLM, which are strong in code generation and text understanding tasks. These models can be deployed on either local hardware or cloud platforms. The system is flexible and can incorporate other open-source or proprietary models as needed, with fine-tuning performed on datasets from scientific literature in the life sciences and healthcare domains to improve their performance on domain-specific tasks.

[0197] The system utilizes various databases and knowledge bases. Relational databases like PostgreSQL or MySQL are used to store extracted data, user feedback, and other structured information. Graph databases such as Neo4j are employed for representing and querying the relationships between entities extracted from scientific literature. The system also integrates biomedical ontologies and knowledge bases like MeSH, SNOMED CT, RxNorm, NCBI Gene, and DrugBank, accessed through APIs or local copies, for tasks such as entity linking, relationship extraction, and semantic interpretation.

[0198] Finally, the user interface is built using web technologies, including HTML, CSS, and JavaScript, with potential use of frameworks like React, Angular, or Vue.js. The backend of the user interface can be implemented using a Python framework such as Flask or Django, which interacts with the core data extraction and analysis modules.

[0199] In an embodiment, the system can be deployed on a variety of hardware configurations, depending on the scale of data being processed and the desired performance. For most processing tasks, modern multi-core CPUs, such as Intel Xeon or AMD EPYC, are used. GPUs, including models like the NVIDIA Tesla, RTX, or A-series, are essential for accelerating deep learning computations. These are particularly important for training andinference with the neural network and large language models (LLMs), as well as for computer vision tasks when deep learning-based methods are employed.

[0200] The system requires sufficient RAM to store the data being processed and the model parameters, with specific requirements depending on the size of the models and datasets used. For example, at least 64 GB of RAM may be necessary for a production-level system, and when using large LLMs, the system might require 128 GB or more. Fast storage, such as SSDs or NVMe drives, is recommended to store the system's data, models, and temporary files, ensuring quick data retrieval and efficient operation.

[0201] In terms of deployment, the system can be deployed on cloud platforms such as AWS, Google Cloud, or Azure, leveraging their compute, storage, and database services. This enables scalability and flexibility to handle large datasets and fluctuating workloads, making it easier to scale the system as needed. For deployment, the system canbe implemented as a standalone application or as a web service accessible through an API.

[0202] To streamline deployment and management across diff erentenvironments, containerization technologies like Docker can be used to package the system's components and their dependencies. Orchestration tools like Kubemetes can further enhance deployment, enabling effective orchestration and scaling of the system in cloud or cluster environments.

[0203] The following describes example embodiments of a system and method designed to automatically extract data from tables and graphs in scientific literature, particularly focusing on the life sciences and healthcare domains. These embodiments are meant to illustrate how the invention can be implemented in practice, with variations considered within its scope. This approach involves using sophisticated technology to extract structured datafrom scientific documents, and its application is not limited to the examples provided.

[0204] One embodiment demonstrates the system's application to a scientific article that discusses a clinical trial for a new drug, "Drug X," which is intended fortreating hypertension. The article includes several tables and graphs presenting data related to patient demographics, treatment groups, efficacy outcomes, and adverse events. The system is deployed on a cloud server with powerful hardware specifications, such as an IntelXeon Platinum 8370C CPU, an NVIDIA A100 GPU, 128 GB of RAM, and 1 TB of NVMe SSD storage. It uses an array of software tools, including Python 3.9, TensorFlow 2.x, PyTorch 1.13, Hugging Face Transformers, OpenCV 4.5, SpaCy 3.4, NLTK 3.7, Neo4j 4.x, anda finetuned LLaMA 2 70B large language model (LLM), among others, to process the data. The system begins by receiving a PDF file of the article titled "Efficacy and Safety of Drug X in Patients with Hypertension: A Randomized Controlled Trial," which includes multipletables and figures, such as data on baseline patient characteristics, treatment group assignment, changes in systolic blood pressure, and adverse events.

[0205] The system first processes the input through a preprocessing module, which uses OCR and text extraction techniques to convert the document into machine-readable text. It then uses computer vision algorithms, including a Faster R-CNN model trained on PubTables-lM and a rule-based system, to identify and isolate the tables and graphs within the document. For example, in the case of Table 3, which presents systolic blood pressure changes from baseline, the table structure detection module uses a deep learning model, such as a fine-tuned Vision Transformer (ViT) or CNN model, to identify table boundaries, rows, columns, and cell boundaries. This model takes into account contextual information from the surrounding text, which helps identify key features such as "systolic blood pressure," "treatment group," and "change from baseline." The system further refines this detection using a rule-based system that leverages domain-specific knowledge of clinical trial tables.

[0206] Next, the text recognition pipeline performs OCR on each cell of the table, with techniques designed for scientific text. The system may use various OCR engines, such as Tesseract or Google Cloud Vision API, depending on the document's characteristics. Postprocessing steps are employed to correct errors in the OCR output using dictionary -based corrections, contextual error correction, and domain-specific knowledge. For instance, if the OCR misreads "120 / 80 mmHg" as "120 / 80 mmHg," the system uses the surrounding context to correct this. Additionally, the normalization component standardizes units (e.g., converting blood pressure units to mmHg) and expands abbreviations (e.g., converting "DBP" to "diastolic blood pressure").

[0207] For the graph data, the system processes Figure 2, which shows the change in systolic blood pressure over time. The graph is classified as a line graph using a CNN or ViT model fine-tuned on a dataset of scientific graphs. The system extracts the axes and data points from the graph by employing the Hough transform to detect the axes and using OCR to extract axis labels. It then applies an adaptive scaling detection algorithm to determine the linear scale of the axes. The system extracts data points from the graph by tracing the lines representing each treatment group, and it associates the legend entries with the corresponding lines in the graph. The extracted data is then organized into a table with relevant columns, such as "Time (weeks)," "Treatment Group," and "Change in Systolic Blood Pressure (mmHg)."

[0208] The context-aware data interpretation component analyzes the extracted data alongside the full text of the article. It resolves ambiguities, such as distinguishing between"HR" as "hazard ratio" (rather than "heart rate") in the context of adverse events. The system identifies relationships between various data elements, such as linking the patient data in Table 1 with the corresponding data in other tables and figures. It constructs a knowledge graph, representing entities like "Drug X" and "hypertension" as nodes, with relationships such as "treats" and "has value" represented as edges. This knowledge graph is stored in a Neo4j database.

[0209] The interpretation module further refines the extracted data using the LLM to handle complex queries. For example, users can ask the system questions such as "What is the average reduction in systolic blood pressure for patients in the high-dose group of Drug X after 12 weeks of treatment?" The system leverages the extracted data and the LLM’s reasoning abilities to answer the query. Additionally, the system can handle uncertainty by prompting the LLM to provide more accurate interpretations for ambiguous data. For instance, if an OCR error causes a cell to be incorrectly read, the system might ask the LLM to determine the most likely correct value based on context and typical values in clinical trials.

[0210] The user interface module presents the extracted data in a web-based interface, allowing users to review and validate the extracted information. Users can view the original tables and graphs alongside the extracted data, and the system provides confidence scores for each data element. Users can also interact with the system to correct any errors, adjust the table structure, or refine the graph-to-table conversion results. The interface also allows users to map extracted data elements to standard ontologies, such as mapping "Drug X" to its RxNorm identifier. Additionally, users can explore the knowledge graph, visualizing relationships between different entities and concepts. Feedback from users is captured to retrain the neural network, fine-tune the LLM, and improve the rule-based systems used in the data extraction module. Finally, the system outputs the validated and refined data in a structured format, such as CSV or JSON, which can be imported into other data analysis tools, along with the knowledge graph in formats like GraphML.

[0211] In another example embodiment, the system’s application to a scientific article presenting a meta-analysis of multiple studies that evaluate the effectiveness of a particular intervention for treating a specific disease. The article includes a forest plot summarizing the results of the individual studies and a table listing the characteristics of each included study. This embodiment utilizes a system setup similar to the one described in the above embodiment, with adjustments made to accommodate the specific structure of the meta- analysis data.

[0212] The input data for this embodiment is a PDF file of a scientific article titled "Effectiveness of Cognitive Behavioral Therapy for Depression: A Meta- Analysis," which has been published in a peer-reviewed journal. The article contains Table 1, detailing the characteristics of the included studies, and Figure 1, a forest plot illustrating the effect size and confidence intervals for each study, as well as the overall pooled effect size. The system's preprocessing module extracts the textual content from the article and identifies Table 1 and Figure 1 for further analysis.

[0213] The table structure detection component plays a crucial role in identifying the structure of Table 1. It recognizes columns such as study name, publication year, sample size, intervention type, control group, and outcome measure. Following this, the text recognition pipeline extracts the text from each cell in Table 1, applying domain-specific dictionaries to handle variations in formatting and normalize the data. For example, different outcome measure scales are converted to a common scale to ensure consistency in the extracted information.

[0214] The system then moves on to processing Figure 1, the forest plot. The de-rendering module classifies the figure as a forest plot, which allows the system to apply appropriate graph-to-table conversion techniques. The axis extraction and scaling component identifies the X-axis, which represents the effect size, and extracts the numerical values associated with it. The adaptive scaling detection algorithm is used to determine the scale of the axis, ensuring accurate extraction of the data. The data point extraction component identifies individual study data points, often represented as squares or diamonds, and the horizontal lines that indicate the confidence intervals. The system extracts the effect size and confidence interval for each study and the overall pooled effect estimate. If a legend is present in the figure, the system processes the legend entries using OCR and associates them with the corresponding data points.

[0215] Next, the system organizes the extracted data into a table with columns for "Study," "Effect Size," "Lower Confidence Interval," and "Upper Confidence Interval." The context- aware data interpretation component then analyzes this data in conjunction with the full text of the article. The neural network helps understand the context of the meta-analysis, including the inclusion and exclusion criteria for the studies, the type of intervention being evaluated, and the outcome measures used. The system also identifies relationships between the studies listed in Table 1 and the data points extracted from Figure 1, ensuring that the data is connected appropriately across both components.

[0216] The interpretation module, which uses the large language model (LLM), refines the extracted data and handles complex queries. For example, if there is uncertainty about the extracted effect size or confidence interval for a particular study, the LLM can be prompted to provide a more accurate estimate based on the surrounding text and the data from other studies. The system can also answer complex queries, such as identifying which studies showed a statistically significant effect of the intervention or determining the overall pooled effect size and its statistical significance.

[0217] The user interface module presents the extracted data in a reviewable format, allowing users to validate and refine the results. Users can correct any errors in the extracted data, adjust the structure of the table, or modify the graph-to-table conversion results. They can also interact with the knowledge graph to explore the relationships between different studies, interventions, and outcomes. Finally, the system outputs the validated and refined data in a structured format, such as CSV or JSON, ensuring that the extracted data can be easily imported into other data analysis tools for further use.

[0218] In another example embodiment, the system constructs and utilizes a knowledge graph to enhance data extraction and facilitate knowledge discovery across multiple scientific articles. This embodiment focuses on extracting data from a set of articles related to the genetic basis of Alzheimer's disease. The articles contain valuable information about genes, proteins, pathways, and their relationships to the disease, presented in both text and tables. The system setup for this embodiment is similar to an embodiment described above, with a particular emphasis on using a graph database, such as Neo4j, for storing and querying the knowledge graph.

[0219] The data input consists of a collection of ten PDF files of scientific articles related to Alzheimer's disease genetics, retrieved from PubMed and other relevant databases. These articles include tables listing genes associated with Alzheimer's disease, tables describing protein-protein interactions, textual descriptions of biological pathways implicated in the disease, and figures illustrating biological pathways or networks. The preprocessing module extracts the textual content from all ten articles using OCR and other text extraction techniques, while tables and figures are identified within each document using computer vision algorithms.

[0220] For data extraction and contextual analysis, the de-rendering module processes the identified tables and figures by performing table structure detection, text recognition, and graph-to-table conversion, as described in previous embodiments. The context-aware data interpretation component, powered by the neural network, analyzes the extracted data inconjunction with the full text of each article, refining the data and extracting additional information, such as descriptions of relationships between genes, proteins, and pathways. This refined data is then used for constructing the knowledge graph.

[0221] The knowledge graph construction module processes the extracted data and the relationships identified by the neural network and large language model (LLM). Entities such as genes (e.g., APOE, APP, PSEN1), proteins (e.g., Amyloid-beta, Tau), pathways (e.g., Amyloidogenic pathway, Tau phosphorylation), diseases (e.g., Alzheimer's disease), phenotypes (e.g., cognitive decline, memory impairment), and other relevant concepts (e.g., mutations, risk factors) are extracted from the tables, figures, and text. Relationships between these entities are also identified, such as "Gene encodes Protein" (e.g., APP encodes Amyloid-beta), "Protein interacts with Protein" (e.g., Amyloid-beta interacts with Tau), and "Gene associated with Disease" (e.g., APOE associated with Alzheimer's disease). The knowledge graph is constructed with nodes representing the extracted entities and edges representing the relationships between them. These nodes and edges are stored in aNeo4j graph database, where they can be queried and analyzed further.

[0222] The knowledge graph utilization module helps resolve ambiguities by leveraging the graph. For example, if the OCR process extracts the term "APP" from a table, the system queries the knowledge graph to determine whether it refers to the "amyloid precursor protein" or another entity with the same acronym, using the context provided by the relationships in the graph to disambiguate the term. The system also uses graph algorithms to infer new relationships, such as identifying indirect connections between entities. Pathfinding allows the system to traverse the graph and infer links between a gene and a disease through an involved pathway, while graph embeddings capture the structural and semantic relationships between entities, predicting new links or filling in missing information.

[0223] When the LLM is used to refine data extraction or answer complex queries, the knowledge graph utilization module retrieves relevant subgraphs from the knowledge graph to provide contextual information. For instance, if a user asks about the role of the Tau protein in Alzheimer's disease, the system retrieves a subgraph containing nodes related to Tau protein and Alzheimer's disease and their connecting paths, which is then provided as context to the LLM. The user interface is enhanced to allow users to interact with the knowledge graph, enabling them to visualize the graph or specific subgraphs, search for specific entities or relationships, and construct complex queries using a query builder. Tools for exploration are also provided, allowing users to browse the graph starting from any entity and examine its connections.

[0224] Iterative improvement is facilitated by user feedback on the extracted data, which is used to refine the knowledge graph. New scientific articles are regularly processed and added to the knowledge graph, ensuring its coverage remains up-to-date. The neural network and LLM models are periodically retrained using the growing knowledge graph and user feedback. The system outputs the extracted data, enhanced by the knowledge graph, in structured formats such as CSV or JSON. The knowledge graph itself can be exported in various formats (e.g., GraphML) for further analysis or integration with other systems.

[0225] The system's use of a knowledge graph provides several benefits, including enhanced accuracy in data extraction by resolving ambiguities and providing contextual information. It also enables the discovery of new relationships and insights by inferring connections between entities across multiple articles, offering a more comprehensive and interconnected understanding of the data. This facilitates accelerated research by integrating data from multiple sources, making it easier for researchers to build upon existing knowledge.

[0226] Further, the system can be adapted and extended to handle a variety of scientific documents and data extraction tasks across different fields. It can be used for extracting data from tables and graphs in supplementary materials, and for handling different types of graphs, such as pie charts, area plots, and network diagrams. The system is also capable of extracting data from tables with complex hierarchical structures, and it can be integrated with other data analysis and visualization tools. Moreover, the system can build and query domain-specific knowledge graphs, making it adaptable to various scientific domains beyond life sciences and healthcare. This adaptability is achieved by retraining the models on relevant corpora and integrating domain-specific ontologies.

[0227] The system can also incorporate natural language generation capabilities, enabling it to automatically generate textual summaries or reports based on the extracted data. Specialized modules can be developed for specific tasks, such as extracting data from clinical trial registries, analyzing gene expression data, or identifying drug-drug interactions. The system supports real-time data extraction from newly published articles, ensuring researchers remain up-to-date with the latest findings. Additionally, a platform for collaborative data curation can be created, allowing researchers to collectively contribute to the knowledge graph and improve the accuracy of the extracted data.

[0228] These example embodiments illustrate how the invention can be applied to real- world scenarios, particularly in the life sciences and healthcare domains. While the specific algorithms, parameters, and configurations may vary depending on the application and thenature of the data being processed, the core principles of the invention remain consistent. These principles include the hybrid approach combining traditional methods with LLM-based techniques, the two-stage architecture that includes de-rendering and interpretation modules, the use of a neural network for deep contextual understanding, an emphasis on domain specificity, advanced prompt engineering for the LLM, and iterative improvement through user feedback.

[0229] The present invention provides significant advantages over prior art methods for data extraction from scientific literature, particularly in the life sciences and healthcare domains. These advantages stem from the innovative combination of techniques, the strategic use of advanced Al models, the system's adaptability through continuous learning, and its robust error handling and scalability.• High Accuracy through Hybrid Approach: The synergistic integration of traditional computer visi on and NLP techniques with a strategically deployed LLM-based data refinement module results in significantly higher accuracy than either approach could achieve alone. The traditional module provides a robust foundation for initial data extraction, while the LLM enhances precision, resolves ambiguities, and handles complex cases that are challenging for rule-based or purely statistical methods. This contrasts with the limitations of prior art, which often relied on only one of these approaches. s Deep Contextual Understanding: Unlike many existing systems that struggle with the nuances of scientific text, the present invention achieves a deep understanding of context through the use of a neural network and advanced prompt engineering for the LLM. The network generates rich contextual embeddings that capture the meaning of words and phrases within the document, while the LLM leverages this context to perform complex reasoning and inference tasks. This allows the system to accurately interpret ambiguous terms, resolve inconsistencies, and understand the relationships between data elements in tables, graphs, and the surrounding text, outperforming prior art.• Adaptability to Diverse Formats: The system's flexible architecture, incorporating adaptable algorithms for table structure detection, graph-to-table conversion, and text recognition, allows it to handle the wide variety of table and graph formats encountered in scientific literature. The use of deep learning models, such as CNNs or Vision Transformers, for object detection and graph type classification, combined with adaptive rule-based systems and user feedback, enables the system to adapt tonew and unusual formats without requiring manual reprogramming. This contrasts with prior art that often relied on brittle, template-based approaches.* Adaptability to Technological Advancements: The system's modular design and flexible architecture allow it to readily incorporate new and improved OCR technologies, such as those based on Vision Transformers, as they become available and new LLM models like the new DeepSeek LLM model. This ensures that the system remains at the forefront of data extraction technology and maintains its high accuracy over time.* Domain Specificity: The system is specifically optimized for the complexities of life sciences and healthcare data through: o Fine-tuning of deep learning models (neural network, LLM, and others) on large corpora of scientific text from these domains. o Integration of domain-specific ontologies and knowledge bases (e.g., MeSH, SNOMED CT, RxNorm, NCBI Gene). o Advanced prompt engineering that incorporates domain knowledge and terminology. o Tailoring of algorithms (e.g., NER, relationship extraction) to the specific entities, relationships, and data representation conventions common in these fields.* This contrasts with many prior art data extraction systems that were designed for general-purpose applications and struggle with the specialized language and data formats of scientific publications.8Transparency and Customizability: The use of user-defined and LLM-generated code snippets for data transformation provides transparency and allows researchers to customize the system to their specific needs. This contrasts with prior art systems that rely on opaque, hard-coded transformations.* Continuous Improvement: The system's ability to learn from user feedback and incorporate new data represents a significant advancement. The iterative refinement process, which involves retraining the neural network, fine-tuning the LLM, and adjusting the parameters of the traditional module, ensures that the system continuously improves its accuracy, robustness, and adaptability over time. This contrasts with prior art systems that often had fixed parameters and were unable to adapt to new data or evolving user needs.* Robust Error Handling: The system incorporates multiple error handling mechanisms, including uncertainty estimation, LLM-based refinement, user-guided correction,consistency checks, and domain-specific validation. This multi-faceted approach ensures the reliability and accuracy of the extracted data, even in cases of noisy images, ambiguous text, or complex data relationships. The ability to automatically detect and handle errors is a significant improvement over prior art systems that often produced inaccurate or incomplete results without indicating the level of confidence or providing mechanisms for correction.* Efficiency and Scalability: By automating a traditionally manual and time-consuming process, the system significantly reduces the time and effort required for researchers to extract and analyze data from scientific literature. The system's modular design, coupled with its ability to be deployed on cloud platforms and leverage powerful hardware (e.g., GPUs), makes it scalable to large datasets and high-throughput applications. The use of efficient algorithms, parallel processing, asynchronous operations, database optimization and caching further enhances performance, enabling the system to handle the ever-growing volume of scientific publications. This addresses a major bottleneck in scientific research and facilitates more rapid knowledge discovery.* Enhanced Transparency and Trust: The LLM's ability to provide explanations for its reasoning, prompted by specific instructions, enhances the transparency and trustworthiness of the system's outputs. This is particularly important in scientific applications, where understanding the basis for conclusions is crucial. This contrasts with "black box" models that do not offer explanations for their results.* Multi-Modal Understanding: By processing both images (tables and graphs) and text, and intelligently integrating information from these different modalities, the system achieves a more comprehensive understanding of the scientific document than text- only or image-only approaches. The use of multi-modal input to the LLM, guided by the contextual analysis, is a key innovation that contributes to this advantage.8Knowledge Graph Construction and Utilization: The system's ability to construct a knowledge graph from extracted data and relationships enables advanced reasoning, inference, and enhanced data exploration through a dedicated user interface. This facilitates a deeper understanding of the interconnected nature of scientific knowledge and supports new forms of discovery that go beyond the capabilities of traditional data extraction systems.

[0230] The present invention provides a significant advancement in the field of automated data extraction from scientific literature. By combining traditional computer vision and NLPtechniques with advanced prompt engineering for multi-modal LLMs, and by incorporating a user interface for validation and refinement, the invention achieves high accuracy and efficiency in extracting data from tables and graphs, particularly in the complex domain of life sciences and healthcare. The core innovation of the two-stage, iterative approach with user refinement allows researchers to guide the extraction process, ensuring the accuracy and relevance of the extracted data. The system's ability to create a knowledge graph from extracted data allows for advanced reasoning, inference, and enhanced data exploration. This technology has the potential to greatly accelerate research and discovery by automating a tedious and error-prone manual process, enabling researchers to focus on higher-level analysis and interpretation of scientific findings. Furthermore, by enabling more efficient data extraction and analysis, this invention can contribute to accelerating the pace of scientific discovery and ultimately improving healthcare outcomes. The system's ability to learn from user feedback and continuously improve its performance ensures its long-term effectiveness and adaptability to evolving data formats and research needs. This invention will be a valuable tool for researchers, clinicians, and other professionals who rely on data from scientific publications in the life sciences and healthcare fields and beyond. While the detailed description emphasizes the application of the invention to life sciences and healthcare, the system can be adapted and applied to other domains of scientific literature where data extraction from tables and graphs is required.

[0231] Terms and phrases used in this document, and variations thereof, unless otherwise expressly stated, should be construed as open-ended as opposed to limiting. As examples of the foregoing: the term “including” should be read as mean “including, without limitation” or the like; the term “example” is used to provide exemplary instances of the item in the discussion, not an exhaustive or limiting list thereof; and adjectives such as “conventional,” “traditional,” “standard,” “known” and terms of similar meaning should not be construed as limiting the item described to a given time period or to an item available as of a given time, but instead should be read to encompass conventional, traditional, normal, or standard technologies that may be available or known now or at any time in the future. Likewise, a group of items linked with the conjunction “and” should not be read as requiring that each and every one of those items be present in the grouping, but rather should be read as “and / or” unless expressly stated otherwise. Similarly, a group of items linked with the conjunction “or” should not be read as requiring mutual exclusivity among that group, but rather should also be read as “and / or” unless expressly stated otherwise. Furthermore, although item, elements or components of the disclosure may be described or claimed in the singular, theplural is contemplated to be within the scope thereof unless limitation to the singular is explicitly stated. The presence of broadening words and phrases such as “one or more,” “at least,” “but not limited to” or other like phrases in some instances shall not be read to mean that the narrower case is intended or required in instances where such broadening phrases may be absent.

[0232] For the purposes of this specification and appended claims, unless otherwise indicated, all numbers expressing amounts, sizes, dimensions, proportions, shapes, formulations, parameters, percentages, quantities, characteristics, and other numerical values used in the specification and claims, are to be understood as being modified in all instances by the term “about” even though the term “about” may not expressly appear with the value, amount, or range. Accordingly, unless indicated to the contrary, the numerical parameters set forth in the following specification and attached claims are not and need not be exact, but may be approximate and / or larger or smaller as desired, reflecting tolerances, conversion factors, rounding off, measurement error and the like, and other factors known to those of skill in the art depending on the desired properties sought to be obtained by the subject matter of the present invention..

[0233] Further, the term “about” when used in connection with one or more numbers or numerical ranges, should be understood to refer to all such numbers, including all numbers in a range and modifies that range by extending the boundaries above and below the numerical values set forth. The recitation of numerical ranges by endpoints includes all numbers, e.g., whole integers, including fractions thereof, subsumed within that range (for example, the recitation of 1 to 5 includes 1, 2, 3, 4, and 5, as well as fractions thereof, e.g., 1.5, 2.25, 3.75, 4.1, and the like) and any range within that range.

[0234] All publications, patent applications, patents, and other references mentioned in the specification are indicative of the level of those skilled in the art to which the presently disclosed subject matter pertains. All publications, patent applications, patents, and other references are herein incorporated by reference to the same extent as if each individual publication, patent application, patent, and other reference was specifically and individually indicated to be incorporated by reference. It wifi be understood that, although a number of patent applications, patents, and other references are referred to herein, such reference does not constitute an admission that any of these documents forms part of the common general knowledge in the art. Although the foregoing subject matter has been described in some detail by way of illustration and example for purposes of clarity of understanding, it will beunderstood by those skilled in the art that certain changes and modifications can be practiced within the scope of the appended claims.

Claims

CLAIMSWhat is claimed is:

1. A method for automatically extracting data from tables and graphs in scientific literature, the method comprising: o receiving a scientific document containing at least one table or graph; o applying a two-stage data extraction process, including: s a de-rendering stage that converts the at least one table or graph into a standardized intermediate data representation, wherein the de-rendering stage utilizes a combination of computer vision techniques and a neural network for processing document context; and s an interpretation stage that maps the intermediate data representation to a user- defined output format, wherein the interpretation stage utilizes a large language model (LLM) guided by prompts tailored to the specific characteristics of scientific data; o providing a user interface for iteratively refining the extracted data at both the de- rendering and interpretation stages, wherein the user interface allows a user to: g review the intermediate data representation; s modify the intermediate data representation; s review the mapping to the user-defined output format; and s modify the mapping to the user-defined output format; o utilizing user modifications to the intermediate data representation and the mapping to the user-defined output format to improve the performance of the de- rendering and interpretation stages; and o outputting the extracted data in the user-defined output format.

2. A system for automatically extracting data from tables and graphs in scientific literature, the system comprising: o a processor and memory; o a de-rendering module, executed by the processor, configured to convert tables and graphs from a scientific document into a standardized intermediate data representation, the de-rendering module utilizing: s computer vision techniques, including deep learning models, and s a neural network for processing document context; o an interpretation module, executed by the processor, configured to map the intermediate data representation to a user-defined output format, the interpretationmodule utilizing a large language model (LLM) guided by prompts tailored to the specific characteristics of scientific data; o a user interface module, executed by the processor, configured to: s display the intermediate data representation to a user; s receive user modifications to the intermediate data representation; s display the mapping to the user-defined output format; g receive user modifications to the mapping to the user-defined output format; and g transmit the user modifications to the de-rendering and interpretation modules; and o a dynamic update module, executed by the processor, configured to utilize the user modifications to improve the performance of the de-rendering and interpretation modules.

3. The method of claim 1, wherein the de-rendering stage further comprises performing text recognition using optical character recognition (OCR) or other computer vision techniques, such as Vision Transformers (ViTs), on text within the at least one table or graph, and wherein the user interface allows the user to correct errors in the recognized text.

4. The method of claim 1, wherein the de-rendering stage further comprises performing table structure detection using a combination of deep learning models and rule-based systems, and wherein the user interface allows the user to modify the detected table structure.

5. The method of claim 1, wherein the de-rendering stage further comprises performing graph-to-table conversion, and wherein the user interface allows the user to modify parameters of the graph-to-table conversion process.

6. The method of claim 1, wherein the interpretation stage further comprises using a library of callable functions to transform data elements from the intermediate data representation to the user-defined output format, and wherein the user interface allows the user to select, modify, or add callable functions to the library.

7. The method of claim 6, wherein the interpretation stage further comprises prompting an LLM to generate code snippets for data transformations not covered by existing callable functions in the library.

8. The system of claim 2, wherein the de-rendering module is further configured to perform text recognition using optical character recognition (OCR) or other computer vision techniques, such as Vision Transformers (ViTs), on text within tables andgraphs, and wherein the user interface module is further configured to allow the user to correct errors in the recognized text.

9. The system of claim 2, wherein the de-rendering module is further configured to perform table structure detection using a combination of deep learning models and rule-based systems, and wherein the user interface module is further configured to allow the user to modify the detected table structure.

10. The system of claim 2, wherein the de-rendering module is further configured to perform graph-to-table conversion, and wherein the user interface module is further configured to allow the user to modify parameters of the graph-to-table conversion process.

11. The system of claim 2, wherein the interpretation module is further configured to use a library of callable functions to transform data elements from the intermediate data representation to the user-defined output format, and wherein the user interface module is further configured to allow the user to select, modify, or add callable functions to the library.

12. The system of claim 11, wherein the interpretation module is further configured to prompt an LLM to generate code snippets for data transformations not covered by existing callable functions in the library.

13. The system of claim 2, further comprising a knowledge graph construction and utilization module configured to build and use a knowledge graph to enhance data extraction, wherein the user interface module is further configured to allow the user to interact with and explore the knowledge graph.

14. The method of claim 1, wherein the neural network for processing document context comprises a Bi-directional Long Short-Term Memory (Bi-LSTM) network, a Transformer, or a State Space Model.

15. The method of claim 1, wherein the computer vision techniques used in the de- rendering stage include Convolutional Neural Networks (CNNs) or Vision Transformers (ViTs).

16. The method of claim 5, wherein the graph-to-table conversion handles various graph types including bar charts, line graphs, scatter plots, box plots, heatmaps, survival curves, forest plots, flow charts, pie charts, and area plots.

17. The method of claim 1, wherein the prompts used to guide the LLM include instructions for identifying table structure, extracting specific data elements, interpreting the meaning of data elements in the context of the document, inferringrelationships between data elements, and generating explanations for the LLM's reasoning.

18. The method of claim 1, wherein the interpretation stage further comprises utilizing the LLM to resolve ambiguities in the extracted data by providing the LLM with contextual information from the neural network and the extracted data.

19. The method of claim 1, wherein the user modifications are utilized to improve the performance of the de-rendering and interpretation stages through retraining or fine- tuning of the deep learning models, the neural network, or the LLM.

20. The method of claim 1, wherein the scientific document is from the life sciences or healthcare domain.

21. The system of claim 2, wherein the neural network for processing document context comprises a Bi-directional Long Short-Term Memory (Bi-LSTM) network, a Transformer, or a State Space Model.

22. The system of claim 2, wherein the computer vision techniques used in the derendering stage include Convolutional Neural Networks (CNNs) or Vision Transformers (ViTs).

23. The system of claim 10, wherein the graph-to-table conversion handles various graph types including bar charts, line graphs, scatter plots, box plots, heatmaps, survival curves, forest plots, flow charts, pie charts, and area plots.

24. The system of claim 2, wherein the prompts used to guide the LLM include instructions for identifying table structure, extracting specific data elements, interpreting the meaning of data elements in the context of the document, inferring relationships between data elements, and generating explanations for the LLM's reasoning.

25. The system of claim 2, wherein the interpretation module further comprises utilizing the LLM to resolve ambiguities in the extracted data by providing the LLM with contextual information from the neural network and the extracted data.

26. The system of claim 2, wherein the user modifications are utilized to improve the performance of the de-rendering and interpretation stages through retraining or fine- tuning of the deep learning models, the neural network, or the LLM.

27. The system of claim 2, wherein the scientific document is from the life sciences or healthcare domain.

28. The method of claim 1, wherein the de-rendering stage is performed prior to the interpretation stage, and wherein the intermediate data representation produced by the de-rendering stage serves as input to the interpretation stage.

29. The system of claim 2, wherein the de-rendering module is configured to operate independently of the interpretation module, and wherein the output of the de- rendering module serves as input to the interpretation module.

30. The method of claim 5, wherein the de-rendering stage further comprises performing graph-to-table conversion on at least one graph in the scientific document, the graph- to- table conversion including: o classifying the graph type using a convolutional neural network (CNN) or a vision transformer (ViT); o extracting axis information using optical character recognition (OCR) or other computer vision techniques; o extracting data points using algorithms specific to the classified graph type; and o constructing a table representing the extracted data points and axis information.

31. The system of claim 10, wherein the de-rendering module is further configured to perform graph-to-table conversion on at least one graph in the scientific document, the graph-to-table conversion including: o classifying the graph type using a convolutional neural network (CNN) or a vision transformer (ViT); o extracting axis information using optical character recognition (OCR) or other computer vision techniques; o extracting data points using algorithms specific to the classified graph type; and o constructing a table representing the extracted data points and axis information.

32. The method of claim 30, wherein extracting data points includes using connected component analysis for bar charts, line detection and interpolation for line graphs, and data point detection algorithms for scatter plots.

33. The system of claim 31, wherein extracting data points includes using connected component analysis for bar charts, line detection and interpolation for line graphs, and data point detection algorithms for scatter plots.

34. The method of claim 30, wherein extracting data points further comprises using color segmentation to differentiate data points belonging to different categories and shape recognition using a machine learning model to differentiate data point markers.

35. The system of claim 31, wherein extracting data points further comprises using color segmentation to differentiate data points belonging to different categories and shape recognition using a machine learning model to differentiate data point markers.

36. The system of claim 2, further comprising an uncertainty handling module configured to:o assign a confidence score to each extracted data element based on factors including agreement between outputs of the de-rendering and interpretation stages, an assessment of the inherent ambiguity of visual features in tables and graphs, an assessment of the certainty of textual content extracted using OCR or other computer vision techniques, and contextual information from the neural network; o identify data elements with confidence scores below a predetermined threshold; and o trigger at least one of: s refinement of the data extraction using the LLM-based data extraction module; or s flagging of the data element for review and potential correction by a user through the user interface.

37. The method of claim 1, further comprising assigning a confidence score to each extracted data element based on factors including agreement between outputs of the derendering and interpretation stages, an assessment of the inherent ambiguity of visual features in tables and graphs, an assessment of the certainty of textual content extracted using OCR or other computer vision techniques, and contextual information from the neural network; identifying data elements with confidence scores below a predetermined threshold; and performing at least one of: o refining the extraction of the data element using the LLM-based data extraction module; or o flagging the data element for review and potential correction by a user through the user interface.

38. The method of claim 17, wherein the prompts used to guide the LLM include instructions for resolving ambiguities in extracted data by considering contextual information from the neural network, such as resolving unclear OCR output based on surrounding text or identifying the meaning of abbreviations based on domain- specific knowledge.

39. The system of claim 24, wherein the prompts used to guide the LLM include instructions for resolving ambiguities in extracted data by considering contextual information from the neural network, such as resolving unclear OCR output based on surrounding text or identifying the meaning of abbreviations based on domain- specific knowledge.

40. The method of claim 5, wherein the user interface allows the user to modify parameters of the graph-to-table conversion process, including axis scaling, data point extraction, and legend association.

41. The system of claim 10, wherein the user interface module is further configured to allow the user to modify parameters of the graph-to-table conversion process, including axis scaling, data point extraction, and legend association.

Images