System and method for extracting statistical information from documents

Abstract

Each embodiment of this disclosure provides an end-to-end system for extracting statistical relationships from scientific literature using various machine learning (ML) models, including generative large-scale language models (LLMs), and statistical models. The system is designed to identify and extract two types of statistical relationships: “general” or effect-size relationships, and “paired” or group comparison relationships.

Description

Detailed Description of the Invention

[0001] [Cross - Reference to Related Applications] This patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 463374, filed May 2, 2023, entitled "System and Methods for Extracting Statistical Information from Documents", the entire disclosure of which (including any amendments) is incorporated herein by reference.

[0002] References to "systems" within the context of the architecture herein, or references to system architectures or platforms, refer to the architecture, platform, and processes for performing statistical search and other forms of data compilation described in U.S. Patent Application No. 16 / 421249, filed May 23, 2019, now U.S. Patent No. 11354587, issued Jun. 7, 2022, entitled "Systems and Methods for Organizing and Finding Data". This patent claims the priority of U.S. Provisional Patent Application No. 62 / 799981, filed Feb. 1, 2019, entitled "Systems and Methods for Organizing and Finding Data", the entire content of which (and the entire content of any related applications claiming the priority of one or more of these applications) is incorporated herein by reference.

[0003] [Background] The information and relationships contained in documents that describe research, investigations, and scientific work can be very useful to people interested in the same or related topics. However, identifying and extracting statistical relationships from a set of sources is at least partially challenging because it takes time and is computationally very costly to examine and process large numbers of such documents. Each embodiment of the systems and methods disclosed herein is aimed at solving these problems and related problems, either individually or collectively.

[0004] [overview] The terms “invention,” “the invention,” “this invention,” “the present invention,” and “the present disclosure,” as used herein, refer broadly to the subject matter and claims disclosed and / or described herein, in the drawings, or in each drawing. No description containing these terms is intended to limit the meaning or scope of the disclosed or described subject matter or claims. Embodiments of the disclosure are defined by the claims, not by this summary. This summary is an overview of various aspects of the disclosure and introduces some of the concepts further described in the detailed description. This summary is not intended to identify any important, essential, or necessary features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by referring to the appropriate portions of this entire specification, any or all of the figures or drawings, and each claim.

[0005] Each embodiment of this disclosure relates to the fields of machine learning (ML) and natural language processing (NLP), and more specifically, provides an end-to-end system or platform for extracting statistical relationships from scientific literature using one or more machine learning models and / or statistical models, including a generative large language model (LLM).

[0006] In the context of this disclosure, and as used herein, “statistical relationship” means a relationship established between two variables using an effect size measure / metric or significance test (i.e., a valid and accepted methodology for establishing such a relationship). In one embodiment, the system disclosed and / or described operates to identify and extract two types of statistical relationships from one or more sources: (1) “general” or “effect size” relationships, and (2) “paired” or “group comparison” relationships.

[0007] In one embodiment, the disclosure relates to a method for extracting statistical relationships from scientific literature or other sources. The method identifies and extracts two types of statistical relationships from one or more sources: (1) “effect size” relationships and (2) “group comparison” relationships. Embodiments of the method of the disclosure may include one or more of the following steps, stages, elements, components, functions, operations, or processes.

[0008] ● Access the published summary from the document source and begin processing; ○ In one non-exclusive example, this is a PubMed server that displays publications from the U.S. National Institutes of Health (NIH) (https: / / pubmed.ncbi.nlm.nih.gov / ).

[0009] ● Perform sentence splitting for each of the accessed summaries. ● Apply a pattern-based and / or model-based tagging (labeling) process to each sentence to isolate the relevant text sections. ○ In non-restrictive examples of such text sections, one or more of the following may be included: text describing the experimental methods (i.e., acceptance / exclusion criteria for the study, statistical methods used) and text describing the results / findings of the study.

[0010] ● This is followed by a statistical relationship extraction process. ○ In one embodiment, the relation extraction process (referred to herein as REx) comprises a single processing flow.

[0011] ■ The REx process requires the LLM to identify and extract both "effect size" relationships and "group comparison" relationships from the summary and to place the output into a predefined JSON object that reflects the structure defined herein (a non-exclusive example provided as part of this specification).

[0012] ■ For each extracted relationship (i.e., effect size or group comparison), the LLM model is required to provide (1) the text excerpt used to extract the relationship, and (2) the rationale for the extraction. Acceptance of the rationale is an application of Chain of Thought (CoT) prompting, aimed at improving the LLM's reasoning ability as part of ongoing LLM training.

[0013] ○ In one embodiment, the relation extraction process comprises two separate processing flows (referred to herein as GREx and PREx), each of which, instead of executing the REx flow, individually extracts either effect-size relations (GREx) or group comparison relations (PREx).

[0014] ■ In one embodiment, the PREx process disclosed and / or described requires the LLM to extract “paired” or group comparison relationships from the source of interest and to store that information in a string representation of a predefined JSON object.

[0015] ■ In one embodiment, the sentence segmentation and model-based tagging or labeling process may be applied before the paired relationship extraction process. ■ This is followed by the use of a general or effect size relationship extraction process (referred to herein as GREx). In one embodiment, the GREx process disclosed and / or described requires the LLM to extract “general” relationships from the source of interest and to place the information within a string representation of a predefined JSON object. ● In one embodiment, the pattern-based tagging or labeling process may be applied before general relationship extraction processing.

[0016] ● The output of one relation extraction process (REx) or multiple relation extraction processes (PREx and GREx) is input to a structured relation process flow. In one non-limiting example, this structured relation process flow may operate as shown below.

[0017] ○ By converting the extracted LLM output into structured relationships, it becomes possible to process each relationship based on its constituent elements. This may include verifying that the output conforms to one or more desired definitions of the relationship. This allows for the elimination of bad data and false positives, and the verification of relationships by performing data validation on each constituent element.

[0018] ○ When using a two-flow relationship extraction process to extract effect size relationships (GREx) in a separate processing flow from group comparison relationships (PREx), if necessary, the raw predictions returned directly from LLM as a string representation of a JSON (JavaScript Object Notation) object are parsed / loaded into a "valid" JSON object. Relationships that cannot be properly parsed into a valid JSON object may be discarded to reduce the risk of relying on inaccurate extractions.

[0019] ○ Verify the elements of the relationships extracted (from the REx flow, or from either the GREx flow and the PREx flow) to ensure that they meet the acceptable requirements in both their individual components and their interrelationships. This is done to eliminate relationships that may be inaccurate or insufficiently analyzed. In one non-limiting example, this verification process may include one or more of the following:

[0020] ■ Verify that the confidence interval boundary (if found) is valid. That is, the CI lower limit (confidence interval lower limit) must be less than and not equal to the CI upper limit, and the statistical value must be between the CI boundaries.

[0021] ■ If a p-value is found, confirm that it is within the interval (0,1) (and does not contain 0 or 1).

[0022] ○ In the case of a "group comparison" relationship, that relationship may be converted into a structured relationship using the following method. ■ Assign the relationship to the default statistic type for mean difference. ■ Combine the names of two independent variable groups / times into a single variable name, _1. ● In one embodiment, the effect size relationships are already in the desired format (as shown in Figure 1(e)). Since the group comparison relationships are converted to this format, all relationships can be represented in a single format.

[0023] ● Next, "clean" the variables obtained from the output of the structured relation process flow (if necessary). ○ In one non-restrictive example, this may include one or more of the following: ■ Normalizes Unicode text to its canonical ASCII counterpart, enabling better compatibility with downstream applications (e.g., displaying web pages). ■ Spell out abbreviations used in variables as they are defined in the source text. For example, the variable ART extracted from the text is defined as an abbreviation for Antiretroviral therapy (ART). This variable is modified to Antiretroviral therapy, making it clearer and more informative than its original extracted form.

[0024] ● Next, perform a semantic grounding process to more efficiently clarify and / or expand variable names or identifiers. ○ As a non-limiting example, this may involve accessing one or more comprehensive ontologies (and / or related dictionaries or thesauruses) to identify similar or generalized forms of terms and concepts.

[0025] ● After completion of the aforementioned steps or stages, save the obtained variables, relationships (effect sizes and group comparisons), and related statistical information in a database for later access and evaluation.

[0026] ○ Such access and evaluation may include one or more of the usage methods disclosed and / or described in other pending applications and issued patents assigned to the assignee. ■ For example, but not limited to, U.S. Patent Application No. 17 / 983180. This application is a partial continuation of U.S. Patent Application No. 17 / 736897, which is a continuation of U.S. Patent Application No. 16 / 421249 (filed on May 23, 2019), and is now U.S. Patent No. 11354587 (issued on June 7, 2022). This application claims the benefit of priority of U.S. Provisional Patent Application No. 62 / 799981, entitled "Systems and Methods for Organizing and Finding Data," filed on February 1, 2019.

[0027] ○ As a non-limiting example, one use case is to improve the accuracy and utility of search results provided in response to a user query by identifying results that represent content including effect size relationships and / or group size relationships. This is because such sources are expected to be more likely to be related to the query or to support the search results.

[0028] ■ In the context of a "system" architecture, information and data used to subsequently add a knowledge graph or feature graph may be extracted using the disclosed and / or described processing flow. These graphs can identify potentially useful datasets and information when searched by a user.

[0029] In one embodiment, the disclosure relates to a system for extracting statistical relationships from scientific or other literature. The system may include a set of computer-executable instructions and an electronic processor or coprocessor. When executed by the processor or coprocessor, the instructions cause the processor or coprocessor (or a device in which they are part) to perform a set of operations that implement one or more embodiments of the methods disclosed and / or described.

[0030] In one embodiment, the disclosure relates to a set of computer-executable instructions contained in (or on) one or more non-temporary computer-readable media, where the set of instructions is executed by an electronic processor or coprocessor, and the processor or coprocessor (or a device in which they are part) executes a set of operations that implement embodiments of the disclosed and / or described methods.

[0031] In some embodiments, the systems and methods disclosed and / or described herein may be provided through a SaaS or multi-tenant platform. The platform provides access to multiple entities, each having a separate account and associated data storage device. Each account may correspond to, for example, a user, a set of users, an entity (such as an assignee) that provides a knowledge graph or feature graph that enables users to identify datasets for training models or to use when generating metrics of interest, a set of documents or document sources, or a mechanism that identifies such sources and directs access to the knowledge graph or feature graph. An account or user may, in non-limiting examples, want to use the embodiments to search for scientific research / insights, synthesize or summarize research in a particular area of ​​interest, or conduct a literature review. Each account may have access to one or more services, which are instantiated in that account and perform one or more of the methods or functions disclosed and / or described herein.

[0032] A detailed description and the accompanying drawings will reveal to those skilled in the art other purposes and advantages of the systems, apparatus, and methods disclosed and / or described herein. Throughout the drawings, the same reference numerals and descriptions represent similar elements, but not necessarily identical elements. While the embodiments disclosed and / or described herein are open to various variations and alternative forms, each drawing shows a specific embodiment as an example and describes it in detail herein. However, the embodiments of this disclosure are not limited to the exemplary or specific forms described. Rather, this disclosure covers all variations, equivalents, and alternatives that fall within the scope of the appended claims. [Brief explanation of the drawing]

[0033] Each embodiment of this disclosure will be described with reference to the following drawings. [Figure 1(a)]Figure 1(a) is a flowchart or flow diagram illustrating a set of steps, stages, functions, operations, or processes that may be implemented in embodiments of the disclosed and / or described systems and methods, where a single process flow is used as part of a statistical relation extraction process. [Figure 1(b)] Figure 1(b) is a flowchart or flow diagram illustrating a set of steps, stages, functions, operations, or processes that may be implemented in embodiments of the disclosed and / or described systems and methods, in which two process flows are used as part of a statistical relationship extraction process. [Figure 1(c)] Figure 1(c) is a block diagram illustrating a set of elements, components, functions, processes, or operations that may be part of a system architecture or data processing pipeline, where embodiments of the disclosed and / or described systems and methods may be implemented for a single process flow used as part of a statistical relation extraction process. [Figure 1(d)] Figure 1(d) is a block diagram illustrating a set of elements, components, functions, processes, or operations that may be part of a system architecture or data processing pipeline, where embodiments of the disclosed and / or described systems and methods may be implemented to use two process flows as part of a statistical relation extraction process. [Figure 1(e)] Figure 1(e) is a block diagram illustrating a data structure that may be used to represent statistical relationships extracted from the document as part of implementing embodiments of the disclosed and / or described systems and methods. [Figure 2(a)] Figure 2(a) is a block diagram illustrating a set of elements, components, functions, processes, or actions that may be performed to generate tags or labels for text extracted from a corpus of documents, as part of implementing embodiments of the disclosed and / or described systems and methods. [Figure 2(b)]Figure 2(b) shows a process flow or data processing pipeline for extracting statistical relationships from a corpus of documents in a single process flow (REx), as part of implementing embodiments of the disclosed and / or described systems and methods. [Figure 2(c)] Figure 2(c) shows a process flow or data processing pipeline (GREx) for extracting effect size (general) statistical relationships from a document corpus, as part of implementing embodiments of the disclosed and / or described systems and methods. [Figure 2(d)] Figure 2(d) shows a process flow or data processing pipeline (PREx) for extracting group comparison (paired) statistical relationships from a document corpus, as part of implementing embodiments of the disclosed and / or described systems and methods. [Figure 2(e)] Figure 2(e) is a diagram illustrating a process flow or data processing pipeline that “grounds” the identified variables using one or more ontologities, as part of implementing embodiments of the disclosed and / or described systems and methods. [Figure 2(f)] Figure 2(f) is a diagram illustrating the statistical relationship extraction process flow (multiple process flows) to be disclosed and / or described, which is used as part of identifying the concepts and related data or metadata of a search by adding a feature graph or knowledge graph of the type enabled by the assignee and then performing a search using that feature graph or knowledge graph. [Figure 2(g)] Figure 2(g) is a diagram showing elements, components, or processes that may be present in or performed by one or more computing devices, servers, platforms, or systems, which are configured to perform methods, processes, functions, or operations according to some embodiments of the systems and methods disclosed and / or described. [Figure 3]Figure 3 is a diagram showing the architecture of a multi-tenant or SaaS platform that may be used when implementing embodiments of the disclosed and / or described systems and methods. [Figure 4] Figure 4 is a diagram showing the architecture of a multi-tenant or SaaS platform that may be used when implementing embodiments of the disclosed and / or described systems and methods. [Figure 5] Figure 5 is a diagram showing the architecture of a multi-tenant or SaaS platform that may be used when implementing embodiments of the disclosed and / or described systems and methods. [Modes for carrying out the invention]

[0034] [Detailed explanation] The same numbering is used throughout this disclosure and the drawings to refer to similar components and features.

[0035] One or more embodiments of the disclosed subject matter are described herein with limitations to satisfy legal requirements, but this description does not limit the scope of the claims. The claimed subject matter may be embodied in other embodiments, may include different elements or steps, and may be used in combination with other existing or subsequently developed technologies. This description should not be construed as suggesting a required order or arrangement among the various steps or elements unless it is explicitly stated that a particular order of individual steps or arrangement of elements is required.

[0036] In this specification, each embodiment of the disclosure is described more fully with reference to the accompanying drawings, which constitute part of this specification and illustrate exemplary embodiments that enable the implementation of this disclosure. However, this disclosure may be embodied in other forms and should not be construed as being limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure satisfies statutory requirements and conveys the scope of this disclosure to those skilled in the art.

[0037] In particular, the subject matter of this disclosure may be embodied in whole or in part as a system, as one or more methods, or as one or more devices. Each embodiment may take the form of a hardware-implemented embodiment, a software-implemented embodiment, or an embodiment combining software and hardware aspects. For example, in some embodiments, one or more of the operations, functions, processes, or methods described herein may be implemented by one or more suitable processing elements (in non-limiting examples, such as a processor, coprocessor, microprocessor, CPU, GPU, TPU, QPU, or controller) that are part of a client device, server, network element, remote platform (such as a SaaS platform), "in-cloud" service, or other form of computing or data processing system, device, or platform.

[0038] One or more processing elements may be programmed using a set of executable instructions (e.g., software instructions), where the instructions may be stored on (or within) one or more suitable non-temporary data storage elements. In some embodiments, the set of instructions may be communicated to a user (e.g., over a network such as the Internet) through the transfer of instructions or an application that executes the set of instructions. In some embodiments, the set of instructions or the application may be made available to the end user through access to a SaaS platform or a service provided through such a platform.

[0039] In some embodiments, one or more of the operations, functions, processes, or methods described herein may be implemented by special forms of hardware, such as programmable gate arrays or application-specific integrated circuits (ASICs). However, some embodiments of this disclosure may be implemented in the form of applications, subroutines that are part of larger applications, "plug-ins," extensions to the functionality of data processing systems or platforms, or other suitable forms. Therefore, the detailed descriptions below should not be constrained to be restrictive.

[0040] As described above, in some embodiments, the systems and methods disclosed and / or described herein may be provided through a SaaS or multi-tenant platform. The platform provides access to multiple entities, each having a separate account and associated data storage. Each account may correspond to, for example, a user, a set of users, an entity (such as an assignee) that provides a knowledge graph enabling users to identify datasets for training models or to use when generating target metrics, a set of documents or document sources, or a mechanism that identifies such sources and directs access to and guidance from the knowledge graph. Each account may access one or more services, which are instantiated in that account and perform one or more of the methods or functions disclosed and / or described herein.

[0041] As described above, each embodiment relates to the fields of machine learning (ML) and natural language processing (NLP), and specifically provides an end-to-end system or platform for extracting statistical relationships from scientific literature using one or more machine learning models and / or statistical models, including generative large language models (LLMs). The extracted relationships may be used to add searchable knowledge graphs or other forms of data structures to identify datasets and information that are likely to be relevant to the user's queries or research.

[0042] In the context of this disclosure, and as used herein, “statistical relationship” means a relationship established between two variables using an effect size measure / metric or significance test (i.e., a valid and accepted methodology for establishing such a relationship). In one embodiment, the system disclosed and / or described operates to identify and extract two types of statistical relationships from one or more sources: (1) “general” or “effect size” relationships, and (2) “paired” or “group comparison” relationships.

[0043] The components of such statistical relationships generally include the following: ●2 variables ●Type of statistic (measurement of effect size, e.g., odds ratio or Pearson correlation (as a non-restrictive example)) ●Statistical value (the value of its effect size) ● Confidence interval (if any) ● p-value (if any) In the context of this disclosure, and as used herein, “effect size” or “general” relationship means a relationship that is clearly measured by effect size and found in the textual information of a reference or document. The relationship is established by the clear basis of the effect size measurement by the researcher and is documented in the reference or document.

[0044] In the context of this disclosure, and as used herein, “group comparisons” or “paired” relationships are not explicitly measured by effect sizes in the study literature or other records. Instead, the literature and records provide comparisons of two values ​​accompanied by statistically significant measurements. These pairs of values ​​may, in non-limiting examples, refer to different groups, different trials, or different time periods. For these relationships, the disclosed methodology can be used to calculate effect sizes based on the paired values ​​and their associated significant values ​​(multiple values) (for example, by using the default statistic type of “mean difference” disclosed herein).

[0045] One or more embodiments may also (or instead) extract research metadata and characteristics, such as sample size, population characteristics (e.g., age, social sex, biological sex, disease status, or comorbidity rates), and / or control variables using methods similar to those disclosed and / or described herein.

[0046] As a non-limiting example, in one study reported in the source document, 100 women aged 25–40 with a history of diabetes were randomly assigned to receive either drug X 100 mg or placebo (saline solution). This plan yields the following metadata describing the study, extracted from the study or one or more reference documents:

[0047] ● Specimen size: 100 ● Research population: ○ Gender: Female ○ Age: 25-40; ○ Disease / Health condition: Diabetes ○ Treatment: Drug X 100mg ○ Control variable: 100 mg of saline placebo The degree of freedom in handling both effect size (general) relationships and group comparison (paired) relationships allows the disclosure and / or description systems and methods to extract many (if not all) statistical relationships presented in scientific and other literature.

[0048] In some embodiments, the disclosed and / or described system (or platform, method, apparatus, or device) uses a combination of rules and machine learning models (e.g., one or more disclosed tagging process flows) to identify candidate documents that may contain a statistical relationship. Then, using patterns and / or supervised transformer models, these documents are classified (in one embodiment as part of one or more tagging process flows) to "predict" the likelihood of containing either an effect-size relationship or a group comparison relationship.

[0049] The extraction of relationships from documents is performed using a Large-Scale Language Model (LLM), which is facilitated and guided by examples specifically designed for the corresponding type of relationship (i.e., effect size or group comparison). The raw data representing the extracted relationships is then parsed into valid JSON objects (if this is not done during the extraction process itself).

[0050] To ensure that the extracted relationships adhere to predefined definitions or formats, each embodiment "cleans" and verifies the individual components of the extracted relationships. Next, the variables derived from the relationships are grounded to concepts from an ontology, such as (as a non-limiting example) the Unified Medical Language System (UMLS). UMLS aggregates specialized terminology from a wide variety of scientific ontologities and knowledge bases, including Medical Subject Headings (MeSH), ICD-10, and SNOMED CT. In this example, the ontologities were selected for their comprehensiveness, interoperability, and interoperability with non-specialized ontologs such as Wikidata. However, each embodiment has sufficient flexibility to incorporate grounding using one or more other related ontologities.

[0051] The resulting relationships can be used, along with their respective variables and UMLS concepts (which may in some cases provide alternative or generalized forms of terms, variables, labels, or concepts), to generate meta-analyses, visualize research networks, and simplify other applications in the scientific or technological domain. Each embodiment facilitates the understanding and application of statistical relationships in research and practice by providing a robust and efficient solution for extracting valuable insights from scientific literature.

[0052] For example, the resulting relationships can be loaded into an application's relational or graph database, enabling one or more of the following: Semantic searches are performed based on components such as variable names (e.g., searching for relationships related to metabolic syndrome), types of statistics (e.g., searching for odds ratios), statistical values ​​(e.g., searching for those with odds ratios greater than 1.0), or the significance of the relationship at a specific confidence level (e.g., filtering for relationships that are significant at a 95% confidence level). ● Perform a meta-analysis by filtering findings obtained from a specific scope of the study (for example, all relationships related to metabolic syndrome as determinants), performing statistical analysis, and calculating the consensus (or lack thereof) of relationships between variables based on statistical findings.

[0053] The following are non-limiting, exemplary embodiments of implementing one or more of the data processing techniques disclosed and / or described. In this example, PubMed (including biomedical publications, available at https: / / pubmed.ncbi.nlm.nih.gov / ) is used as the primary source of data for identifying statistical relationships in a study or survey. PubMed's public FTP server provides a comprehensive collection of XML files containing summaries and other metadata related to a wide selection of scientific publications.

[0054] In some embodiments, the primary focus or objective is to identify key findings presented in the summaries and one or more results / conclusion sections found in the publication's text. These sections are expected to be useful as they generally provide an overview of the most significant results and insights derived from the research described in the publication. By focusing on these parts of the document, each embodiment can efficiently extract one or more statistical relationships that support the scientific findings described in the publication. This approach allows embodiments to efficiently identify and analyze the data expected to be most relevant, thereby increasing the efficiency of the extraction process and improving the accuracy and reliability of the results.

[0055] Figure 1(a) is a flowchart or flow diagram illustrating a set of steps, stages, functions, operations, or processes that may be implemented in embodiments of the disclosed and / or described systems and methods, where a single process flow is used as part of the statistical relation extraction process. This may be preferable to implementing the extraction process flow in two separate flows or pipelines (one for effect size relations and the other for group comparison relations) for one of the reasons described below.

[0056] ● Combining the two tasks into one can help prevent situations where one type of relationship is mistaken for the other (for example, an LLM attempting to extract effect size relationships rather than group comparison relationships). ● By using a single processing flow, its implementation and data flow are simplified. ● By using a single processing flow, the cost and overhead of one or more LLMs used within the process are reduced.

[0057] As shown in Figure 1(a), in an exemplary embodiment, the processing flow may include the following steps, stages, operations, or functions. ● Access the published summary from the document source and begin processing (as presented in step or stage 102). ○ In one non-exclusive example, this is a PubMed server displaying publications from the U.S. National Institutes of Health (NIH) (which can be found at https: / / pubmed.ncbi.nlm.nih.gov / ). ○ In some embodiments, other parts of the paper (such as results, conclusions, or other potentially relevant sections) may be accessed and their relevance assessed.

[0058] ● Perform sentence splitting for each of the accessed summaries (as presented in step or stage 104). This can be accomplished using numerous techniques, including rule-based sentence boundary disambiguation (SBD), or by using transformer models trained or fine-tuned for this particular task.

[0059] ● Apply a pattern-based and / or model-based tagging (labeling) process to each sentence to isolate the relevant text sections (as presented in step or stage 106). ○ Specifically, one or more text sections describing the experimental methods (i.e., acceptance / exclusion criteria for the study, statistical methods used) and the results / findings of the study.

[0060] ● This is followed by the relationship extraction process (as presented in step or stage 108). ○ In one embodiment, the relation extraction process (referred to herein as REx) comprises a single processing flow. The REx process requires the LLM to identify and extract both "effect size" relationships and "group comparison" relationships from the summary, and to place the output into a predefined JSON object that reflects the structure defined herein.

[0061] ■ As a non-limiting example, in one embodiment, this JSON object may be defined as follows:

[0062] TIFF2026519421000002.tif113169

[0063] TIFF2026519421000003.tif101169

[0064] TIFF2026519421000004.tif68169

[0065] TIFF2026519421000005.tif130169

[0066] TIFF2026519421000006.tif107169

[0067] TIFF2026519421000007.tif107169

[0068] TIFF2026519421000008.tif98169

[0069] TIFF2026519421000009.tif75169

[0070] TIFF2026519421000010.tif112169

[0071] TIFF2026519421000011.tif145169

[0072] ■ For each extracted relationship (i.e., effect size or group comparison), the LLM model is required to provide (1) the text excerpt used to extract the relationship, and (2) the rationale for the extraction. Acceptance of the rationale is an application of Chain of Thought (CoT) prompting, aimed at improving the LLM's reasoning ability as part of ongoing training.

[0073] ○ A non-restrictive example of the output of the REx extraction process returned by LLM is as follows:

[0074] TIFF2026519421000012.tif168169

[0075] TIFF2026519421000013.tif160169

[0076] ● The output of the Relation Extraction (REx) process is input into the Structured Relations process flow (as presented in step or stage 110). By converting the extracted LLM output into structured relations, each relation can be processed based on its components. This may include verifying that the output conforms to one or more desired definitions of the relation. This makes it possible to eliminate bad data and false positives and validate the relation by performing data validation on each component. In some embodiments, this may include the following:

[0077] ○ Verify and / or clean the relationship elements extracted from the output of the structured relationship process flow to ensure that each individual component, as well as its relationships with others, meets the acceptable requirements. This may be done by eliminating relationships that were analyzed incorrectly or insufficiently. In one non-limiting example, the verification process may include one or more of the following:

[0078] ■ Verify that the confidence interval boundary (if found) is valid. That is, the CI lower limit (confidence interval lower limit) must be less than and not equal to the CI upper limit, and the statistical value must be between the CI boundaries.

[0079] ■ If a p-value is found, confirm that it is within the interval (0,1) (and does not contain 0 or 1).

[0080] ○ In the case of a "group comparison" relationship, that relationship may be converted into a structured relationship using the following method. ■ Assign the relationship to the default statistic type for mean difference. ■ Combine the names of two independent variable groups / times into a single variable name, _1. for example,

[0081] TIFF2026519421000014.tif189169

[0082] ● Next, you may “clean” and / or validate the variables obtained from the output of the structured relationship process flow (if necessary and not performed as part of the structured relationship process flow) (as presented in step or stage 112).

[0083] ○ In one non-restrictive example, this may include one or more of the following: ■ Normalizes Unicode text to its canonical ASCII counterpart, enabling better compatibility for downstream uses (e.g., displaying web pages). ■ Spell out abbreviations used in variables as they are defined in the source text. For example, the variable ART extracted from the text is defined as an abbreviation for Antiretroviral therapy (ART). This variable is modified to Antiretroviral therapy, making it clearer and more informative than its original extracted form.

[0084] ● Next, perform a semantic grounding process to more efficiently clarify and / or expand variable names or identifiers (as presented in step or stage 114). ○ As a non-restrictive example, this may involve accessing one or more comprehensive ontologs (and / or related dictionaries or thesauruses) to identify analogous or generalized forms of terms or concepts.

[0085] ● After completing the aforementioned steps or stages, the obtained variables, relationships (effect sizes and group comparisons), and related statistics are stored in a database for later access and evaluation (as presented in step or stage 116). Such access and evaluation may include one or more uses disclosed and / or described in other pending applications and patented applications assigned to the assignee.

[0086] ■ For example, U.S. Patent Application No. 17 / 983180, but not limited to this. This application is a continuation-in-part application of U.S. Patent Application No. 17 / 736897, which is a continuation-in-part application of U.S. Patent Application No. 16 / 421249 (filed on May 23, 2019) and is currently U.S. Patent No. 11354587 (issued on June 7, 2022). This application claims priority from U.S. Provisional Patent Application No. 62 / 799981, titled "Systems and Methods for Organizing and Finding Data," filed on February 1, 2019.

[0087] ○ As a non-limiting example, one use case is to improve the accuracy and usefulness of search results provided in response to user queries by identifying results that represent content including effect size relationships and / or group size relationships. This is because such sources can be expected to be more relevant to the query or more likely to support the search results.

[0088] ■ In the context of the "system" architecture, information and data may be extracted using the disclosed and / or described processing flows to subsequently add knowledge graphs and feature graphs. These graphs may allow users to search for potentially useful datasets and information, identify relevant results, rank results, suggest further searches, or provide other information of interest.

[0089] Figure 1(b) is a flowchart or flow diagram illustrating a set of steps, stages, functions, operations, or processes that may be implemented in embodiments of the disclosed and / or described systems and methods, where two process flows are used as part of a statistical relation extraction process. This may be preferable to implementing the extraction process flows in a single flow or pipeline (one for both effect-size relations and group comparison relations) in situations where the user has different definitions or requirements for each extraction type, or when they are interested in only one form of statistical relation.

[0090] As shown in Figure 1(b), in an exemplary embodiment, the processing flow may include the following steps, stages, operations, or functions.

[0091] ● Access the published summary from the document source and begin processing (as presented in step or stage 120). ○ In one non-exclusive example, this is a PubMed server displaying publications from the U.S. National Institutes of Health (NIH) (which can be found at https: / / pubmed.ncbi.nlm.nih.gov / ). ○ In some embodiments, other parts of the paper (such as results, conclusions, or other potentially relevant sections) may be accessed and their relevance assessed.

[0092] ● Perform sentence splitting for each of the accessed summaries (as presented in step or stage 122). This can be accomplished using numerous techniques, including rule-based sentence boundary determination (SBD), or by using transformer models trained or fine-tuned for this particular task.

[0093] ● Perform a model-based tagging (labeling) process for each sentence to isolate the relevant text sections (as presented in step or stage 124).

[0094] ● This is followed by a relation extraction process. In one embodiment, the relation extraction process comprises two separate processing flows (referred to herein as PREx and GREx, as presented in steps or stages 126 and 130), each individually extracting either group comparison (PREx) relationships or effect size relationships (GREx).

[0095] ○ In one embodiment, the PREx process disclosed and / or described requires the LLM to extract “paired” relationships or group comparison relationships from summaries of interest and to place that information within a string representation of a predefined JSON object. ■ As a non-restrictive example of PREx extracted output (returned by LLM as a string),

[0096] TIFF2026519421000015.tif103169

[0097] ● Using the same or different sets of summaries, perform a pattern-based tagging (labeling) process and assign a tag or label to each summary (as presented in step or stage 128). ○ Note that the model-based tagging process is used in the PREx extraction process, and the pattern-based tagging process is used in the GREx extraction process.

[0098] ● This is followed by the use of a process for extracting general relationships or effect size relationships (referred to herein as GREx, as presented in step or stage 130). ○ In one embodiment, the GREx process disclosed and / or described requires the LLM to extract “general” relationships from the summary of interest and to place that information within a string representation of a predefined JSON object. ■ As a non-restrictive example of GREx extracted output (returned by LLM as a string),

[0099] TIFF2026519421000016.tif111169

[0100] ● The outputs of the relationship extraction processes (PREx and GREx) are fed into the structured relationship process flow (as presented in step or stage 132). In one non-exclusive example, this structured relationship process flow may operate as shown below.

[0101] ○ When using a two-flow relationship extraction process to extract effect size relationships (GREx) in a separate processing flow from group comparison relationships (PREx), if necessary, the raw predictions returned directly from LLM as string representations of JSON objects should be parsed / loaded into valid JSON objects (e.g., by parsing into JSON notation format). Relationships that cannot be properly parsed into valid JSON objects may be discarded to reduce the risk of relying on inaccurate extractions.

[0102] ○ Verify the relationship elements extracted (from the GREx flow and PREx flow) to ensure that each component, both individually and in its relationships with others, meets the acceptable requirements. This may be done to eliminate relationships that were analyzed inaccurately or insufficiently. In one non-limiting example, this verification process may include one or more of the following:

[0103] ■ Verify that the confidence interval boundary (if found) is valid. That is, the CI lower limit (confidence interval lower limit) must be less than and not equal to the CI upper limit, and the statistical value must be between the CI boundaries. ■ If a p-value is found, confirm that it is within the interval (0,1) (and does not contain 0 or 1).

[0104] ○ In the case of a "group comparison" relationship, the relationship is converted into a structured relationship using the following method (however, in this case, in some embodiments, the effect size relationship extracted by either the GREx or REx process flow is extracted in the desired format). ■ Assign the relationship to the default statistic type for mean difference. ■ Combine the names of two independent variable groups / times into a single variable name, _1. for example,

[0105] TIFF2026519421000017.tif188169

[0106] ● Next, clean and / or validate the variables obtained from the output of the structured relation process flow (if necessary) (as presented in step or stage 134). ○ In one non-restrictive example, this may include one or more of the following: ■ Normalizes Unicode text to its canonical ASCII counterpart, enabling better compatibility for downstream uses (e.g., displaying web pages). ■ Spell out abbreviations used in variables as they are defined in the source text. For example, the variable ART extracted from the text is defined as an abbreviation for Antiretroviral therapy (ART). This variable is modified to Antiretroviral therapy, making it clearer and more informative than its original extracted form.

[0107] ● Next, perform a semantic grounding process to more efficiently clarify and / or expand variable names or identifiers (as presented in step or stage 136). ○ In non-restrictive examples, this may involve accessing one or more comprehensive ontologs (and / or related dictionaries or thesauruses) to identify analogous or generalized forms of terms or concepts.

[0108] ● After completing the aforementioned steps or stages, store the obtained variables, relationships (effect sizes and group comparisons), and related statistics in a database for later access and evaluation (as presented in step or stage 138). Such access and evaluation may include one or more uses disclosed and / or described in other pending applications and patented applications assigned to the assignee.

[0109] ■ For example, U.S. Patent Application No. 17 / 983180, but not limited to this. This application is a continuation-in-part application of U.S. Patent Application No. 17 / 736897, which is a continuation-in-part application of U.S. Patent Application No. 16 / 421249 (filed on May 23, 2019) and is currently U.S. Patent No. 11354587 (issued on June 7, 2022). This application claims priority from U.S. Provisional Patent Application No. 62 / 799981, titled "Systems and Methods for Organizing and Finding Data," filed on February 1, 2019.

[0110] ○ As a non-limiting example, one use case is to improve the accuracy and usefulness of search results provided in response to user queries by identifying results that represent content including effect size relationships and / or group size relationships. This is because such sources can be expected to be more relevant to the query or more likely to support the search results.

[0111] ■ In the context of the "system" architecture, information and data may be extracted using the disclosed and / or described processing flows to subsequently add knowledge graphs and feature graphs. These graphs may allow users to search for potentially useful datasets and information, identify relevant results, rank results, suggest further searches, or provide other information of interest.

[0112] Figure 1(c) is a block diagram illustrating a set of elements, components, functions, processes, or operations that may be part of a system architecture or data processing pipeline, where embodiments of the disclosed and / or described systems and methods may be implemented for a single process flow used as part of a statistical relation extraction process. As shown in this figure, in one embodiment, such a system architecture or data processing pipeline may include the following:

[0113] ● A set of accessible publications, summaries, or other information relating to research or studies (as presented on PubMed FTP server 140). ● A process or service for "ingesting" a publication and identifying parts or sections of that publication (as presented in Ingestion Service 142). ○ In one embodiment, this may comprise one or more processes, such as a scheduled data pipeline implemented using a tool like Apache Airflow or Prefect. These pipelines can be configured to operate at regular intervals (e.g., daily, weekly) to fetch new data from a source such as a PubMed FTP server.

[0114] For example, using Apache Airflow, it is possible to define a directed acyclic graph (DAG) that represents a data ingestion workflow. This DAG may include tasks such as the following: ■ Connect to the PubMed FTP server using the Python ftplib library. ■ Download new or updated publications in a structured format (e.g., XML) using the ftplib.FTP.retrbinary() method. ■ Use a library like xml.etree.ElementTree for XML parsing to parse the downloaded file and extract the relevant sections or metadata. ■ The extracted data is stored in a database or file system for further processing.

[0115] Similarly, Prefect can be used to create a flow that encapsulates the data ingestion process. This flow may consist of tasks similar to those described for Apache Airflow, but with additional features such as automatic retries and error handling.

[0116] These scheduled pipelines can be triggered using Python code, which can be version-controlled and maintained within a repository. The code can define schedules (e.g., using cron expressions) and dependencies between tasks. By leveraging these tools and the Python library ecosystem, the data ingestion process can be automated, ensuring that the system regularly fetches and processes new publications as they become available on the PubMed FTP server (or other sources).

[0117] ● The ingestion service process outputs a set of summaries (or other identified sections (or multiple sections) of the accessed publication, such as the results / conclusions section) (as presented in PubMed summary 144). ● Apply the sentence segmentation process to each summary to "parse" the sentences within the summary (or other identified section) (as shown in sentence segmentation 146).

[0118] ● Next, apply a pattern-based tagging or labeling process to each sentence obtained from one or more sentence splitting operations (as presented in Pattern-Based Tagging 148). ● Next, apply the relation extraction process (REx) (as presented in relation extraction 150). ● Next, apply a process to put the output of the extraction process into a structured format (as presented in Structure Relationship 152).

[0119] ● Next, validate and / or clean the variables within the structured format (as presented in Variable Cleanup 154). Next, perform the semantic grounding process on the structured variables and / or other aspects (as presented in Semantic Grounding 156). ● Next, the resulting data, metadata, and information are stored in a database (as presented in system database 158) for subsequent access, analysis, retrieval, and presentation as a knowledge graph or feature graph.

[0120] Figure 1(d) is a block diagram illustrating a set of elements, components, functions, processes, or operations that may be part of a system architecture or data processing pipeline, where embodiments of the disclosed and / or described systems and methods may be implemented to use two process flows as part of a statistical relation extraction process. As shown in this figure, in one embodiment, such a system architecture or data processing pipeline may include the following:

[0121] ● A set of accessible publications, abstracts, or other information relating to research or studies (as presented on PubMed FTP server 160). ● A process or service for "importing" a publication and identifying parts or sections of that publication (as presented in Import Service 162). ○ In one embodiment, this may comprise one or more processes, such as those disclosed and / or described with respect to Apache Airflow or Prefect.

[0122] ● The ingestion service process outputs a set of summaries (or other identified sections (or multiple sections) of the accessed publication, such as the results / conclusions section) (as presented in PubMed summary 164). ● Apply the sentence segmentation process to each summary to "parse" the sentences within the summary (or other identified section) (as shown in sentence segmentation 166).

[0123] ● The output of the sentence splitting process may be subject to a model-based tagging or labeling process (as presented in Model-Based Tagging 170). Model-based methods are useful and are sometimes necessary for identifying text that cannot be reliably defined using rules. In particular, for text ranges describing group comparisons (i.e., "The proportion of patients with disease X is higher in group A than in group B [10% vs. 5%, p<0.05]"), model-based methods are preferable to ensure that these ranges are accurately identified. This can be achieved by training or fine-tuning a model, such as a transformer or recurrent neural network (RNN), to perform the task of text classification using examples of relevant tags.

[0124] ● The identified summaries may be subjected to a pattern-based tagging or labeling process (as presented in Pattern-Based Tagging 168). Pattern-based methods are applicable to relatively simple patterns that can be reliably obtained using rules. In particular, regular expressions can be used to reliably tag text that describes statistical methods such as "odds ratio" and "Pearson correlation."

[0125] ● The output of the model-based tagging process 170 is provided as input to a paired or group relation extraction process (as presented in PREx, 172). ● The output of the pattern-based tagging process 168 is provided as input to a general or effect-size relation extraction process (as presented in GREx, 174). ● Next, apply a process to put the output of one or more extraction processes into a structured format (as presented in structuring relation 176).

[0126] ● Next, validate and / or clean the variables within one or more structured forms (as presented in Variable Cleanup 178). Next, perform the semantic grounding process on the structured variables and / or other aspects (as presented in Semantic Grounding 180). ● Next, the resulting data, metadata, and information are stored in a database (as presented in system database 182) for subsequent access, analysis, retrieval, and presentation as a knowledge graph or feature graph.

[0127] Figure 1(e) is a block diagram illustrating a data structure that may be used to represent statistical relationships extracted from a document as part of implementing embodiments of the disclosed and / or described systems and methods. In one embodiment, the data structure comprises a “column” of relationships consisting of eight components. In this exemplary embodiment, there are four components necessary to write to a database of the disclosed system (or platform): namely, variable 1, variable 2, type of statistic, and statistic value.

[0128] Furthermore, the data structure includes four optional components: p-values, confidence interval levels, lower CI limits, and upper CI limits. These values ​​can enhance the understanding of the strength of the relationships identified and expressed by the variables and their associated data. The structure shown in Figure 1(e) accommodates a variety of variable and statistic types, enabling the representation of relationships across various fields and types of surveys and studies.

[0129] This expression (as a specific result of a variable) encodes the statistical relationship between variables. For relationships measured by directional effect sizes (e.g., odds ratios or hazard ratios), variable 1 represents the independent variable reported in the publication, and variable 2 represents the dependent variable. For bidirectional or non-directional effect sizes (e.g., correlations), either variable can be assigned to either variable 1 or variable 2 (i.e., the assignment is not directional).

[0130] The following is an example of the definition of elements in the data structure shown in Figure 1(e). ● Variables (Required): Two strings representing the variables to be compared using effect size statistics. ● Statistical type (required): The type of effect size measurement, e.g., odds ratio. Mapped to a set of acceptable statistical types. ● Statistical value (required): The measured value of the effect size, e.g., 1.5. Follow floating representation. ● Confidence interval level (optional): The degree of confidence, e.g., 95% or 99%. This must be convertible to a floating-point representation between 0 and 1. ● Confidence Interval Boundary (Optional): Confidence interval. This value must be convertible to a floating-point representation. The extracted lower bound must be smaller than the upper bound, and the statistical value must fall within this range. ● P-value (optional): The p-value is used to calculate the significance of the relationship between variables. This value must be convertible to a floating-point expression between 0 and 1, and it must be accompanied by an equality or inequality.

[0131] Identifying potential documents within the corpus for relation extraction. In one embodiment, the process flow for identifying document candidates for relation extraction may comprise a tagging and classifier model having regular expressions. Figure 2(a) is a block diagram illustrating a set of elements, components, functions, processes, or actions that may be performed as part of an implementation of embodiments of the disclosed system and method to generate tags or labels for text extracted from one or more documents. The process flow shown in Figure 2(a) may be used to narrow down or filter the set of accessed publications to a subset that is expected to be of great value for relation extraction.

[0132] As is well known, large-scale language models (LLMs) are relatively slow and costly to apply. While it is possible to determine whether input text contains statistical relationships to extract, running every document in a corpus through an LLM is inefficient and potentially costly.

[0133] To address this problem, in some embodiments, a filtering process including a tagging service or function is performed first (or possibly later). This service or function uses patterns and / or trained models to tag / label publications that are predicted to be relatively likely to contain (extractable) statistical relationships.

[0134] Tags may be assigned to a specific range of text that indicates the presence of a particular type of relationship. For example, the text "...odds ratio 1.5" indicates a statistical relationship calculated by an odds ratio of 1.5, and this text may be passed to the REx or GREx pipeline. In another example, the text "...(10% vs 20%, p<0.05)" indicates a statistical relationship calculated by a comparison between two groups, and this text may be passed to the REx or PREx pipeline.

[0135] In one embodiment, tagging patterns are identified and / or a model is trained on sample publications annotated by subject matter experts (as presented in the elements, components, or process 202). High-accuracy pattern definitions / descriptions are developed to detect occurrences of 50–100 types of statistics supported by the disclosed database (i.e., steps, stages, elements, components, or as mentioned in processes 116, 138, 158, and 182). Publications containing matches with one or more of these patterns are then passed to the REx or GREx pipeline disclosed and / or described herein.

[0136] The trained model is then used as part of a tagging service (as elements, components, or as presented in process 204). The tagging service 204 is attached to a corpus of documents or publications, as shown in PubMed corpus 203 in this diagram. The output of the tagging service 204 is a filtered or selected set of documents or publications (as elements, components, or as presented in process 206). For example, a filtered PubMed corpus refers to the text in each of the pipelines that has been found to have the relevant tags (in the case of pattern-based) or classified (in the case of model-based).

[0137] In one embodiment, a high-accuracy (F1 ~0.9) sentence classification model was trained by fine-tuning a pre-trained language model on a dataset of annotated examples. For example, this can be achieved by fine-tuning a BERT model (Bidirectional Encoder Representations from Transformers, https: / / arxiv.org / abs / 1810.04805). In one embodiment, the annotation or label was a binary flag attached to a sentence indicating whether it contained relational information necessary for successful extraction by the REx or PREx (and possibly GREx) pipeline disclosed and / or described herein.

[0138] Figure 2(a) illustrates both pattern-based and model-based tagging workflows, illustrating how the tagging service enables relatively fast filtering of the corpus of documents more likely to contain relationships of interest. Pattern-based tagging allows for the identification of tokens indicating the presence of effect-size relationships (e.g., by searching for "odds ratio"), while model-based tagging allows for the identification of more complex text ranges indicating the presence of group comparison relationships. The thus filtered corpus is then passed to the more computationally intensive LLM pipeline.

[0139] Relational Extraction (REx) Pipeline Figure 2(b) shows a process flow or data processing pipeline for extracting statistical relationships from a corpus of documents in a single process flow (REx), as part of implementing embodiments of the disclosed and / or described systems and methods.

[0140] In one embodiment, the relation extraction (REx) pipeline disclosed and / or described may include one or more of the following elements, components, functions, or processes, as shown in Figure 2(b):

[0141] ● G1: This system filters text that contains at least one of the following: ○ Non-restrictive examples include odds ratios, hazard ratios, or Pearson correlations, which are measures of effect size used to identify potential statistical relationships. ○ Sentences containing at least two numerical values ​​and one p-value that identify possible group comparisons.

[0142] ● G2: Construct an LLM prompt that provides the following (1)-(4): (1) a prompt containing task definitions and instructions, and one or more basic and high-level examples; (2) an example of interaction between a “user” (an example of scientific text) and an “assistant” (the LLM’s extraction of the scientific text) as a representative example of how LLM works; (3) the scientific text from which LLM will extract relationships (if any); and (4) the desired output (i.e., function calls), and one or more JSON schemas to define the LLM’s output structure.

[0143] ● G3: Pass the constructed prompt to the LLM to perform inference. ○ This step involves feeding the generated prompts to a large-scale language model (LLM) that has been trained on a large amount of text data and fine-tuned using instruction-based learning or reinforcement learning from human feedback (RLHF). ○ The LLM used for this purpose can be either a general-purpose model or a domain-specific model, depending on the nature of the task and the desired level of expertise.

[0144] ● G4:LLM outputs valid JSON predictions containing potential statistical relationships from the target scientific text / publication / document. ● G5: Clean and validate the individual components of a structured relationship according to the definition or defined format. Components that fail validation may result in the discarding of part or all of the relationship. ● G6: The resulting structured relationships are output via a pipeline and made available for storage and / or other processes.

[0145] As described above, in one embodiment, two separate pipelines may be used to extract statistical relationships, namely, one for group comparisons (i.e., paired relationships) and the second pipeline for effect size (i.e., general) relationships.

[0146] General Relationship Extraction (GREx) Pipeline Figure 2(c) shows a process flow or data processing pipeline (GREx) for extracting effect size (general) statistical relationships from a document corpus, as part of implementing embodiments of the disclosed and / or described systems and methods.

[0147] In one embodiment, the general relational extraction (GREx) pipeline disclosed and / or described may include one or more of the following elements, components, functions, or processes, as shown in Figure 2(c):

[0148] ● G1: This process filters text corpora containing effect size measurements that may include statistical relationships, such as odds ratios, hazard ratios, or Pearson correlations, as non-limiting examples. ● G2: Construct an LLM prompt that performs the following 1) to 3): 1) Define structured relationships within the JSON schema, 2) Provide examples of scientific texts and the relationships extracted from them, and 3) Extract scientific texts from which LLM will extract relationships (if any).

[0149] ● G3: Pass the constructed prompt to the LLM to perform inference. ○ This step involves feeding the generated prompts to a large-scale language model (LLM) that has been trained on a large amount of text data and fine-tuned using instruction-based learning or reinforcement learning with human feedback (RLHF). ○ The LLM used for this purpose can be either a general-purpose model or a domain-specific model, depending on the nature of the task and the desired level of expertise.

[0150] ● G4:LLM outputs raw text predictions containing potential statistical relationships from the target scientific text / publication / document. ● G5: Parses raw predictions, which consist of one or more JSON strings representing structured relationships, into a valid JSON object. ● G6: Clean and verify the individual components of the structured relationship according to the definitions defined herein. Components that fail verification may result in the termination of part or all of the relationship. ● G7: The resulting structured relationships are output via a pipeline and made available for storage and / or other processes.

[0151] Paired relationship (PREx) pipeline Figure 2(d) shows a process flow or data processing pipeline (PREx) for extracting group comparison (paired) statistical relationships from a document corpus, as part of implementing embodiments of the disclosed and / or described systems and methods.

[0152] Group comparisons, or the extraction of paired relationships, allow for the identification and acquisition of relationships defined by two statistical values ​​rather than one. This is the case when two groups or time periods (for example) are directly compared to each other by the authors of a publication. In such situations, the system disclosed and / or described extracts pairs of values ​​whose relationship is clearly measured in terms of significance by a single p-value.

[0153] In one embodiment, the disclosed and / or described paired relationship extraction (PREx) pipeline may include one or more of the following elements, components, functions, or processes, as shown in Figure 2(d):

[0154] ● P1: Sentences identified by the sentence classifier as containing at least two numerical values ​​and one p-value (see, for example, the description of the document candidate identification process in this specification) were classified as positive for containing the expected form of relationships and were classified with a score above a selected threshold (confidence). ● Summary of P2:P1 sentences extracted ● P3: Combine sentences classified as affirmative with their original summaries and one or more prompt questions to generate one prompt per publication or set of texts. ○ As a non-restrictive example, we construct prompts that are input to a Large-Scale Language Model (LLM). Prompt: Identify the relationships between two groups / times and request that information about those relationships be returned in a strict JSON structure. For pairs of numerical values ​​that compare results between groups, or changes in results over time, and are linked by p-values, write the following JSON schema.

[0155] TIFF2026519421000018.tif95169

[0156] ○ Use chained questions to ask the model to validate its own answers to previous questions / prompts. For each JSON object, verify that the JSON is faithful to the source text. Correct any inaccuracies and return the complete JSON. If there is no valid JSON, return "None".

[0157] ● P4: Pass the prompt to LLM. ○ This step involves feeding the generated prompts to a large-scale language model (LLM) that has been trained on a large amount of text data and fine-tuned using instruction-based learning or reinforcement learning with human feedback (RLHF). ○ The LLM used for this purpose can be either a general-purpose model or a domain-specific model, depending on the nature of the task and the desired level of expertise.

[0158] ● P5:LLM outputs raw data predictions. ● P6: Parse - Load a JSON object from the string returned as a prediction from LLM (capable of performing a certain form of GPT). More than two JSON predictions can be returned per prompt / publication. ● P7: Verification - Verify the components of the prediction against their required values ​​(or ranges or other characteristics) and the required form and structure of those values. ○ This may include post-processing, such as filtering / cleaning malformed strings, correcting the summary content, etc., to the extent that this process stage can be automated.

[0159] ● P8: Calculate the difference between the numerical values ​​of each result pair and save it as the average difference in the database. ○ In some embodiments, this may include converting the extracted relationships (e.g., relationships extracted using PREx process flow) into a standard format.

[0160] Relationship extraction does not necessarily involve supervised training of an LLM, but in some embodiments, the methods disclosed and / or described may undergo ongoing analysis on a large number of randomly selected samples from studies (e.g., thousands) to ensure that the model used can accurately extract relationships and other information. In this regard, the models employed by the assignee have consistently achieved 87–90% end-to-end accuracy on relationship extraction tasks.

[0161] Grounding variables in scientific ontology In one embodiment, the Integrated Medical Terminology System (UMLS) was selected as the ontology due to its interoperability with dozens of scientific ontologeries, general ontologeries, and knowledge bases. This selected ontology may be other than UMLS and may be determined by one or more scopes of research described in the publication and the terminology and concepts used in that one or more scopes. Regardless of the selected ontology, the grounding of variables may be carried out in the same or equivalent manner as disclosed and / or described herein.

[0162] Figure 2(e) is a diagram illustrating a process flow or data processing pipeline that “grounds” the identified variables using one or more ontologities, as part of implementing embodiments of the disclosed and / or described systems and methods.

[0163] UMLS Search Index In one embodiment, a search index is constructed using concepts from UMLS, along with their definitions and aliases (as presented in Steps, Stages, Elements, Components, or Process 210). This data is then transformed into a numerical representation known as an embedding (212) (as presented in 211) using a transformer model designed and pre-trained specifically for biomedical literature. Examples of such transformer models include, but are not limited to, BioGPT, as well as various BioBERT, BlueBERT, and PubMedBERT models.

[0164] Link variables to UMLS concepts Variables are similarly transformed into embeddings using the same models and processes. These variables are then linked to UMLS concepts by semantic similarity using transformer models designed and pre-trained specifically for biomedical literature (214). The UMLS knowledge graph is constructed and traversed using the system architectures, platforms, and processes of the assignee disclosed and / or described for statistical searching and other forms of data organization, and may be connected to other scientific ontologeries such as Medical Subject Headings (MeSH), SNOMED-CT, or scientifically inferior sources such as Wikidata, thereby facilitating the understanding and application of statistical relationships in research and practice. The lower flow of the process flow shown in Figure 2(e) represents the same or similar process flow as the flow applied to the variables in the extracted relationships.

[0165] Risk reduction LLM (Generative Specialization Model) has been shown to produce inaccurate results under certain assumptions (known as "hallucination"), which can be problematic for applications that rely on high-quality information. To mitigate this risk, each embodiment may employ one or more of the following precautions.

[0166] 1. Pre-filtering: Using a tagging service to filter out portions of the PubMed corpus that are less likely to contain the information you want to extract. This helps limit the risk of LLM returning answers based on insufficient information.

[0167] 2. Prompt Design: Using structured methods, prompts are generated for the generative model, providing summaries, examples, and hints, and instructing the model not to answer if uncertain. Follow-up validation can also be employed within the same prompt to improve accuracy.

[0168] 3. Rigorous validation of generative model output: Verifying the output of the generative model to confirm that text matches numerical values, and grounding variables using human-curated classifications / ontologies. 4. Post-processing of extracted relationships: Address extraction errors using rule-based logic (this may be combined with or replaced by follow-up prompt generation in some cases). Follow-up prompt generation is driven by ongoing quality checks to gain insights into types and categories of publications requiring special attention.

[0169] Examples of situations where LLM errors may occur include the use of diagnostic studies, where statistics are often used to compare two different diagnostic methods or tools rather than comparing treatments and outcomes. In this category, the disclosed and / or described models may benefit from specific follow-up prompt generation to minimize the risk of errors. Variable names and duplicate variables may be improved to enhance interoperability and improve downstream use.

[0170] The methods disclosed and / or described herein were developed by System Inc. (www.system.com). System Inc. is a non-profit organization specializing in creating platforms that enable users to reference, analyze, and use connections between various forms of available information and data. In non-limiting examples, this may include the impact of treatments and risk factors on medical outcomes, or the impact of socio-economic status on overall health.

[0171] In the context of the embodiments of the systems or platforms disclosed and / or described, statistical relationships serve as essential foundational elements of information and knowledge. This disclosure covers technologies that facilitate the large-scale extraction and representation of these relationships, thereby enabling users to explore and better understand the connections and relationships present in published research and studies. These relationships can be used in a variety of applications, such as meta-analysis, systematic reviews, and knowledge discovery, among others, as non-exclusive examples.

[0172] In contrast, existing methods for extracting statistical relationships from scientific literature have been limited in scope and performance. For example, INDRA (Integrated Network and Dynamical Reasoning Assembler, https: / / www.indra.bio / ) focuses on specific domains with a large amount of human-curated data sources, while other methods primarily focus on extracting semantic relationships rather than statistical ones. Furthermore, current research on general causal relationship extraction has not reached the performance level considered necessary for reliable application to real-world tasks.

[0173] Each embodiment of this disclosure offers several advantages compared to existing solutions, including the elimination of the need for data training for semantic relationship extraction and the relatively small amount of training data required to train the transformers. Furthermore, the structured output generated by one or more embodiments allows for more effective visualization and synthesis of data, providing a more comprehensive understanding of the extracted relationships and the systems in which they form. By addressing and overcoming the limitations of existing solutions, each embodiment significantly improves the efficiency and accuracy of extracting statistical relationships from scientific literature, enabling new applications and insights in both research and practice.

[0174] The following are non-limiting examples of the application of embodiments of the disclosed systems and methods, based on the summary below. Title: Effects of a Novel Therapy on the Incidence of Disease X: A Randomized Controlled Trial Summary: Background: Disease X is a significant public health concern affecting millions of people worldwide. This study aimed to verify the effectiveness of a novel treatment (Treatment A) in reducing the incidence of Disease X compared to placebo. Methods: A randomized controlled trial was conducted in which 500 participants aged 18–65 years were randomly assigned to either Treatment A (n=250) or a placebo (n=250). The primary endpoint was the incidence of Disease X after 12 months of treatment. Results: The incidence of Disease X was significantly lower in Treatment A compared to the placebo group (odds ratio [OR] = 0.65, 95% confidence interval [CI] = 0.45–0.93, p = 0.02). Conclusion: Treatment A significantly reduced the incidence of Disease X compared to placebo, suggesting its potential as an effective measure for preventing Disease X.

[0175] Using this example, the implementation of one or more steps of the disclosure and / or described method is described below. Tagging This system identifies PubMed summaries as candidate texts that contain potential statistical relationships, particularly odds ratios (odds ratio [OR] = 0.65, 95% confidence interval [CI] = 0.45 to 0.93, p = 0.02).

[0176] Prompt construction LLM constructs prompts for extracting the relationship between treatment A and the incidence of disease X, as well as the odds ratio, confidence interval, and p-value.

[0177] Prompt: Provide several example summaries, along with the statistical relationships that you expect to extract from them, and then provide the relevant summary for inference. Extract all statistical relationships from the following PubMed summaries and place them into this JSON schema.

[0178] TIFF2026519421000019.tif29169

[0179] rule: - Spell out all abbreviated terms. - Report the statistics exactly as they are written in the text. Summary: Background: Disease X is a major public health concern… relationship: LLM Reasoning LLM processes prompts and generates raw text predictions that include statistical relationships.

[0180] TIFF2026519421000020.tif29169

[0181] Analysis: The raw text predictions are parsed into a valid JSON object that represents the structured relationship between treatment A and the incidence of disease X. Verification: Clean and verify the individual components of the structured relationships to ensure that odds ratios, confidence intervals, and p-values ​​conform to the required format and values.

[0182] Grounding: Link the variables "Treatment A" and "Incidence of Disease X" to relevant UMLS concepts, such as specific treatment names or disease identifiers. Downstream applications: The extracted relationships can then be used for various purposes, such as meta-analysis, systematic reviews, and research network visualization, which can provide valuable insights into the effectiveness of treatment A in reducing the incidence of disease X.

[0183] As another non-limiting example, Figure 2(f) is a diagram illustrating a statistical relationship extraction process flow (multiple process flows) that may be disclosed and / or described, which is used as part of identifying concepts and related data or metadata for a search by adding a feature graph or knowledge graph of a type enabled by the assignee, and then performing a search using that feature graph or knowledge graph. This may also suggest other or modified searches that may be of interest to the user.

[0184] Searching is a focused task that begins with knowing what you're looking for. Suggested queries may be presented to help users find additional information that might be of interest (e.g., detailed information or simply incidental information). Traditional search engines achieve this by tracking searches across all users, digging up these records, and determining additional searches that might be relevant to the current user's search based on other users' history logs. These solutions rely on the history and prevalence of other users' behavior rather than a systematic or scientific understanding of relevant concepts to contextualize the user's current query.

[0185] Instead, we propose leveraging a knowledge graph or feature graph of statistically grounded relationships (disclosed and / or described in one or more issued patents and / or pending applications assigned to the assignee of this disclosure) to identify concepts relevant to the user's query and to identify the context of the user's search.

[0186] As a non-limiting example, this may be done to identify statistically relevant concepts to recommend to a user using graph architectures disclosed and / or described in one or more of the assignee's issued patents and / or pending applications. This recommendation method is made possible by the disclosed and / or described graph architectures and associated functions for identifying, extracting, encoding, and storing statistical relationships.

[0187] As shown in Figure 2(f), in one embodiment, the user enters a query or search for concept E (as presented in step or stage 240). The user's query is semantically broken down into one or more known concepts or relationships between multiple concepts on the knowledge / feature graph using a vector database (as presented in step or stage 242). The "location" of the desired concept is identified on the knowledge / feature graph (as presented in step or stage 244). An example of a section of the knowledge / feature graph that can be retrieved or identified in response to the search or query is shown as element 246 in the figure.

[0188] Next, the peripheral or "local" relationships between the concept (E) and the searched items (and related metadata) are identified and returned by traversing the graph (step or stage 248), and then ranked according to one or more criteria or rules (step or stage 250). These are done before presenting them to the user (step or stage 252).

[0189] In one non-specific example, one or more recommended searches may be the result of learning the user's interests and providing recommendations that follow the same recommendation logic. To determine which queries to suggest to the user, concepts and their relationships can be ranked by one or more criteria (as shown in step or stage 250 in the diagram), and this is done before presenting them to the user.

[0190] ● Ranking based on relational elements: Directions can be used to filter proposals and surface pathways. For example, if a user searches for the relationship between concept E and concept D, proposals can be limited to concepts and relationships that are upstream of concept D and downstream of concept E.

[0191] ● Furthermore, by reinforcing the proposal based on the depth of information of the given concept or relationship, it is possible to ensure that the proposal made is well-established or, if necessary, reveals a lack of depth (and thus an opportunity to fill in data gaps). As a non-limiting example, ○ Quality or quantity of evidence supporting the relationships in the graph ○ Based on the statistically most unexpectedly relevant ○ Based on identifying higher-level "neighborhoods" of related concepts and recommending concepts from diverse "neighborhoods" (for example, recommending statistically relevant environmental concepts to a user searching for a health topic).

[0192] ● Metadata-based ranking: Other ranking criteria include enhancing suggestions based on metadata extracted from the underlying statistical evidence, such as the relevance of the evidence, the strength of the relationship, indicators of the relationship, and the nature of the relationship (i.e., whether the evidence studies biological or sociological factors, and which is more relevant to the user's query). Journal and author metadata can also be used to enhance relationships based on user preferences. ● Ranking based on search logs: Suggestions can also be ranked based on past user engagement and reputation among other users.

[0193] While several embodiments and examples of the Disclosure have been described and illustrated, further extensions and modifications are possible and are part of the Disclosure. Successful extensions and modifications require tailoring the selection of one or more models and training data to the specific domain and task at hand, selecting models with sufficient bias and nonlinearity to adequately capture real-world patterns in space / domain, and ensuring that the desired task is learnable (i.e., not entirely probabilistic). In non-limiting examples, extensions of the disclosed and / or described process flows, concepts, use cases, and examples may include one or more of the following:

[0194] ● Modifications of the Examples - Use different definitions of statistical relationships and their components. - Use different algorithms and models in one or more steps of one or more processing pipelines. - Use other ontologaries to perform grounding. - Extract the related elements individually and then connect them together (instead of doing it in one step). or - Use a model fine-tuned with tens of thousands to hundreds of thousands of human-labeled annotations.

[0195] ● Other use cases made possible by the embodiments of the disclosed method - Application in other areas such as law and / or finance - The methods disclosed and / or described may be applied to areas outside of life sciences. In such applications, it is considered beneficial to use a dedicated ontology to ground the variables in that field. The methods for defining and extracting statistical relationships are largely the same, and this application can be adapted to the area of ​​interest with minor adjustments to the LLM prompts (e.g., by providing domain-specific examples).

[0196] - Generalization to extract highly accurate structured information (mechanistic relationships, causal relationships, etc.) from unstructured text. - In this regard, the disclosure provides an extensible pattern for extracting structured information from text. Similar to relationships, define a structured and verifiable model for the data. Use a combination of instruction-based and example-based LLM prompts to extract information from unstructured text. Use programmatic tools to validate structured data. or Ground the components of structured data within an ontology or knowledge base appropriate for a specific domain.

[0197] Figure 2(g) shows elements, components, or processes that may reside in or be executed by one or more computing devices, servers, platforms, or systems, which are configured to perform methods, processes, functions, or operations according to some embodiments of the systems and methods disclosed and / or described. In some embodiments, the systems and methods disclosed and / or described may be implemented in the form of one or more devices (such as a server or client device that is part of a system or platform) that include processing elements and a set of executable instructions. The executable instructions may be part of a software application (or a set of applications) and may be located within a software architecture.

[0198] In general, some embodiments of this disclosure may be implemented using a set of software instructions executed by appropriately programmed processing elements (such as GPUs, TPUs, CPUs, microprocessors, processors, controllers, or other forms of computing devices). In complex applications or systems, such instructions are typically arranged in “modules,” each of which typically performs a specific task, process, function, or operation. The operation of a set of modules may be controlled or coordinated by an operating system (OS) or other form of organizational platform.

[0199] Modules and / or submodules may include appropriate computer executable code or instruction sets, such as computer executable code corresponding to a programming language. For example, the source code of a programming language may be compiled into computer executable code. Alternatively, or additionally, the programming language may be an interpreted programming language, such as a scripting language.

[0200] As shown in Figure 2(g), system 200 may represent one or more of the following: a server, a client device, a platform, or other forms of arithmetic or data processing devices. Each module 202 contains an executable instruction set, and when the instruction set is executed by an appropriate electronic processor (such as those indicated in the figure by “physical processors (multiple physical processors) 230”), system (or server, platform, or device) 200 operates to perform a particular process, operation, function, or method (or a combination thereof).

[0201] Module 202 may include one or more sets of instructions for performing methods, operations, processes, or functions disclosed herein and / or described with reference to the drawings and descriptions provided herein. Modules may include those shown, or more or fewer modules than those shown. Furthermore, modules and the sets of computer-executable instructions contained within them may be executed (in whole or in part) by the same processor or by multiple processors. If executed by multiple processors, the processors may be located in separate devices, for example, a processor in a client device and a processor in a server that is part of a platform.

[0202] Module 202 is stored in (non-temporary) memory 220, which typically includes an operating system module 203 containing instructions used to access and control the execution of instructions contained in other modules (among other functions). Module 202 in memory 220 is accessed using a “bus” or communication line 216 for the purpose of data transfer and instruction execution, and the bus or communication line 216 also has the function of enabling a processor(s) 230 to communicate with the module for the purpose of access and instruction execution. The bus or communication line 216 also enables the processor(s) 230 to interact with other elements of the system 200, such as input or output devices 222, communication elements 224 for exchanging data and information with devices outside the system 200, and additional memory devices 226.

[0203] Each module or submodule may include a set of computer executable instructions that, when executed by one or more programmed processors, cause one or more processors (or one or more devices containing them, or one or more servers) to perform a specific function, method, process, or action.

[0204] As described above, the device containing the processor may be a client device, a remote server, or a platform, or one or all of them. Therefore, the module may include instructions that are executed (in whole or in part) by the client device, server, platform, or any of them. Such functions, methods, processes, or operations may include those used to carry out one or more aspects of the systems and methods disclosed and / or described in order to do, for example, the following:

[0205] ● Access the published summary from the document source and begin processing (as presented in Module 204). ○ In one non-exclusive example, this is a PubMed server displaying publications from the NIH (which can be found at https: / / pubmed.ncbi.nlm.nih.gov / ). ○ In one embodiment, this may include performing some kind of filtering or selection of accessed summaries (or other parts of the publication) to identify those that are most likely to contain statistical relationships.

[0206] ● Perform sentence splitting on the set of summaries (as shown in Module 206). ○ This is followed by the use of one or more model-based tagging (labeling) processes and / or pattern-based tagging processes, which have the function of identifying and labeling effect size relationships (e.g., "odds ratio", "OR=1.5", "Pearson r=0.5") and group comparison relationships (e.g., "the proportion of patients with disease X is higher in group A than in group B [10% vs. 5%, p<0.05]"). ■ Use a model-based tagging process to identify / extract group comparison relationships as part of the REx or PREx flow. ■ Use a pattern-based tagging process to identify / extract effect size relationships as part of the REx or GREx flow.

[0207] ● This is followed by the implementation of the statistical relationship extraction process (as presented in Module 208). ○ In one embodiment, the extraction process is carried out as a single process flow or pipeline (referred to herein as REx). ○ In another embodiment, the extraction process is carried out as two separate process flows or pipelines: one for extracting group comparison relationships (referred to herein as PREx) and the other for extracting effect size relationships (referred to herein as GREx).

[0208] ● Provide the output of a relation extraction process (REx) or multiple relation extraction processes (PREx and GREx) to a structured relation process flow (as presented in Module 210). ○ Structured relational process flows have the function of organizing extracted data into a standardized, more uniform structure that is better suited to subsequent processing and evaluation. ● Verify and / or clean the variables obtained from the output of the structured relation process flow (if necessary) (as presented in Module 211).

[0209] ● Perform a semantic grounding process to efficiently clarify and / or expand variable names or identifiers (as presented in Module 212). ○ As a non-restrictive example, this may involve accessing one or more comprehensive ontologs, dictionaries, or thesauruses (for example) to identify analogues or generalized forms of variables, terms, or concepts.

[0210] ● After completing the aforementioned steps or stages, the obtained variables, relationships (group comparisons and effect sizes), and related statistical information are stored in a database for later access and evaluation (as presented in Module 214). Later, by accessing the database and using it to generate a knowledge graph or feature graph (or part thereof), users can traverse that knowledge graph to identify information, data, metadata, relationships, research, or databases that are expected to be relevant when responding to user searches or queries (as presented in Module 215).

[0211] As described herein, the system platform described in U.S. Patent Application No. 16 / 421249, now U.S. Patent No. 11354587, entitled “Systems and Methods for Organizing and Finding Data,” discloses and describes the construction and use of knowledge graphs or feature graphs that help identify and access information that is expected to be valuable due to statistical relationships described in research or studies.

[0212] In some embodiments, the functions and services provided by the systems and methods disclosed and / or described herein may be made available to a large number of users by accessing accounts maintained by a server or service platform. Such a server or service platform may be referred to as a Software-as-a-Service (SaaS) format. Figure 3 is a diagram showing a SaaS system in which one embodiment can be implemented. Figure 4 is a diagram showing elements and components of an example operating environment in which one embodiment can be implemented. Figure 5 is a diagram showing further details of elements or components of the multi-tenant distributed computing service platform of Figure 4 in which one embodiment can be implemented.

[0213] In some embodiments, the systems or services disclosed and / or described herein may be implemented as microservices, processes, workflows, or functions that are executed in response to user responses. These microservices, processes, workflows, or functions may be executed by servers, data processing elements, platforms, or systems. In some embodiments, data analytics and other services may be provided by a service platform located “in the cloud.” In such embodiments, the platform may be accessible through APIs and SDKs. The functions, processes, and capabilities described herein may be provided as microservices within the platform. Interfaces with the microservices may be defined by REST or GraphQL endpoints. An administration console may enable users or administrators to securely access basic request and response data, manage accounts and access, and, optionally, modify processing workflows or configurations.

[0214] Figures 3-5 illustrate a multi-tenant or SaaS architecture that may be used to provide business-related or other applications and services to a large number of accounts / users. However, such an architecture may also be used to provide other types of data processing services and access to other applications. In some embodiments, the type of platform or system illustrated in Figures 3-5 may be operated by a third-party provider to provide a specific set of business-related applications, while in other embodiments, the platform may be operated by a provider, and different businesses may provide applications or services to users through the platform.

[0215] Figure 3 shows a system 300 that can implement one embodiment or access an embodiment of the service disclosed and / or described herein. In accordance with the advantages of an application service provider (ASP)-hosted business service system (such as a multi-tenant data processing platform), users of the service may include individuals, businesses, or organizations. Users may access the service using appropriate clients, including but not limited to, desktop computers 303, laptop computers 305, tablet computers, scanners, or smartphones 304. Users interact with the service platform over the Internet 308 or another appropriate communication network or combination of networks.

[0216] Platform 310, which may be hosted by a third party, may include a set of services that assist users in accessing the data processing and relation extraction services 312 disclosed and / or described herein, and a connected web interface server 314 as shown in Figure 3. Either or both of the services 312 and the web interface server 314 may be implemented on one or more different hardware systems and components, although they are represented as a single unit in Figure 3.

[0217] Service 312 may include one or more functions, processes, or actions that enable the user to access a set of sources, filter those sources, and extract one or both effect size statistical relationships and group comparison statistical relationships from the sources. This may be followed by the user constructing a knowledge graph or feature graph to traverse and identifying potentially useful data, metadata, information, datasets, or other aspects of the source or corpus of sources.

[0218] As an example, in some embodiments, a set of functions, operations, processes, or services that become available through platform 310 may include the following:

[0219] ● Account management service 318. For example. ○ A process or service that authenticates users (in conjunction with the transmission of user credentials using a client device). ○ A process or service that generates a container or instance of a service or application that is made available to users.

[0220] ● A service that accesses and processes document 320. For example. ○ A process or service that accesses a publicly available summary from a document source (with the desired filtering) and initiates processing.

[0221] ○ A process or service that performs sentence splitting on a set of summaries. ■ This is followed by the use of model-based and / or pattern-based tagging (labeling) processes.

[0222] ○ A process or service that performs the statistical relationship extraction process as a single process flow (REx) that extracts both effect size relationships and group comparison relationships, or as two process flows (effect size relationship extraction process flow GREx and group comparison relationship extraction process flow PREx).

[0223] ○ A process or service that provides the output of one or more effect size and group comparison relationship extraction processes to a structured relationship process flow. ○ A process or service that verifies and / or cleans (if necessary) variables obtained from the output of a structured relational process flow.

[0224] ○ A process or service that performs semantic grounding to more efficiently clarify and / or expand variable names or identifiers. ■ This may include accessing one or more comprehensive ontologs, dictionaries, or thesauruses to identify analogous or generalized forms of variables, terms, or concepts.

[0225] ○ A process or service that stores obtained variables, relationships (effect sizes and group comparisons), and related statistical information in a database for later access and evaluation. ■ This may include metadata, links to relevant databases, or other information or data related to the source or document.

[0226] ○ A process or service that accesses a database and generates a knowledge graph or feature graph, enabling a user to traverse that knowledge graph to identify information, data, relationships, metadata, research, or databases that are likely to be relevant when responding to a user search or query.

[0227] ● Management service 326. For example. ○ A process or service that enables a service provider and / or platform to manage and configure the processes and services provided to users. ■ This may include modifying process flows, authentication processes, the format of data output by processes, rules or criteria used to filter a set of sources, and may also enable or modify other related processes, functions, or operations.

[0228] The platform or system shown in FIG. 3 may be hosted on a distributed computing system composed of at least one, but perhaps multiple, "servers". A server is a physical computer dedicated to providing a data storage area and an execution environment for one or more software applications and services aimed at meeting the needs of users of other computers in data communication with the server via a public network such as the Internet. The server and the services it provides may be referred to as "hosts", and the remote computer and the software application running on the provided remote computer may be referred to as "clients". Depending on one or more computing services provided by the server, it may also be referred to as a database server, a data storage server, a file server, a mail server, a print server, or a web server.

[0229] Figure 4 is a diagram showing the elements and components of an example operating environment 400 in which one embodiment can be implemented. As shown, various clients 402 incorporating and / or incorporated within various computing devices may communicate with a multi-tenant service platform 408 through one or more networks 414. For example, a client may incorporate and / or be incorporated within a client application (i.e., software) that is at least partially implemented by one or more computing devices. Examples of suitable computing devices include personal computers, server computers 404, desktop computers 406, laptop computers 407, notebook computers, tablet computers or personal digital assistants (PDAs) 410, smartphones 412, mobile phones, and consumer electronic devices incorporating one or more computing device components such as one or more electronic processors, microprocessors, central processing units (CPUs), or controllers. Examples of suitable networks 414 include networks utilizing wired and / or wireless communication technologies, and networks operating according to any suitable networking and / or communication protocol (e.g., the Internet).

[0230] A distributed computing service / platform (sometimes also referred to as a multi-tenant data processing platform) 408 may include multiple processing tiers, including a user interface tier 416, an application server tier 420, and a data storage tier 424. The user interface tier 416 may hold multiple user interfaces 417, including graphical user interfaces and / or web-based interfaces. The user interfaces may include a default user interface for the service (shown as the "Service UI" in the diagram) that provides access to applications and data for users or service "tenants" (or platform administrators), as well as one or more user interfaces specialized / customized according to user-specific requirements (for example, represented in the diagram by "Tenant A UI", ..., "Tenant Z UI", which may be accessed via one or more APIs).

[0231] The default user interface may include user interface components that enable a tenant to manage access to and use of the functions and capabilities provided by the service platform. This may include access to tenant data, initiation of an instance of a particular application, initiation of execution of a particular data processing operation, and the like. Each application server and processing tier 422 shown in the figure may be implemented as a set of computers and / or components including computer servers and processors, and may perform various functions, methods, processes, or operations as determined by the execution of a software application or set of instructions. The data storage tier 424 may include one or more data stores that may include a service data store 425 and one or more tenant data stores 426. The data store may be implemented with any suitable data storage technology, including a structured query language (SQL)-based relational database management system (RDBMS).

[0232] The service platform 408 may be multi-tenant and may be operated by entities that provide multiple tenants with a set of applications, data storage, and functionality related to the business or other data processing. For example, the applications and functionality may include providing web-based access to functionality used by the business to serve end users, thereby enabling users with a browser and connectivity to the Internet or intranet to view, input, process, or modify certain information. Such functionality or applications are typically held on one or more servers 422, which are part of the platform's application server tier 420, and are implemented by one or more modules of software code / instructions executed by those servers. As described in relation to Figure 3, the platform system shown in Figure 4 may be hosted on a distributed computing system consisting of at least one, but typically more, “servers”.

[0233] As mentioned above, businesses may utilize systems provided by third parties rather than building and maintaining such platforms or systems themselves. These third parties may implement business systems / platforms as described above within the context of a multi-tenant platform, where individual instances of a business's data processing workflow are provided to users, with each business representing a tenant of the platform. One advantage of such a multi-tenant platform is that each tenant can customize the instantiation of its data processing workflow to suit its specific business needs or operational methods. Each tenant may be a business or entity using the multi-tenant platform to provide business services and functionality to multiple users.

[0234] Figure 5 is a diagram showing further details of the elements or components of the multi-tenant distributed computing service platform of Figure 4, in which one embodiment can be implemented. Generally, one embodiment of the present invention may be implemented using a set of software instructions designed to be executed by appropriately programmed processing elements (such as CPUs, GPUs, microprocessors, processors, controllers, or computing devices). In complex systems, such instructions are typically arranged as “modules,” each of which performs a specific task, process, function, or operation. The operation of the entire set of modules may be controlled or coordinated by an operating system (OS) or other form of organizational platform.

[0235] As described above, Figure 5 is a diagram showing further details of elements or components 500 of a multi-tenant distributed computing service platform in which one embodiment can be implemented. The exemplary architecture includes a user interface layer or tier 502 having one or more user interfaces 503. Examples of such user interfaces include graphical user interfaces and application programming interfaces (APIs). Each user interface may include one or more interface elements 504. For example, a user may interact with interface elements to access functions and / or data provided by the application and / or data storage layer of the example architecture. Examples of graphical user interface elements include buttons, menus, checkboxes, drop-down lists, scroll bars, sliders, spinners, text boxes, icons, labels, progress bars, status bars, toolbars, windows, hyperlinks, and dialog boxes. The application programming interface may be local or remote and may include interface elements such as various controls, parameterized continuation calls, program objects, and messaging protocols.

[0236] The application layer 510 may include one or more application modules 511, each module may have one or more submodules 512. Each application module 511 or submodule 512 may correspond to a function, method, process, or operation performed by the module or submodule (for example, a function or process related to data processing and providing services to users of the platform). Such functions, methods, processes, or operations may include those used to perform one or more aspects of the systems and methods disclosed and / or described, for example, the following:

[0237] ● Access the published summary from the source of the document (with the desired filtering) and begin processing. ● Perform sentence splitting on the set of summaries. ○ This is followed by the use of model-based and / or pattern-based tagging (labeling) processes.

[0238] ● The statistical relationship extraction process is implemented as a single process flow (REx) that extracts both effect size relationships and group comparison relationships, or as two process flows (effect size relationship extraction process flow GREx and group comparison relationship extraction process flow PREx). ● Provide the output of one or more effect sizes and group comparison relationship extraction processes to the structured relationship process flow. ● Verify and / or clean the variables obtained from the output of the structured relational process flow (if necessary). ● Perform a semantic grounding process to more efficiently clarify and / or expand variable names or identifiers. ○ This may include accessing one or more comprehensive ontologs, dictionaries, or thesauruses to identify analogous or generalized forms of variables, terms, or concepts.

[0239] ● The obtained variables, relationships (effect size and group comparisons), and related statistical information will be stored in a database for later access and evaluation. ○ This may include metadata, links to relevant databases, or other information or data related to the source or document.

[0240] ● By accessing the database and generating a knowledge graph or feature graph, enable users to traverse that knowledge graph to identify information, data, relationships, metadata, research, or databases that are likely to be relevant when responding to user searches or queries.

[0241] Application modules and / or submodules may include any suitable computer executable code or set of instructions (e.g., one that would be executed by a properly programmed processor, microprocessor, or CPU), such as computer executable code corresponding to a programming language. For example, the source code of a programming language may be compiled into computer executable code. Alternatively, or additionally, the programming language may be an interpreted programming language such as a scripting language. Each application server (e.g., represented by element 422 in Figure 4) may include each application module. Alternatively, separate application servers may include different sets of application modules. Such sets may be separate or partially overlapping.

[0242] The data storage layer 520 contains one or more data objects 522, each data object may have one or more data object elements 521, such as attributes and / or behaviors. For example, a data object may correspond to a table in a relational database, and a data object component may correspond to a column or field in such a table. Alternatively, or additionally, a data object may correspond to a data record having fields and associated services. Alternatively, or additionally, a data object may correspond to a persistent instance of a programmatic data object, such as a structure or a class. Each data store in the data storage layer may contain each data object. Alternatively, separate data stores may contain different sets of data objects. Such sets may be separate or partially overlapping.

[0243] The examples of computing environments depicted in Figures 3-5 are not intended to be limiting examples. Further environments that enable the implementation of embodiments of the present invention, in whole or in part, include devices (including mobile devices), software applications, systems, apparatus, networks, SaaS platforms, IaaS (infrastructure-as-a-service) platforms, or other configurable components that may be used by multiple users for data entry, data processing, application execution, or data review.

[0244] This disclosure includes the following sections and embodiments. 1. A method for extracting information from a document, Access the summary of the published document, Perform a sentence splitting operation on the accessed summary. Apply one or more model-based tagging processes or pattern-based tagging to the sentences determined by the sentence splitting operation to identify one or more text sections relevant to the summary or document content. A statistical relationship extraction process is performed on the selected sentences, and effect size relationships and group comparison relationships are extracted from the summaries. The output of the statistical relationship extraction process is provided as input to the structured relationship process flow, and the structured relationship process flow filters and validates the output of the statistical relationship extraction process. A semantic grounding process is performed on the output of the structured relational process flow to clarify and expand the variable names within the output. The variable names obtained through the semantic grounding process, the extracted statistical relationships, and related statistical information are stored in a database. The system receives a user query that represents the search the user desires, and the query includes topics of interest to the user. Access the database and perform a search on the saved variable names, extracted statistical relationships, and related statistical information. Generate a graph from the results of a search, the graph including a set of nodes and a set of edges, each edge in the set of edges connecting one node in the set of nodes to one or more other nodes, and further, each node representing one of a topic of interest, a variable found to be statistically related to the topic of interest, or a topic found to be statistically or semantically related to the topic of interest, and each edge representing a statistical relationship between a node and a topic of interest, or a statistical relationship between a first node and a second node, including.

[0245] The method according to item 2, further comprising Traverse the graph generated from the results of the search, Identify one or more datasets related to one or more variables that are statistically related to a topic of interest or statistically related to a topic that is semantically related to the topic of interest, Present the results of traversing the graph and the results of identifying one or more datasets to the user, including.

[0246] The method according to item 3, wherein a plurality of published abstracts each having a respective abstract corresponding to a document are accessed and processed. The method according to item 4, wherein the plurality of published abstracts are accessed from a server hosting a plurality of scientific papers or research papers.

[0247] The method according to item 5, wherein the semantic grounding process is performed using one or more ontologies. The method according to item 6, wherein the one or more ontologies include an integrated medical terminology system.

[0248] The method according to claim 1, further comprising performing one or more of the steps of the method on a document related to the abstract. 8. The method according to claim 1, wherein the storage of variable names, extracted statistical relationships, and associated statistical information obtained by a semantic grounding process in a database further includes storing metadata related to the variable names, extracted statistical relationships, or associated statistical information.

[0249] 9. The method described in Section 1, wherein the statistical relationship extraction process extracts effect size relationships and group comparison relationships together in a single process and outputs a JSON object, or the statistical relationship extraction process extracts effect size relationships and group comparison relationships in separate process flows.

[0250] 10. A system, One or more electronic processors configured to execute a set of computer executable instructions, A system comprising one or more non-temporary electronic data storage media containing a computer executable instruction set, wherein when an instruction is executed, the instruction is transmitted to one or more electronic processors, Access the summary of the published document, Perform a sentence splitting operation on the accessed summary. Apply one or more model-based tagging processes or pattern-based tagging to the sentences determined by the sentence splitting operation to identify one or more text sections relevant to the summary or document content. A statistical relationship extraction process is performed on the determined sentences to extract effect size relationships and group comparison relationships from the summary or document. The output of the statistical relationship extraction process is provided as input to the structured relationship process flow, and the structured relationship process flow filters and validates the output of the statistical relationship extraction process. Perform a semantic grounding process on the output of a structured relational process flow to clarify or expand the variable names within the output. The variable names obtained through the semantic grounding process, the extracted statistical relationships, and related statistical information are stored in a database. The system receives a user query that represents the search the user desires, and the query includes topics of interest to the user. Access the database and perform a search on the saved variable names, extracted statistical relationships, and related statistical information. The results of the search are used to generate a graph containing a set of nodes and a set of edges, where each edge in the set of edges connects one node in the set of nodes to one or more other nodes, and each node represents one of the following: a topic of interest, a variable found to be statistically related to the topic of interest, or a topic found to be statistically or semantically related to the topic of interest, and each edge represents a statistical relationship between a node and a topic of interest, or a statistical relationship between the first node and the second node.

[0251] 11. The system described in Section 10, wherein the instruction further relates to one or more electronic processors, By crossing the graphs generated from the search results, Identify one or more datasets associated with one or more variables that are statistically related to a topic of interest, or statistically related to a topic that is semantically related to a topic of interest. The system will present the user with cross-sectional results from graphs and results for identifying one or more datasets.

[0252] 12. One or more non-temporary computer-readable media comprising a set of computer-executable instructions, wherein, when an instruction is executed by one or more programmed electronic processors, the processors... Access the summary of the published document, Perform a sentence splitting operation on the accessed summary. Apply one or more model-based tagging processes or pattern-based tagging to the sentences determined by the sentence splitting operation to identify one or more text sections relevant to the summary or document content. A statistical relationship extraction process is performed on the determined sentences to extract effect size relationships and group comparison relationships from the summary or document. The output of the statistical relationship extraction process is provided as input to the structured relationship process flow, and the structured relationship process flow filters and validates the output of the statistical relationship extraction process. Perform a semantic grounding process on the output of a structured relational process flow to clarify or expand the variable names within the output. The variable names obtained through the semantic grounding process, the extracted statistical relationships, and related statistical information are stored in a database. The system receives a user query that represents the search the user desires, and the query includes topics of interest to the user. Access the database and perform a search on the saved variable names, extracted statistical relationships, and related statistical information. The results of the search are used to generate a graph containing a set of nodes and a set of edges, where each edge in the set of edges connects one node in the set of nodes to one or more other nodes, and each node represents one of the following: a topic of interest, a variable found to be statistically related to the topic of interest, or a topic found to be statistically or semantically related to the topic of interest, and each edge represents a statistical relationship between a node and a topic of interest, or a statistical relationship between the first node and the second node.

[0253] 13. One or more non-temporary computer-readable media as described in Section 16, wherein the instructions are further transmitted to one or more electronic processors. By crossing the graphs generated from the search results, Identify one or more datasets associated with one or more variables that are statistically related to a topic of interest, or statistically related to a topic that is semantically related to a topic of interest. The system will present the user with cross-sectional results from graphs and results for identifying one or more datasets.

[0254] The disclosed systems and methods can be implemented in the form of control logic using computer software in a modular or integrated manner. Based on the disclosures and teachings provided herein, those skilled in the art will understand and comprehend other methods and / or ways of implementing the present invention using hardware and combinations of hardware and software.

[0255] Machine learning (ML) is increasingly being used across various industries to enable data analysis and support decision-making. To benefit from the use of machine learning, a machine learning algorithm is applied to a set of training data and labels to generate a "model," which represents what the algorithm has "learned" from the training data. Each element of the training dataset (or an instance or example in the form of one or more parameters, variables, features, or "features") is associated with a label or annotation that defines how that element should be classified by the trained model. A machine learning model in the form of a neural network is a set of layers of connected neurons that work to make decisions (such as classification) about samples of input data. Once training is complete (i.e., the weights connecting the neurons converge and stabilize or fall within an acceptable range of variability), the model works on new elements of the input data and produces the correct label or classification as output.

[0256] In some embodiments, parts of the methods, models, or functions described herein may be embodied in the form of a trained neural network, which is implemented by executing a set of computer-executable instructions or a representation of a data structure. One or more of the methods, functions, processes, or operations disclosed and / or described herein may be implemented using a trained neural network, a trained machine learning model, or any other form of decision or classification process. The neural network or deep learning model may be characterized in the form of a data structure in which data representing a set of layers containing nodes is stored, connections between nodes of different layers are generated (or formed), and which operates on inputs to provide decisions or values ​​as outputs.

[0257] Generally speaking, a neural network can be thought of as a system of interconnected artificial "neurons" or nodes that exchange messages with each other. These connections have numerical weights that are "tuned" during the training process, so that a properly trained network will respond correctly when presented with (for example) an image or pattern to be recognized. In this characterization, the network consists of multiple layers of feature-detection "neurons," each layer having neurons that respond to various combinations of inputs from the previous layer. The network is trained using a "labeled" dataset of inputs, with a wide range of representative input patterns associated with the intended output responses. In terms of the computational model, each neuron calculates the dot product of its input and weights, adds a bias, and applies a nonlinear trigger or activation function (for example, using a sigmoid response function).

[0258] The software components, processes, or functions disclosed and / or described in this application may be implemented as software code executed by a processor using a suitable computer language such as Python, Java, JavaScript, C, C++, or Perl, using either prior art or object-oriented techniques. The software code may be stored as a set of instructions or commands in (or on) a non-temporary computer-readable medium such as random access memory (RAM), read-only memory (ROM), a magnetic medium such as a hard drive, or an optical medium such as a CD-ROM. A non-temporary computer-readable medium is a medium suitable for storing data or a set of instructions, excluding temporary waveforms. Such computer-readable media may reside on or within a single computing device, or on or within different computing devices in a system or network.

[0259] In one embodiment, the term processing element or processor may refer to a central processing unit (CPU) as used herein, or to a conceptualized CPU (such as a virtual machine). In this embodiment, the CPU or a device incorporating a CPU may connect, link, and / or communicate with one or more peripheral devices such as a display. In another embodiment, the processing element or processor may be incorporated into a mobile computing device such as a smartphone or tablet computer.

[0260] Non-temporary computer-readable storage media as referred to herein may include a number of physical drive units, such as RAID (redundant array of independent disks), flash memory, USB flash drives, external hard disk drives, thumb drives, pen drives, key drives, high-density digital versatile disk (HD-DVD) optical disk drives, internal hard disk drives, Blu-ray optical disk drives, or holographic digital data storage (HDDS) optical disk drives, synchronous dynamic random access memory (SDRAM), or similar devices or forms of memory based on similar technology. Such computer-readable storage media enable processing elements or processors to access computer-executable process steps or application programs stored on removable and non-removable storage media to offload data from or upload data to devices. As stated above, with respect to embodiments disclosed and / or described herein, non-temporary computer-readable media may include structures, techniques, or methods, excluding temporary waveforms or similar media.

[0261] Exemplary embodiments of this disclosure are described herein with reference to block diagrams of systems and / or flowcharts or flow diagrams of functions, operations, processes, or methods. One or more blocks in a block diagram, or one or more stages or steps in a flowchart or flow diagram, and combinations of blocks in a block diagram and stages or steps in a flowchart or flow diagram may each be executed by computer executable program instructions. In some embodiments, one or more of the blocks or stages or steps do not necessarily have to be executed in the order presented, or may not have to be executed at all.

[0262] Computer executable program instructions may be loaded into a general-purpose computer, a special-purpose computer, a processor, or other programmable data processing device to generate a specific example of a machine. Instructions executed by a computer, processor, or other programmable data processing device create means to perform one or more of the functions, operations, processes, or methods disclosed and / or described herein. Computer program instructions may be stored in (or on) computer-readable memory that can be instructed to function in a particular manner, so that when the instructions are executed, the instructions will produce a product containing instruction means to perform one or more of the functions, operations, processes, or methods disclosed and / or described herein.

[0263] While each embodiment of this disclosure has been described in relation to what is currently considered the most practical methods and techniques, the embodiments are not limited to those disclosed. Rather, the disclosed embodiments are intended to include and encompass modifications and equivalent configurations that fall within the scope of the appended claims. Certain terms are used herein, but these are used solely for general and descriptive purposes and not for limiting purposes.

[0264] This specification uses examples to illustrate one or more embodiments of the disclosure and to enable a person skilled in the art to practice the disclosed methods and techniques, including manufacturing and using a device or system and performing related methods. The patentable scope of the disclosure is defined by the claims and may include other examples that a person skilled in the art can conceive. Such other embodiments are intended to be within the scope of the claims if they have structural and / or functional elements that are not inconsistent with the language of the claims, or if they include structural and / or functional elements that are substantially inconsistent with the language of the claims.

[0265] All references cited herein, including publications, patent applications, and patents, are incorporated by reference to the same extent as they are individually and specifically indicated, and / or as are included herein.

[0266] In this specification and claims, the use of the words “a and an” and “the,” and similar references, shall be construed as encompassing both singular and plural unless otherwise specifically indicated herein or the context clearly contradicts this. In this specification and claims, the words “having,” “including,” and “containing,” and similar references, shall be construed as open-ended terms (e.g., “including, but not limited to”) unless otherwise specified.

[0267] The descriptions of value ranges herein are merely abbreviations intended to refer individually to each separate value encompassed within the range, unless otherwise specified herein, and each separate value is incorporated herein as if it were individually listed herein. The method steps or stages disclosed and / or described herein may be performed in any suitable order unless otherwise specified herein or unless it is clearly inconsistent with the context.

[0268] The use of examples and illustrative language (e.g., “such as”) within this specification is intended to illustrate the embodiments of this disclosure and, unless otherwise indicated, does not impose any limitation on the scope of the claims. Nothing in this specification should be construed as indicating that any element not within the scope of the claims is essential to the embodiments of this disclosure.

[0269] As used in this application (i.e., the claims, drawings, and specification), the word “or” is used inclusively to refer to items selectively and in combination.

[0270] Different arrangements of elements, structures, components, or steps depicted in the drawings or described herein, as well as components and steps not shown or described herein, are available. Similarly, some features and sub-combinations are useful and may be adopted without reference to other features or sub-combinations. Each embodiment is described for illustrative purposes only and not to limit, and alternative embodiments may become apparent to the reader of this specification. Accordingly, this disclosure is not limited to the embodiments described herein or depicted in each figure, and modifications may be made without departing from the scope of the appended claims.

Claims

1. A method for extracting information from a document, Access the summary of the published document, Perform a sentence splitting operation on the accessed summary. Apply one or more of the model-based tagging process or pattern-based tagging to the sentence determined by the sentence splitting operation, and identify one or more text sections related to the content of the summary or document. A statistical relationship extraction process is performed on the determined sentences, and effect size relationships and group comparison relationships are extracted from the summary or the document. The output of the statistical relationship extraction process is provided as input to the structured relationship process flow, and the structured relationship process flow filters and verifies the output of the statistical relationship extraction process. A semantic grounding process is performed on the output of the structured relation process flow, clarifying or expanding the variable names within the output. The variable names obtained through the semantic grounding process, the extracted statistical relationships, and related statistical information are stored in a database. The system receives a user query that represents the search desired by the user, and the query includes topics of interest to the user. Access the aforementioned database and perform the search on the saved variable names, extracted statistical relationships, and related statistical information. A method comprising generating a graph from the results of performing the search, wherein the graph includes a set of nodes and a set of edges, each edge in the set of edges connecting one node in the set of nodes to one or more other nodes, each node representing one of the following: the topic of interest, a variable found to be statistically related to the topic of interest, or a topic found to be statistically or semantically related to the topic of interest, and each edge representing a statistical relationship between a node and the topic of interest, or a statistical relationship between a first node and a second node.

2. moreover, The graph generated from the results of the aforementioned search is crossed, Identify one or more datasets related to one or more variables that are statistically related to the topic of interest, or statistically related to a topic that is semantically related to the topic of interest, The method according to claim 1, further comprising presenting to the user the cross-sectional results of the graph and the results of identifying one or more datasets.

3. The method according to claim 1, wherein a plurality of published summaries, each having a summary corresponding to a document, are accessed and processed.

4. The method according to claim 3, wherein the aforementioned multiple published abstracts are accessed from a server hosting multiple scientific or research papers.

5. The method according to claim 1, wherein the semantic grounding process is performed using one or more ontologities.

6. The method according to claim 5, wherein the one or more ontologies include an integrated medical terminology system.

7. The method according to claim 1, further comprising performing one or more of the steps of the method on the document relating to the summary.

8. The method according to claim 1, wherein the storage of the variable names, extracted statistical relationships, and associated statistical information obtained by the semantic grounding process in a database further includes storing metadata related to the variable names, extracted statistical relationships, or associated statistical information.

9. The method according to claim 1, wherein the statistical relationship extraction process extracts the effect size relationship and the group comparison relationship together in a single process and outputs a JSON object, or the statistical relationship extraction process extracts the effect size relationship and the group comparison relationship in separate process flows.

10. One or more electronic processors configured to execute a set of computer executable instructions, A system comprising one or more non-temporary electronic data storage media having a set of computer executable instructions, When the instruction is executed, the instruction is directed to one or more electronic processors, Access the summary of the published document, Perform a sentence splitting operation on the accessed summary. Apply one or more of the model-based tagging process or pattern-based tagging to the sentence determined by the sentence splitting operation, and identify one or more text sections related to the content of the summary or document. A statistical relationship extraction process is performed on the determined sentences, and effect size relationships and group comparison relationships are extracted from the summary or the document. The output of the statistical relationship extraction process is provided as input to the structured relationship process flow, and the structured relationship process flow filters and verifies the output of the statistical relationship extraction process. A semantic grounding process is performed on the output of the structured relation process flow, clarifying or expanding the variable names within the output. The variable names obtained through the semantic grounding process, the extracted statistical relationships, and related statistical information are stored in a database. The system receives a user query that represents the search desired by the user, and the query includes topics of interest to the user. Access the aforementioned database and perform the search on the saved variable names, extracted statistical relationships, and related statistical information. A system that generates a graph from the results of the aforementioned search, the graph comprising a set of nodes and a set of edges, each edge in the set of edges connecting one node in the set of nodes to one or more other nodes, each node representing one of the following: the topic of interest, a variable found to be statistically related to the topic of interest, or a topic found to be statistically or semantically related to the topic of interest, and each edge representing a statistical relationship between a node and the topic of interest, or a statistical relationship between a first node and a second node.

11. The instruction further directs one or more electronic processors to: The graph generated from the results of the aforementioned search is crossed, Identify one or more datasets related to one or more variables that are statistically related to the topic of interest, or statistically related to a topic that is semantically related to the topic of interest, The system according to claim 10, which causes the system to present to the user the cross-sectional results of the graph and the results of identifying one or more datasets.

12. The system according to claim 10, wherein multiple published summaries, each having a corresponding summary for a document, are accessed and processed.

13. The system according to claim 10, wherein the semantic grounding process is performed using one or more ontologities.

14. The system according to claim 10, wherein the instructions further cause one or more electronic processors to perform one or more of the steps performed on the document relating to the summary.

15. The system according to claim 10, wherein the statistical relationship extraction process extracts the effect size relationship and the group comparison relationship together in a single process and outputs a JSON object, or the statistical relationship extraction process extracts the effect size relationship and the group comparison relationship in separate process flows.

16. One or more non-temporary computer-readable media comprising a set of computer-executable instructions, wherein, when an instruction is executed by one or more programmed electronic processors, the processors... Access the summary of the published document, Perform a sentence splitting operation on the accessed summary. Apply one or more of the model-based tagging process or pattern-based tagging to the sentence determined by the sentence splitting operation, and identify one or more text sections related to the content of the summary or document. A statistical relationship extraction process is performed on the determined sentences, and effect size relationships and group comparison relationships are extracted from the summary or the document. The output of the statistical relationship extraction process is provided as input to the structured relationship process flow, and the structured relationship process flow filters and verifies the output of the statistical relationship extraction process. A semantic grounding process is performed on the output of the structured relation process flow, clarifying or expanding the variable names within the output. The variable names obtained through the semantic grounding process, the extracted statistical relationships, and related statistical information are stored in a database. The system receives a user query that represents the search desired by the user, and the query includes topics of interest to the user. Access the aforementioned database and perform the search on the saved variable names, extracted statistical relationships, and related statistical information. A non-temporary computer-readable medium that generates a graph from the results of the search, the graph comprising a set of nodes and a set of edges, each edge in the set of edges connecting one node in the set of nodes to one or more other nodes, each node representing one of the following: the topic of interest, a variable found to be statistically related to the topic of interest, or a topic found to be statistically or semantically related to the topic of interest, and each edge representing a statistical relationship between a node and the topic of interest, or a statistical relationship between a first node and a second node.

17. The instruction further directs one or more electronic processors to: The graph generated from the results of the aforementioned search is crossed, Identify one or more datasets related to one or more variables that are statistically related to the topic of interest, or statistically related to a topic that is semantically related to the topic of interest, One or more non-temporary computer-readable media according to claim 16, which causes the cross-sectional results of the graph and the identification results of one or more datasets to be presented to the user.

18. One or more non-temporary computer-readable media according to claim 16, wherein multiple published summaries, each having a corresponding summary of a document, are accessed and processed.

19. The semantic grounding process is performed using one or more ontologities in one or more non-temporary computer-readable media according to claim 16.

20. The instructions further cause one or more electronic processors to perform one or more of the steps performed on the document relating to the summary, according to one or more non-temporary computer-readable media according to claim 16.

Images