Systems and methods for ETL pipeline processing

Abstract

The present invention provides an ETL pipeline system including an interface configured to retrieve a plurality of graph component parts. Each graph component part includes information representing one or more computational logic rules. The system further includes a computation graph generator configured to generate a computation graph based on the retrieved graph component parts. The generated computation graph includes nodes for each graph component part and one or more links representing relationships between the nodes. The system further includes a computation graph adapter configured to receive external information related to the graph component parts from an external source and adapt the generated computation graph based on the external information. The system further includes a computation graph runner configured to execute the adapted computation graph.

Description

[Technical Field]

[0001] The present invention relates to the field of data processing, and in particular to the field of extract, transform, load (ETL) pipeline processing. [Background technology]

[0002] Extract, transform, and load (ETL) is a pipeline (i.e., a series of processes) used by data processing and analysis systems to collect data from various sources, transform the data according to business requirements, and load the data into a destination data store. In an ETL system, the transformations are performed in specialized engines, and staging tables are typically used to temporarily hold the data as it is transformed and finally loaded into its destination. The process of data transformation involves many traditional operations such as filtering, sorting, aggregating, joining, cleaning, deduplication, and validation.

[0003] For medical data applications, complex data transformations need to be performed to generate meaningful insights. Yet, these transformations must first rely on data (i.e., data elements) being structured into tables of phenotypes before the data can be utilized for more advanced processing. Thus, in traditional medical data processing systems, an extraction step involves extracting useful phenotypes. Transformation refers to the calculations and derivations performed based on the extracted phenotypes. The output of a transformation can be a new phenotype derived from existing phenotypes, or the result of computing a decision rule, such as for diagnosis or treatment selection. Summary of the Invention [Problem to be solved by the invention]

[0004] However, while existing ETL systems widely adopt natural language processing (NLP) techniques for phenotype extraction, their architectures to support knowledge-driven computation in the transformation stage are quite limited. To handle advanced computations after ETL is completed, separate systems or applications are usually implemented, which results in inconsistency and more overhead.

[0005] Data extraction is traditionally handled by a separate NLP module. As a result, phenotype extractors are often separated from secondary uses of the extracted data (such as data aggregation). This makes it inefficient to have to compare the performance of multiple phenotype extraction algorithms.

[0006] Graph structures have traditionally been used to encode clinical knowledge and are useful for many applications because they can capture complex relationships between medical concepts. However, graph representations themselves are not computable. Traditional ETL pipelines lack a framework that allows knowledge to be leveraged to accelerate computations. This is typically achieved by hard-coding programming rules during data transformation. As a result, this method lacks the flexibility to accommodate changes (e.g., changes in input data types, algorithms, and decision logic). [Means for solving the problem]

[0007] The invention is defined by the claims.

[0008] According to an example embodiment of the present invention, an ETL pipeline system is provided that includes an interface configured to retrieve a plurality of graph component elements. Each graph component element includes information representing one or more computational logic rules. The system further includes a computation graph generator configured to generate a computation graph based on the retrieved graph component elements. The generated computation graph includes nodes for each graph component element and one or more links representing relationships between the nodes. The system further includes a computation graph adapter configured to receive external information related to the graph component elements from an external source and adapt the generated computation graph based on the external information. The system further includes a computation graph runner configured to execute the adapted computation graph.

[0009] An ETL architecture concept, particularly useful for medical data processing, is proposed in which phenotyping and clinical knowledge computing are embedded in a single, coherent architecture. Such a concept allows clinical knowledge to be constructed independently of the actual clinical data. However, during runtime, clinical data flows through the architecture, where database operations, phenotyping algorithms, and decision logic are applied to the data flow to achieve the desired computational results.

[0010] It is proposed to provide a graph configuration file (e.g., DSL) for intuitively constructing computing graphs. It is further proposed to provide a graph driver for parsing, constructing, and executing computational graphs. It is further proposed to provide an information modeling technique for encoding relationships between phenotypes and clinical documents. The proposed back-end graph data structure can be integrated with fact-based and database-based knowledge graphs to serve downstream applications. Furthermore, applications of multi-layer computation, local recomputation, and adaptive computation are proposed for application in ETL pipeline systems.

[0011] A concept for an ETL architecture that is particularly useful for medical data processing has been proposed. In particular, the claimed invention provides an ETL pipeline system including an interface configured to retrieve multiple graph components. Each graph component contains information representing one or more computational logic rules. The proposed invention generates a computation graph based on the retrieved graph components, where the generated computation graph includes nodes for each graph component and one or more links representing relationships between the nodes. External information related to the graph components (e.g., user-defined graph components and / or prior knowledge) is employed to adapt the generated computation graph.

[0012] In one embodiment, each graph configuration component includes at least one of a database connection, a document, a data element, an algorithm, and a computational logic rule.

[0013] In one embodiment, the algorithm comprises a computational decision rule and / or the computational logic rule comprises a computational decision rule.

[0014] In one embodiment, the external information includes at least one of a user-defined graph construction component and external knowledge from an external knowledge database. By way of example, the external knowledge includes a previous / previous computation graph stored in a database. Such previous / previous computation graph is added to the generated computation graph to form an adapted computation graph. That is, the generated computation graph is adapted by incorporating the previous / previous computation graph. By way of further example, the user-defined graph construction component includes a graph construction component configured to redefine or modify the computation graph.

[0015] In one embodiment, the interface includes at least one of a user interface and an application programming interface.

[0016] In one embodiment, the system further includes a graph configuration file, and the user-defined graph configuration component includes redefining the graph configuration file or dependency information indicating dependencies between different graph configuration components.

[0017] In one embodiment, the system further comprises a computational graph parser configured to receive graph configuration components from the interface; and establish one or more links between the different graph configuration components based on the component information and dependency information indicating dependencies between the different graph configuration components.

[0018] In one embodiment, the computational graph parser is further configured to establish links between the data elements and the documents based on data items associated with the data elements and / or the documents.

[0019] In one embodiment, the data item includes at least one of an identifier, a timestamp, a content item, a context item, and an encoded object.

[0020] In one embodiment, the computation graph runner is further configured to execute the adapted computation graph in an adaptive high throughput computation.

[0021] According to an example aspect of the present invention, there is provided a medical data processing system including the ETL pipeline system described above.

[0022] According to an example aspect of the present invention, there is provided a method for ETL pipeline processing. The method includes the step of acquiring a plurality of graph components, each graph component including information representing one or more computational logic rules. The method further includes the step of generating a computation graph based on the acquired graph components, the generated computation graph including nodes for each graph component and one or more links representing relationships between the nodes. The method further includes the step of receiving external information related to the graph components from an external source. The method then includes the steps of adapting the generated computation graph based on the external information and executing the adapted computation graph.

[0023] According to an exemplary aspect of the present invention, a computer program product for ETL pipeline processing is provided, the computer program product comprising a computer-readable storage medium having program instructions thereon executable by a processing unit to cause the processing unit to perform a method. The method includes obtaining a plurality of graph components, each graph component including information representing one or more computational logic rules. The method further includes generating a computation graph based on the obtained graph components, the generated computation graph including nodes for each graph component and one or more links representing relationships between the nodes. The method further includes receiving external information related to the graph components from an external source. The method then includes adapting the generated computation graph based on the external information and executing the adapted computation graph.

[0024] According to an example aspect of the present invention, there is provided a processing system comprising at least one processor and a computer program product as described above, wherein the at least one processor is adapted to execute computer program code of the computer program product.

[0025] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

[0026] For a better understanding of the present invention, and to show more clearly how it may be carried into effect, reference should be made, by way of example only, to the accompanying drawings in which: [Brief explanation of the drawings]

[0027] [Figure 1] A simplified block diagram of an ETL pipeline system. [Figure 2] A simplified block diagram of state transition flow in an ETL pipeline system. [Figure 3] FIG. 1 illustrates the dependencies between different graph configuration components. [Figure 4] FIG. 10 is a diagram illustrating an example of a generated computation graph. [Figure 5] FIG. 1 is a simplified flow diagram of a method for ETL pipeline processing according to an embodiment. [Figure 6] FIG. 1 illustrates a simplified block diagram of a knowledge computing structure, specifically an adaptive computing structure, implemented in an ETL pipeline system. [Figure 7] FIG. 1 shows a simplified block diagram of a knowledge computing structure, specifically a multi-tier computer structure, implemented in an ETL pipeline system. [Figure 8] FIG. 1 illustrates an example computer implementing a controller or processor according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0028] The present invention will be described with reference to the drawings.

[0029] While exemplary embodiments of the devices, systems, and methods are set forth, it should be understood that the detailed description and specific examples are for purposes of illustration and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the devices, systems, and methods of the present invention will become better understood from the following description, appended claims, and accompanying drawings. It should be understood that the figures are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the figures to indicate the same or similar parts.

[0030] The present invention provides an ETL pipeline system including an interface configured to retrieve a plurality of graph component parts. Each graph component part includes information representing one or more computational logic rules. The system further includes a computation graph generator configured to generate a computation graph based on the retrieved graph component parts. The generated computation graph includes nodes for each graph component part and one or more links representing relationships between the nodes. The system further includes a computation graph adapter configured to receive external information related to the graph component parts from an external source and adapt the generated computation graph based on the external information. The system further includes a computation graph runner configured to execute the adapted computation graph.

[0031] Figure 1 shows an ETL pipeline system according to an embodiment. The system includes interfaces 11 and 12 for users to interact with the system, which helps users generate (high-level) computable knowledge representations according to their specific requirements. The system in Figure 1 includes two interfaces 11 and 12. Interface 12 is a user interface (UI) that enables "drag-and-drop" of key components in the system front-end, accelerating computational graph generation. Interface 11 is an application programming interface (API) that serves the same purpose as the user interface 12 but interacts at a programming level. API 11 primarily supports three actions. First, submission of a specially designed graph configuration file (GCF) 13, which is subsequently converted into a computable graph. Second, submission and saving / loading of user-defined (UDF) algorithms 14 for database transactions, data calculations, and NLP phenotyping to a central function repository. Third, communication with a common graph database 15, which allows the generated graph structures to be saved in the graph database 15 or existing graphs to be loaded into the system from the graph database 15.

[0032] There are state transition flows between the above mentioned components. The graph configuration is transformed (both via UI and API) into a basic data structure of a directed acyclic graph (DAG). The DAG can be transformed to and from the data structures of common graph databases (e.g. Neo4j, GraphDB, and Cayley). So far, the DAG is a static graph structure that encodes the information required for execution. The DAG is further transformed into an executable graph that can be executed at runtime.

[0033] A graph configuration file (GCF)13 is a container for graph structures that perform both data extraction and decision rule computation tasks. By following a specially designed framework, users can declare or define a set of major components—database connections, documents, representation types, algorithms, and computation logic—to perform a specific computational task and organize them into GCF13 groups. GCF13s can be generated as human-readable text files in JSON and YAML formats. In each file, users define a structure containing the major components to represent a single node in the graph. Multiple nested nodes can be defined within the same file or distributed across several files in the same folder. Previously defined GC nodes, whether in the same file or in external files in the same folder, can be referenced by name during parsing.

[0034] GCF13 is configured at the programming level to incorporate existing knowledge into a common graph structure supported by the system, which can sometimes be difficult and / or time-consuming for a human to curate via a UI. For example, the BCLC classification system and the Child-Pugh classification system are persisted in the system via GCF and can be calculated at runtime to derive Child-Pugh staging for liver cancer patients.

[0035] The Graph Construction UI 12 serves as an alternative to the GCF 13 and allows intuitive construction of graph representations of clinical knowledge and decision rules. The UI Adapter 16 is responsible for converting front-end input into a primitive data structure (DAG). The UI allows the user to construct a graph by creating nodes and links. For each node, the user can either specify an algorithm to process the data flow through that node or implement decision logic that takes input from the parent node of the current node.

[0036] The graph driver 10 is at the heart of the system. It is responsible for constructing a computation graph capable of capturing data flows, executing the graph, and delivering the desired results to the user. The graph driver 10 has three main components: a GCF parser (GCFP) 17, a graph constructor (GST) 18, a graph runner (GRN) 19, and two adapters 16 and 20. While the implementation of the graph driver is not limited to a specific programming language, developers can implement language-specific drivers by following the same philosophy and common interfaces for each system component.

[0037] Taking as input a configuration file 13 (formatted in JSON, YAML or XML), GCFP 17 converts the graph definition in file 13 into individual components (e.g., database connections, documents, representation types, algorithms and computational logic) under corresponding namespaces, and then links the namespaces together as linked objects in preparation for graph construction.

[0038] More specifically, each GCF 13 includes definitions of different graph components. By way of example, FIG. 3 defines an exemplary set of dependencies between different graph components associated with various namespace components. More specifically, the solid line labeled "A" defines a dependency between a database and a document, the solid line labeled "B" defines a dependency between an algorithm and logic, the solid line labeled "C" defines a dependency between a document and logic, and the solid line labeled "D" defines a dependency between a representation and logic. Such dependencies are defined by the user.

[0039] The GCFP 17 receives graph configuration components from the interface 11 and then establishes links between different graph configuration components based on the component information and dependency information indicating the dependency relationships between the different graph configuration components.

[0040] GST 18 can then convert the linked components into graph nodes and relationships. That is, GST 18 takes the linked namespace objects and builds a basic data representation in which the linked components are converted into graph nodes and relationships between the nodes in the form of a DAG. Furthermore, during the construction process, GST 18 frequently interacts with the graph configuration API to load / save components from / to the function repository (a database that holds all function objects). If the specified component in GCF 13 already exists in this repository, the component reference is fetched directly from that repository. Otherwise, the specified component is new, and GST 18 then saves the component in the function repository for reuse.

[0041] The GRN 19 initially acts as an injector that injects all the information necessary to execute the graph, such as code objects and recursively defined GC components. On the other hand, the graph runner 19 is a connector that incorporates the graph into an existing implementation of a graph-based parallel execution system. As an example, Figure 4 shows an example of a generated computation graph. The generated computation graph is output via interface 11 or 12. The generated computation graph includes nodes 36 for each graph component and one or more links 37 that represent the relationships between the nodes 37.

[0042] Two adapters are included in the driver 10: a graph database adapter 20 and a UI adapter 16. The UI adapter is responsible for converting front-end input into a DAG structure and vice versa, and presenting the constructed DAG to the front-end user. The graph DB adapter also matches the data structures used by our system and existing graph DB systems.

[0043] Referring to Figure 2, a diagram of state transition flow between system components is shown in accordance with an embodiment. From the GCF 13 and a graph configuration 33 provided via a user interface, a DAG 32 is constructed. The DAG 32 can be converted to or from a knowledge graph 34 data structure. The DAG 32 is further converted to an executable graph 35 that can be executed at run time.

[0044] Referring now to FIG. 5, a simplified flow diagram of a method for ETL pipeline processing according to an embodiment is shown. The method begins at step 21 by obtaining a plurality of graph components, each of which includes information representing one or more computational logic rules. Next, at step 22, a computation graph is generated based on the obtained graph components. The generated computation graph includes nodes for each graph component and one or more links representing relationships between the nodes. Thereafter, at step 23, external information related to the graph components is received from an external source. Then, at step 24, the generated computation graph is adapted based on the external information. Finally, at step 25, the adapted computation graph is executed.

[0045] More specifically, the external information includes at least one of a user-defined graph construction component and external knowledge from an external knowledge database. In one embodiment, the external knowledge includes a prior computation graph stored in a database, which is added to the generated computation graph to form the adapted computation graph. In another embodiment, the user-defined graph construction component includes redefining or modifying the graph construction component via an interface.

[0046] The subject matter uses a graphical user interface to redefine graph-forming components or links between graph-forming components of the generated computation graph. For example, such redefinition is accomplished by dragging nodes, deleting links, and reconnecting links between different nodes in an interactive visual representation of the computation graph (e.g., as shown in FIG. 4).

[0047] In another embodiment, a user-defined graph configuration component includes one or more definitions for redefining the graph configuration file and / or the dependencies between different graph configuration components. More specifically, a subject can define different logic (by changing @logic), define different algorithms (by changing @algorithm), or add new components, i.e., subject dependencies.

[0048] To further illustrate, we first describe the GCF grammar and GCF parser that converts configuration files 13 into linked namespace components. For parsing, we introduce each building block. For graph construction, we explain the details of DAG formation. We then detail the data model design that handles the relationship between representation types and documents during the graph construction process. We then explain how existing knowledge graphs can be combined with the proposed embodiment to obtain meaningful results. Finally, we provide details on how the embodiment is employed for efficient computation.

[0049] GCF Grammar The GCF grammar supports the declaration and definition of major components, which begin with the notation "@" followed by the component category. The major components are divided into five different categories: database, representation, document, algorithm, and logic. Each serves a different purpose and can be organized into independent structures. A detailed description of the five components is provided in the next description section below. The difference between declaration and definition depends on whether the component has already been created. To declare a component, only the component name followed by "@" is needed.

[0050] To define a component, the "@def" keyword must be inside the body of the component. To save a defined component, a name must be specified using the "@name" keyword, and the "@save" keyword with value set to "True" should be added. The user attaches a function implementation as a value to "@def", and the function definition is persisted in a central repository. The saved component can then be declared as described above. The GCF grammar supports nested definitions of components, but is constrained by namespace dependencies.

[0051] The following example illustrates the task of detecting malignant tumors given clusters of radiology report data and tumor biomarkers.

[0052] An exemplary GCF performing the task of detecting malignant tumors:

number

[0053] The goal is to know whether there are any malignant tumors (MT) found within each radiology report cluster. The decision rule stipulates that if more than two radiology reports in a cluster contain mention of malignant tumors, it returns true. In another case (fewer than two radiology reports), it also returns true if an abnormal AFP (tumor biomarker) is found. Otherwise, it returns false. In the above example, "@Document:input_data" represents the list of clusters of radiology reports (reports clustered together within two months). "@Document:tumor_markers" injects tumor biomarker information into the document. For imported documents, "@Phenotype:tumor_mention" and "@Phenotype:report_time" are obtained from the corresponding document object according to the document-phenotype relationship model. "@Algorithm:is_MT" and "@Algorithm:select_marker" trigger two decision functions to be imported from the repository during graph construction. "select_marker" is an algorithm for selecting the latest biomarker data recorded in the radiology report in the cluster. "is_M" is an algorithm for determining whether a tumor is malignant according to "@phenotype:tumor_mention". The decision rule is implemented within "@def" in the "@Logic" block. "@save" indicates that this logic will be saved in the function repository under the name "detect_radiology_reports". Next time, this saved function can be directly imported by declaring "@Algorithm:detect_radiology_reports". However, all the imports and code defined so far will not be executed until the graph is constructed, converted into an executable graph, and executed.

[0054] Namespace Components There are five functional component objects corresponding to the five main items defined by the GCF grammar. The database component is responsible for handling database operations, transactions such as database connections and data queries. A phenotype is a key-value entity that is meaningful for a specific decision rule computation task. A document represents a clinical report containing a set of phenotypes. An algorithm is a specific function that processes inputs and obtains outputs. For example, a data extraction algorithm extracts relevant phenotype values ​​from clinical narrative data. Logic is also a function like an algorithm, except that the logic is responsible for the transitions between different nodes.

[0055] The inventors have devised a bidirectional relational model to represent representations and documents. Each representation and document contains mandatory data items: name / ID, timestamp, and content, and optional data items: context and encoding. However, content and context have different meanings for representations and documents, respectively. For documents, content represents the content of the report, whether in narrative or structured / semi-structured form. For representations, content is the value of the representation. For representations, context is the content of the document containing the context. For timestamps, the timestamp of a representation is considered equal to the timestamp of the document, although this is not always the case. The encoding represents an associated standardized code. This code usually indicates the category of the representation or document entity.

[0056] DAG The phenotypes are fetched by an NLP phenotyping algorithm that employs the exemplary GCF provided above and models the phenotyping results using the relationship information model described above for phenotypes and documents. The NLP algorithm is not explicitly declared in the exemplary GCF above; this is likely done through some pre-configuration. In this example, it is explicitly shown that phenotypes are extracted from documents using the NLP algorithm as a function node in the DAG. Again, this DAG representation is static; nothing is computed until the graph becomes an executable graph and is executed.

[0057] ·Connection with knowledge / ontology We now elaborate on the connection between the graph structure and graph data in external knowledge graph databases. Two types of knowledge graph databases can be connected: relation-based knowledge bases and data-based knowledge bases. Knowledge-based connections have the potential to extend the results of graph processing to a larger scope by linking entities in the graph with external entities. This is useful when building intelligent search systems or question-and-answer (QA) systems.

[0058] - Factual knowledge base The fact graphs are existing knowledge graphs and ontologies such as SNOMED-CT, LOINC, MeSH, Drugbank, etc. By connecting the data nodes of this DAG to such external graphs and ontologies, concept standardization is performed, and a major advantage is that the internal and external graphs can be bridged to achieve automatic knowledge extension. This facilitates the development of processing systems that behave as if the semantics of biomedical and healthcare languages ​​are understood.

[0059] - Data-driven knowledge base Fast Healthcare Interoperability Resources (FHIR) is a healthcare standard that defines a set of data resources and aims to facilitate rapid data exchange between healthcare systems. A FHIR database is essentially a knowledge base with different data resources that can be linked together as a graph. FHIR supports the Turtle format, which can be used to store data as RDF triples.

[0060] ·Knowledge calculation Next, details of the knowledge computing strategy are given. The proposed embodiment employs: (i) adaptively selecting the best computation strategy according to data dependencies between two nodes in the graph to improve efficiency; (ii) organizing the accumulated data and knowledge into a multi-tiered structure that is computationally efficient and easy to track for the user; and (iii) local recomputation, which allows graph components that can be replaced with existing computation graphs to efficiently optimize system performance.

[0061] - Adaptive Computational Structure The executable graph is fed into a parallel execution component. The main concept of this component is to transform tree-like sequential computation into high-throughput synchronous computation for sequence detection and subsequent decision making. After generating the DAG, how to process it efficiently remains a challenge, especially for complex DAGs and large-scale data. We propose a computation strategy that transforms the sequential decision graph computation problem into node status detection and node status sequence mapping. A diagram of this concept is given in Figure 6.

[0062] In particular, Figure 6 illustrates a proposed adaptive computation strategy according to an embodiment, where sequential graph computing is transformed into high-throughput synchronous status detection and status sequence mapping. In "A," all nodes 42 in the DAG have no data transfers, and then each node 42 is transformed into a status in the status sequence 43; in "B," data transfers exist between some nodes 42, and those nodes 42 are transformed into one status in the status sequence 43.

[0063] First, a status sequence list 44 is generated that lists all possible status sequences 43 and their corresponding outcomes according to the defined GCF. The relationship between status sequences 43 and outcomes can be many-to-one.

[0064] As detailed above (in the section entitled "DAG"), adaptive strategies are employed to increase efficiency in the transformation process. In the first situation, "A," there is no intermediate data transferred between two nodes (i.e., there is no dependency between the nodes). Therefore, it is considered an independent decision node and can be transformed into a single status in the status sequence. In the second situation, "B," there is data transfer between two nodes in the DAG (i.e., there is a dependency between the nodes), and the nodes are merged as a node group and then transformed into a status in the sequence (as indicated by the dashed box surrounding the merged nodes).

[0065] Two key aspects contribute to improved efficiency during the DAG computation process: (a) parallelized node status detection, rather than sequential decisions that derive each node status; and (b) status sequence mapping. When large amounts of data are fed into the graph, a decision status is generated for every decision node or group of decision nodes. All statuses are then concatenated into a status sequence in a predefined order. The system then maps the sequence to a sequence list to obtain a result. For example, in a clinical decision-making scenario, GCF defines three nodes (A, B, and C), where every node is a binary choice (0 for false, 1 for true). A status sequence of "100" represents A being true, B being false, and C being false, and the result is "having the disease," while "001" can result in "not having the disease." All these sequences form a list of status sequences.

[0066] - Multilayer calculation Multi-layer computation handles situations when both the data and the number of clinical knowledge graphs being injected are increasing, as is typically the case. While new data continues to enter, new clinical knowledge continues to be continuously injected into the system, and longitudinal data (phenotypes) change over time. As more data becomes available, computations for some of the stored graphs can continue. As more knowledge is completed, new computations can be activated using existing data in the system to derive new data. For example, consider a knowledge graph being injected to monitor whether a patient has a recurrent tumor. When there is only one radiology report supporting cancer recurrence, a decision cannot be made according to clinical knowledge. However, when one additional radiology support or tumor biomarker support enters the system, computations can continue to derive new data variables related to tumor recurrence. If the accumulated data and knowledge are not well organized, it becomes impossible for users to track the process of each computation result.

[0067] As described, the proposed system periodically reruns the ETL, retrieving new data from the EMR and performing computations. Here, the data variables generated during the initial ETL are called the base data layer. Newly derived data are recursively placed in a new data layer on top of the base layer. In particular, after injecting a new knowledge graph, each node in that graph is mapped to a standardized, unique system code via concept encoding. Such encoding is used to build an inverted index of all graph nodes, each associated with a graph structure containing the node and layer number. New input data is assigned a concept code based on its data type via the same encoding process. The concept code is then searched using the inverted index to obtain all graphs containing nodes with the same encoding. This is an extremely fast search process. Typically, if a node encoding is covered by an already injected graph, the data is added as a graph node to the matching sequence for further processing. However, if it is not covered, it is added as a node to the unmatched sequence. For nodes within the matching sequence, they are added to the system and added to the active graph computation that derives new data. The algorithm is described as follows:

[0068] The new node is copied and added to all layers where a graph structure containing this node exists in the index. If the added node can cooperate with other nodes in the same layer to derive new data values ​​for some graph, a new layer is created to contain this newly derived data node; otherwise, the new node is added to an existing higher layer. To determine this, the algorithm again checks all graphs in only the higher layer that contain the newly derived node. If a higher layer exists that contains the node, the node is added to that layer. If not, a new higher layer parallel to the other higher layers is created. In such a new layer, all graph structures containing the newly derived node are associated with a new layer number.

[0069] Initially, there is only one layer at the bottom, called the base layer. For nodes in the mismatched sequences, the system currently has no graph structure that covers them. These data nodes may or may not be covered by new knowledge graphs injected into the system in the future. Therefore, when new knowledge is injected, the algorithm always checks whether nodes from the mismatched sequences can be added. If not, these data nodes will only provide normal data points for the system.

[0070] For further explanation, an example is shown in Figure 7. More specifically, Figure 7 shows an example of a multi-layer computation and data architecture according to the proposed embodiment. A and B in the first base layer 53 can derive E in group 55 in layer 1.1. E and F further derive H in group 57 in layer 2.1. However, C and D are not covered by any graph in group 53 of A and B. Therefore, a second, parallel base layer 54 is created to include C and D. C and D derive G in group 56 in layer 1.2. Also, since G is not covered by the graphs of E and F in layer 1.1 and is included by both group 56 in layer 1.2 and group 57 in layer 2.1, G is copied to both layers 56 and 57.

[0071] The advantage of multi-layered computation is that it is based on fast graph exploration and parallelized graph computation. Therefore, it is extremely efficient overall. Furthermore, when deriving new data items, valuable ETL allows users to clearly understand when data is added (through a hierarchical organization of information) and how these data interact with existing data. The impact of such a proposed technique is that, in the same layer, old nodes that contribute to new nodes tend to cluster together, and graph structures that share the same nodes also tend to cluster together. This can be leveraged to design hierarchical information management systems that provide a good archive of related data and knowledge. With this, users no longer need to traverse separate knowledge graphs to find each node of their interest. Users can easily see what data elements they missed in order to derive higher-level data nodes (in higher layers).

[0072] - Local Recalculation Users can reconfigure parts of the graph and specify nodes that should be recalculated. For example, a user may want to test a state-of-the-art classification algorithm to identify whether a radiology report indicates a recurrent tumor. The user replaces the old algorithm node in the graph and reruns the graph. However, the rerun does not affect other nodes, only the nodes involved. Similarly, users can replace any function node in the graph. For example, a user redefined a transaction querying clinical documents from a database to speed it up. The proposed system propagates the recalculation flow to all downstream nodes involved in this transaction.

[0073] Field of application of the present invention The main application area of ​​the proposed invention is information extraction and ETL. The proposed architecture allows extraction and transformation steps to be easily configurable and more consistent, reducing overhead. Another application area falls into knowledge computing and decision support management. While the above example is based on rule-based decision-making, all function nodes are permutable so that automated algorithms can be connected and verified within a graph structure. Because the proposed techniques connect medical databases to computational knowledge and external knowledge bases, they can contribute to search and QA applications, such as medical search engines and chatbots. Furthermore, because the proposed system processes increasingly larger amounts of data, it can be useful for event monitoring, such as early detection of recurrences and side effects.

[0074] FIG. 8 shows an example of a computer 60 for implementing an embodiment of the system as described above.

[0075] The computer 60 may include, but is not limited to, a PC, a workstation, a laptop, a PDA, a palm device, a server, storage, etc. In general, with respect to hardware architecture, the computer 60 includes one or more processors 61, memory 62, and one or more I / O devices 63 communicatively coupled via a local interface (not shown). The local interface may be, for example, but not limited to, one or more buses or other wired or wireless connections as known in the art. The local interface may have additional elements, such as a controller, buffers (caches), drivers, repeaters, and receivers, to enable communication. Additionally, the local interface may include address, control, and / or data connections to enable proper communication between the aforementioned components.

[0076] The processor 61 is a hardware device for executing software that may be stored in the memory 62. The processor 61 may be virtually any custom-made or commercially available processor, a central processing unit (CPU), a digital signal processor (DSP), or an auxiliary processor among several processors associated with the computer 60, and the processor 61 may be a semiconductor-based (in the form of a microchip) microprocessor or microprocessor.

[0077] Memory 62 may include any one or combination of volatile memory elements (e.g., random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and non-volatile memory elements (e.g., ROM, erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), tape, compact disc read-only memory (CD-ROM), disk, diskette, cartridge, cassette, etc.). Furthermore, memory 62 may incorporate electronic, magnetic, optical, and / or other types of storage media. It should be noted that memory 62 may have a distributed architecture in which various components are located remotely from one another but may be accessed by processor 61.

[0078] The software in memory 62 includes one or more separate programs, each of which includes an ordered list of executable instructions for performing a logic function. According to an exemplary embodiment, the software in memory 62 includes a suitable operating system (O / S) 64, a compiler 65, source code 66, and one or more applications 67.

[0079] The application 67 includes many functional components, such as computational units, logic, functional units, processes, operations, virtual entities, and / or modules.

[0080] The operating system 64 controls the execution of computer programs and provides scheduling, input / output control, file and data management, memory management, and communication control and related services.

[0081] An application 67 may be a source program, an executable program (object code), a script, or any other entity that includes a set of instructions to be executed. When a source program, the program is typically translated via a compiler (e.g., compiler 65), assembler, interpreter, etc., which may or may not be contained within memory 62, so that it operates properly with operating system 64. Furthermore, application 67 may be written in an object-oriented programming language with classes of data and methods, or a procedural programming language with routines, subroutines, and / or functions, such as, but not limited to, C, C++, C#, Pascal, BASIC, API calls, HTML, XHTML, XML, ASP script, JavaScript, FORTRAN, COBOL, Perl, Java, ADA, .NET, etc.

[0082] The I / O devices 63 include input devices such as, but are not limited to, a mouse, keyboard, scanner, microphone, camera, etc. Additionally, the I / O devices 63 also include output devices such as, but are not limited to, a printer, display, etc. Finally, the I / O devices 63 further include both input and output communication devices, such as, but not limited to, a network interface controller (NIC) or modulator / demodulator (for accessing remote devices, other files, devices, systems, or networks), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, etc. The I / O devices 63 also include components for communicating over various networks, such as the Internet or an intranet.

[0083] When computer 60 is in operation, processor 61 is configured to execute software stored in memory 62, communicate data to and from memory 62, and generally control the operation of computer 60 in accordance with the software. Applications 67 and operating system 64 are read, in whole or in part, by processor 61, possibly buffered within processor 61, and then executed.

[0084] It should be noted that when the application 67 is implemented in software, the application 67 may be stored on virtually any computer-readable medium for use by or in connection with any computer-related system or method. In the context of this document, a computer-readable medium is an electronic, magnetic, optical, or other physical device or medium that contains or can store a computer program for use by or in connection with a computer-related system or method.

[0085] Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the words "comprises," "has," "comprises," "includes," "includes" do not exclude other elements or steps, and the singular does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

[0086] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the words "having," "comprising," and "including" do not exclude other elements or steps, and the singular does not exclude a plurality. A single processor or other unit performs the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be advantageously used. Where computer programs are described above, the programs may be stored or distributed on suitable media, such as optical storage media or solid-state media, provided together with or as part of other hardware, or in other forms, such as via the Internet or other wired or wireless telecommunications systems. It should be noted that where the term "adapted for" is used in the claims or the specification, the term "adapted for" is equivalent to the term "configured for." Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. an interface for obtaining a plurality of graph components, each graph component containing information representing one or more computation logic rules; a computation graph generator that generates a computation graph based on the obtained graph configuration components, the generated computation graph comprising a node of each graph configuration component and one or more links representing relationships between the nodes; a computation graph adapter that receives external information related to the graph construction component from an external source and adapts the generated computation graph into a directed acyclic graph structure based on the external information; a computation graph runner that executes the adapted computation graph; Equipped with each said graph-construction component comprising an algorithm and / or computational logic rules; The ETL pipeline system, wherein the algorithms include computational data extraction rules and / or the computational logic rules include computational decision-making rules.

2. Each of the graph configuration components is Database connection and Documentation and A data element; 2. The ETL pipeline system of claim 1, comprising at least one of:

3. The external information is a user-defined graph configuration component; External knowledge from external knowledge databases 3. The ETL pipeline system according to claim 1, further comprising at least one of:

4. 4. The ETL pipeline system of claim 3, wherein the external knowledge includes a preceding computation graph stored in a database, and the preceding computation graph is added to the generated computation graph to generate an adapted computation graph.

5. 5. The ETL pipeline system of claim 3, wherein the user-defined graph configuration components include information that redefines and / or modifies one or more graph configuration components.

6. A medical data processing system comprising the ETL pipeline system according to any one of claims 1 to 5.

7. 1. A method of operating an ETL pipeline system, comprising: a processing unit of the ETL pipeline system obtaining a plurality of graph components, each graph component including information representing one or more computational logic rules; The processing unit generates a computation graph based on the obtained graph-constituting components, the generated computation graph including: a node for each graph-constituting component, and one or more links representing relationships between the nodes; receiving, by the processing unit, external information related to the graph-constructing component from an external source; the processing unit adapting the generated computation graph into a directed acyclic graph structure based on the external information; the processing unit executing the adapted computation graph; and each said graph-construction component comprising an algorithm and / or computational logic rules; A method of operating an ETL pipeline system, wherein the algorithms include computational data extraction rules and / or the computational logic rules include computational decision-making rules.

8. 1. A computer-readable storage medium containing program instructions for ETL pipeline processing, the program instructions being configured by a processing unit to: obtaining a plurality of graph-configuration components, each graph-configuration component including information representing one or more computational logic rules; generating a computation graph based on the obtained graph-configuration components, wherein the generated computation graph includes a node for each graph-configuration component and one or more links representing relationships between the nodes; receiving external information related to the graph-composing component from an external source; adapting the generated computation graph based on the external information; executing the adapted computation graph; is executable, each said graph-construction component comprising an algorithm and / or computational logic rules; The computer-readable storage medium, wherein the algorithms include computational data extraction rules and / or the computational logic rules include computational decision-making rules.

Images